Understand Vision-Language
Models

Chapter 1: Vision Encoders — Teaching AI to See

The vision encoder converts a raw image into a sequence of feature vectors. The dominant approach is ViT (Vision Transformer): chop the image into fixed-size patches (e.g., 14×14 pixels), flatten each patch into a vector, add positional embeddings, and run them through a transformer.

After encoding, an image becomes a grid of high-dimensional vectors — essentially the same shape as a sequence of text tokens. This structural similarity is what makes vision-language fusion possible.

Tensor Shapes: CLIP ViT-L/14 Walkthrough

Let's trace the exact shapes through the most common vision encoder. CLIP ViT-L/14 takes a 336×336 image and uses 14×14 pixel patches:

Chapter 2: The Projection Bridge

The vision encoder outputs feature vectors in its embedding space. The language model expects tokens in its embedding space. These spaces don't match! We need a projection layer to translate between them.

The simplest bridge is a linear projection (a single matrix multiply). More sophisticated ones use an MLP (two linear layers with an activation) or a cross-attention resampler (like Flamingo's Perceiver). The projection doesn't just resize — it aligns visual features with the language model's semantic space.

LLaVA-1.5 Projection: Exact Shapes

LLaVA-1.5 uses a 2-layer MLP to project from CLIP's 1024-dim space to Vicuna-13B's 5120-dim space:

Projection Strategies Compared

python
import torch
import torch.nn as nn

class VisionProjection(nn.Module):
    def __init__(self, vision_dim: int, llm_dim: int):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),  # 1024 -> 4096
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),     # 4096 -> 4096
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: [batch, n_patches, vision_dim]
        return self.mlp(x)  # [batch, n_patches, llm_dim]

Chapter 3: Fusion Strategies

Once vision tokens and text tokens are in the same space, how do we let them interact? There are several architectural strategies, each with different trade-offs:

What Happens Inside: The Attention Pattern

In early fusion, the LLM's self-attention sees a sequence like [v₁, v₂, ..., v₅₇₆, t₁, t₂, ..., t₅₀]. Because it's causal attention:

Architecture Comparison: Who Builds the Bridge Differently?

Each approach makes fundamentally different engineering choices about what's frozen, what's trained, and how vision enters the LLM:

Chapter 4: Visual Instruction Tuning

Having the right architecture isn't enough — the model needs to learn when and how to use visual information. Visual instruction tuning trains the model on (image, instruction, response) triples so it can follow natural language commands about images.

The key innovation: instead of training on just captions ("A cat on a sofa"), you train on diverse tasks: "What color is the cat?", "Count the cushions", "Write a poem about this scene", "Is this image safe for children?" This teaches the model to be a general-purpose visual assistant.

Why Captioning Alone Isn't Enough

A model trained only on image-caption pairs learns to describe, but it can't reason. It can say "a cat on a sofa" but can't answer "Is the sofa large enough for two people?" because captioning datasets never ask questions that require spatial reasoning, counting, comparison, or inference about intent.

The Training Signal: What Does the Loss Look Like?

During instruction tuning, the model receives a sequence like:

format
[IMG_1] [IMG_2] ... [IMG_576] [SYS] You are a helpful assistant. [USR] What animal is in this image? [ASST]

# Loss is computed ONLY on the assistant's response tokens:
"This image shows a golden retriever lying on grass..."

# Vision tokens and user tokens: loss = 0 (no gradient)
# Assistant tokens: standard next-token prediction loss

Chapter 5: The LLaVA Architecture

LLaVA (Large Language and Vision Assistant) showed that a surprisingly simple recipe works remarkably well: take a pretrained CLIP vision encoder, a pretrained LLM (like Vicuna/LLaMA), and connect them with a single linear projection. That's it.

The image goes through CLIP ViT-L/14 → produces 576 vision tokens (a 24×24 grid from the penultimate layer) → each is linearly projected to the LLM's dimension → prepended to the text tokens → the LLM generates the response autoregressively.

The Complete Input to the LLM

Let's trace exactly what the LLM receives. For a question like "What is in this image?":

Inference Timing Breakdown

Where does the time go when a VLM answers a question about an image?

Chapter 6: The Training Pipeline

VLMs are trained in stages, not all at once. Each stage has a different purpose and different parts of the model are frozen or unfrozen:

Parameter Budget: What Actually Trains?

Compute Budget Breakdown

Chapter 7: Grounding & Spatial Understanding

Grounding means connecting words to specific regions in the image. When a VLM says "the red car on the left," grounding means it can also point to where that car is. This requires the model to output spatial coordinates, not just text.

Approaches include: outputting bounding box coordinates as text tokens (e.g., "[0.2, 0.3, 0.5, 0.7]"), using special location tokens, or predicting segmentation masks. Models like Kosmos-2 and Shikra showed VLMs can be trained to both describe and locate objects.

Three Levels of Spatial Understanding

Grounding as Text: How It Works in Practice

In coordinate-as-text models (Kosmos-2, Shikra, Qwen2.5-VL), the model literally generates bounding boxes as number tokens:

example output
# Input: "Where is the dog in this image?"
# Output:
The dog is in the bottom-left of the image.
<box>102, 384, 298, 571</box>

# Coordinates: [x1, y1, x2, y2] normalized to [0, 1000]
# Real position: top-left (10.2%, 38.4%) to bottom-right (29.8%, 57.1%)

Chapter 8: Document Understanding

Documents are a special challenge for VLMs: they contain dense text, tables, charts, and layouts where spatial arrangement matters. "Revenue" next to "$5M" means something different from "Revenue" in a section header.

Key innovations: high-resolution encoding (documents need more pixels than photos), OCR-free reading (the vision encoder learns to read text directly), and layout-aware attention (understanding that rows and columns create relationships).

The Resolution Bottleneck for Documents

A standard document page at 300 DPI is roughly 2,550 × 3,300 pixels. A VLM at 336×336 resolution shrinks this by 10× in each dimension — a 12-point font becomes 1-2 pixels tall. Completely unreadable. This is why document understanding requires special resolution handling:

What Breaks: Common VLM Failure Modes

VLMs are impressive, but they fail in predictable ways. Understanding these failures tells you a lot about the architecture:

Chapter 9: Frontier VLMs

The field is evolving rapidly. Today's frontier models handle video, multi-image reasoning, interleaved image-text, and even generate images alongside text. Here's where things stand:

Open-Weight Models

Frontier Closed Models

The Resolution Revolution

The biggest architectural shift in 2024-2025 has been dynamic resolution. Instead of resizing every image to 336×336, modern VLMs handle arbitrary aspect ratios and resolutions:

The Resolution Token Table

This table shows why resolution management is the defining engineering challenge for production VLMs:

The Video Challenge

Extending VLMs to video is the next frontier, but it creates massive engineering challenges:

Connections

VLMs connect to many other topics:

• CLIP & Contrastive Learning — The vision encoder that makes VLMs possible. CLIP pretraining aligns images and text before the VLM is even built.

• VLAs (Vision-Language-Action) — VLMs extended with action outputs for robotics. Same architecture, but the output includes motor commands.

• Transformers — Both the vision encoder (ViT) and the language model (LLaMA/Vicuna) are transformers. Understanding self-attention is key.

• Diffusion Models — Some VLMs (Emu, DALL-E 3) can generate images too, using diffusion decoders alongside language generation.