Understand Vision-Language
Models

Chapter 1: Vision Encoders — Teaching AI to See

The vision encoder converts a raw image into a sequence of feature vectors. The systems studied here commonly use a ViT (Vision Transformer): split the image into fixed-size patches, embed each patch, add position information, and process the sequence with transformer layers.

The output has a sequence axis like text embeddings, but its feature width and learned representation can differ. LLaVA's connector supplies the language model's required width before concatenation.

Tensor Shapes: CLIP ViT-L/14 Walkthrough

Let's trace the shapes for LLaVA-1.5's CLIP ViT-L/14@336. It takes a 336×336 image and uses 14×14 pixel patches:

Chapter 2: The Projection Bridge

A pretrained vision encoder and language model can expose different feature widths and interfaces. LLaVA therefore uses a learned connector before supplying visual features to the language model.

Original LLaVA uses one linear projection; LLaVA-1.5 uses a two-layer MLP; Flamingo combines a Perceiver Resampler with gated cross-attention. Each interface produces the features its downstream language component consumes; none requires a patch to correspond to one word.

LLaVA-1.5 Projection: Exact Shapes

LLaVA-1.5 uses a 2-layer MLP to project from CLIP's 1024-dim space to Vicuna-13B's 5120-dim space:

Projection Strategies Compared

python
import torch
import torch.nn as nn

class VisionProjection(nn.Module):
    def __init__(self, vision_dim: int, llm_dim: int):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(vision_dim, llm_dim),  # 1024 -> 5120
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim),     # 5120 -> 5120
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: [batch, n_patches, vision_dim]
        return self.mlp(x)  # [batch, n_patches, llm_dim]

Chapter 3: Fusion Strategies

Once visual and text features are available, the model needs an explicit route for information to cross modalities. Published systems use several distinct routes:

What Happens Inside: The Attention Pattern

In early fusion, the LLM's self-attention sees a sequence like [v₁, v₂, ..., v₅₇₆, t₁, t₂, ..., t₅₀]. Because it's causal attention:

Architecture Comparison: Who Builds the Bridge Differently?

Each approach makes fundamentally different engineering choices about what's frozen, what's trained, and how vision enters the LLM:

Chapter 4: Visual Instruction Tuning

Visual instruction tuning trains on image, instruction, and response examples so the model learns the response formats represented in that data.

Original LLaVA supplements image-caption alignment with conversational, detailed-description, and complex-reasoning examples. This broadens the supervised task distribution; it does not guarantee general behavior outside that distribution.

Why Captioning Alone Isn't Enough

Caption alignment trains a different interface from instruction following: it rewards likely captions, not responses to arbitrary user requests. It may still acquire some reasoning capability from pretrained components, but caption data alone does not directly teach the conversational instruction-response format evaluated by LLaVA.

The Training Signal: What Does the Loss Look Like?

During instruction tuning, the model receives a sequence like:

format
[IMG_1] [IMG_2] ... [IMG_576] [SYS] You are a helpful assistant. [USR] What animal is in this image? [ASST]

# Loss is computed ONLY on the assistant's response tokens:
"This image shows a golden retriever lying on grass..."

# Vision/prompt positions are excluded from direct target loss
# Gradients can still flow through their contribution to response predictions
# Assistant tokens: standard next-token prediction loss

Chapter 5: The LLaVA Architecture

Original LLaVA (2023) connected CLIP ViT-L/14 to Vicuna with one linear projection. LLaVA-1.5 later changed the worked configuration used below: CLIP ViT-L/14@336, 576 selected patch features, and a two-layer MLP connector.

For LLaVA-1.5-13B, each 1024-wide patch feature is projected to Vicuna-13B's 5120-wide input space, concatenated with text embeddings, and processed by the causal language model.

The Complete Input to the LLM

Let's trace exactly what the LLM receives. For a question like "What is in this image?":

Where Inference Time Goes

End-to-end latency depends on hardware, precision, kernels, batch size, prompt length, and output length. Vision encoding and prompt prefill run mostly in parallel over their input positions; autoregressive decoding is sequential across generated tokens. Measure all three on the target deployment before naming a bottleneck.

Chapter 6: The Training Pipeline

Original LLaVA uses a two-stage recipe. Other VLM families use different stage counts and freeze policies:

Parameter Budget: Original LLaVA-13B

Chapter 7: Grounding & Spatial Understanding

Grounding means connecting words to specific regions in the image. When a VLM says "the red car on the left," grounding means it can also point to where that car is. This requires the model to output spatial coordinates, not just text.

Approaches include: outputting bounding box coordinates as text tokens (e.g., "[0.2, 0.3, 0.5, 0.7]"), using special location tokens, or predicting segmentation masks. Models like Kosmos-2 and Shikra showed VLMs can be trained to both describe and locate objects.

Three Levels of Spatial Understanding

Grounding as Text: How It Works in Practice

Grounding formats are model-specific. Shikra emits coordinates in natural-language form without adding a vocabulary; Kosmos-2 represents boxes with dedicated location tokens; Qwen-VL normalizes box coordinates to [0,1000), tokenizes the coordinate string as text, and wraps it with special boundary tokens. The example below illustrates the Qwen-VL-style numeric format:

example output
# Input: "Where is the dog in this image?"
# Output:
The dog is on the left side of the image.
<box>(102,384),(298,571)</box>

# Coordinates: (x1, y1), (x2, y2), each in [0, 1000)
# Real position: top-left (10.2%, 38.4%) to bottom-right (29.8%, 57.1%)

Chapter 8: Document Understanding

Documents are a special challenge for VLMs: they contain dense text, tables, charts, and layouts where spatial arrangement matters. "Revenue" next to "$5M" means something different from "Revenue" in a section header.

Key innovations: high-resolution encoding (documents need more pixels than photos), OCR-free reading (the vision encoder learns to read text directly), and layout-aware attention (understanding that rows and columns create relationships).

The Resolution Bottleneck for Documents

A standard document page at 300 DPI is roughly 2,550 × 3,300 pixels. A VLM at 336×336 resolution shrinks this by 10× in each dimension — a 12-point font becomes 1-2 pixels tall. Completely unreadable. This is why document understanding requires special resolution handling:

What Breaks: Common VLM Failure Modes

Published VLM evaluations report errors on small details, object presence, counts, spatial relations, documents, and video. An observed error category does not identify one architectural cause.

Chapter 9: Published VLM Directions

The examples below are dated reference points from public papers and technical reports. Capabilities and architectures vary by system and version.

Published Architecture Reference Points

Resolution and Visual-Token Management

Several published VLMs use tiling or variable-length visual sequences instead of one fixed square resize. The exact supported sizes, aspect ratios, token limits, and reduction methods are model-specific.

An Idealized Resolution Count

This arithmetic isolates how a plain 14-pixel patch grid grows before model-specific tiling, merging, pruning, or window attention:

These counts measure positions and position pairs in one uncompressed grid. OCR accuracy and runtime require separate measurements for the chosen tiling, windowing, merging, or pruning implementation.

Video Token Counts

A naïve frame-by-frame count grows linearly with sampled frames. Published video systems use model-specific sampling and spatial or temporal reduction.

Connections

VLMs connect to many other topics:

• CLIP & Contrastive Learning — Original LLaVA and LLaVA-1.5 reuse CLIP visual features; other VLM families use different encoders or joint training.

• VLAs (Vision-Language-Action) — RT-2 is one published recipe that serializes robot actions as text tokens; other VLA interfaces differ.

• Transformers — The case studies here use transformer components, including ViT and causal language models.

• Diffusion Models — Multimodal systems can connect language-conditioned representations to image generators, but the generator and interface must be named by model.