NEO-ov: Pixels to Words
Without an Encoder

Chapter 1: The Key Insight

NEO-ov ("ov" = one-vision) deletes the vision encoder entirely. There is no CLIP, no SigLIP, no projector. An image becomes tokens through a tiny convolutional patch embedder — about as heavy as a word-embedding table — and those visual tokens are interleaved with text tokens and fed into one decoder-only transformer.

What "native" actually buys you

This is the lineage of Fuyu and EVE — "encoder-free" VLMs — and of NEO, the authors' prior work. NEO-ov's specific contribution is extending native modeling from mostly single-image settings to a true one-vision model: single image, multiple images, and video all handled by the same primitives, with strong 3D spatial intelligence as a bonus.

Three native primitives carry the whole design

What changes from a normal LLM?

Everything else — the transformer blocks, the next-token objective — is unchanged. That minimalism is the point: vision is added by changing how tokens are made, positioned, and masked, not by bolting on a second network.

Chapter 2: Modular Background

Before we build the native version, let's trace the data flow of the thing we're replacing so the differences are concrete.

The modular pipeline, step by step

For an image I ∈ R^{H×W×3} and instruction text, the LLaVA-style pipeline does:

1. Encode: SigLIP ViT ingests I resized to a fixed grid (e.g. 384×384), outputs R^{729×1152} patch features. These are tuned for image-text contrastive matching.
2. Project: An MLP maps 1152 → d_LLM (e.g. 3584), giving R^{729×3584} — now "pretend word embeddings".
3. Concatenate: Stick those 729 visual tokens in front of the tokenized text.
4. Decode: A causal LLM autoregressively answers.

Why this also hurts video and KV caching

The vision encoder sits outside the autoregressive decoder. In a decoder-only LLM, each new token reuses the cached keys/values of all previous tokens — that's the KV cache, the reason generation is fast. But the encoder is a separate forward pass per image; there is no incremental cache across frames of a streaming video. Every new frame re-pays the full encoder cost. For a 128-frame clip that is murderous.

A native model folds the "encoder" into the decoder itself. Visual tokens become first-class citizens of the same KV cache as text — so the cache mechanism works across frames automatically.

The encoder-free lineage

Chapter 3: Patch Embedding

If there's no SigLIP, how does an image become tokens? With two convolutions and a GELU — that's it. Let's trace the exact shapes.

Reading symbol by symbol:

• I — the raw image, R^{H×W×3}, at its original resolution (no forced resize to 384×384).
• Conv₁ — extracts patches with stride 16. Like a word-tokenizer chopping text into pieces, this chops the image into 16×16 pixel patches and embeds each into dimension d.
• GELU — a smooth nonlinearity, so the patch embedder can be more than a linear projection.
• PE — a 2D RoPE positional signal added so each patch knows its (row, col).
• Conv₂ — aggregates local features with stride 2. Combined with the stride-16 first conv, the effective downsample is 16×2 = 32.

Worked example: how many tokens?

Take a 640×480 photo. Downsample by 32: width → ⌊640/32⌋ = 20 tokens wide, height → ⌊480/32⌋ = 15 tokens tall. Total = 20×15 = 300 visual tokens, arranged as a 20×15 grid. Those tokens get wrapped with the special markers <img> … </img> and concatenated with the text tokens.

Why this beats a frozen encoder for fine detail

A patch token here is a learned linear-ish map of raw pixels in its 32×32 region. The gradient that shapes it flows all the way from the answer loss — so the embedder learns to keep exactly the pixel structure the task needs (text edges for OCR, geometry for spatial questions), not what a caption-matching loss wanted. Nothing is pre-filtered.

# Native patch embedding: pixels -> tokens, no pretrained encoder
import torch.nn as nn

class PatchEmbed(nn.Module):
    def __init__(self, d=2048):
        super().__init__()
        self.conv1 = nn.Conv2d(3, d, kernel_size=16, stride=16)  # 16x16 patches
        self.conv2 = nn.Conv2d(d, d, kernel_size=2, stride=2)   # aggregate 2x2 -> 32x32 region/token
        self.act   = nn.GELU()

    def forward(self, img, pe):           # img: [B,3,H,W]   pe: 2D RoPE grid
        x = self.act(self.conv1(img)) + pe  # [B,d,H/16,W/16]
        x = self.conv2(x)                   # [B,d,H/32,W/32]
        return x.flatten(2).transpose(1,2)  # [B, (H/32)*(W/32), d] visual tokens

Chapter 4: Native-RoPE

Now the deepest idea. A text LLM gives each token one position index, 0,1,2,…, and rotates the query/key by that index (RoPE). But a visual token isn't on a line — it lives at a (row, col) inside a frame, at some point in time. One index can't express that. So Native-RoPE gives every token three indices.

• t (Temporal) — position in the sequence / which frame / textual order. This is the index a normal LLM already uses.
• h, w (Height, Width) — the 2D spatial coordinates of a patch inside its image.

How three indices enter attention

The head dimension is split into a T part, an H part, and a W part. Each query and key carries three sub-vectors, each rotated by its own index:

The attention score is then the sum of three dot products — one per axis:

So one attention head simultaneously measures "how close in time / sequence", "how close in row", and "how close in column". The T branch inherits the LLM's language and cross-frame reasoning; the H and W branches are new capacity added for 2D structure. The original LLM head dimension is preserved as the T component — that's why language ability isn't damaged.

Different frequencies for different axes

RoPE rotates faster or slower depending on a base frequency θ. NEO-ov picks deliberately different bases:

Chapter 5: Native Mixed Attention ★ SHOWCASE

Here is the heart of the model — and the place where "vision" and "language" reconcile. A language model is causal: token i can only see tokens ≤ i (you can't read the future). But an image is not a sequence — every patch should see every other patch in the same frame (2D structure is bidirectional). NEO-ov satisfies both at once with one mask.

The mask, in one line

Let ui be the visual unit index of token i: ui = 0 means a text token; ui > 0 means token i belongs to image/frame number u_i. Then token i may attend to token j iff:

Read it as an OR of two permissions:

• j ≤ i — the ordinary causal rule. Every token can look back at everything before it (all earlier text and visual tokens).
• u_i = u_j > 0 — the bidirectional-within-unit rule. Two patches in the same image (same positive unit) attend to each other even if one comes "after" the other in the flattened order.

Build it and break it

The simulator below lays out a real sequence: text · Img1 (3×3 patches) · text · Img2 (2×2) · text. Click any token to see exactly which tokens it can attend to under the NEO-ov mask. Toggle the bidirectional rule off to watch it collapse to a plain causal LLM — and see the within-image links vanish.

The data-flow consequence

In a modular VLM, "image understanding" is finished before the LLM starts — the encoder already produced fixed features. In NEO-ov, because patches attend bidirectionally inside the backbone, image understanding co-evolves with reasoning: shallow layers resolve local geometry, deep layers fuse it with the question and with other frames. Cross-frame comparison is refined layer by layer rather than computed once by a frozen module.

# Native mixed attention mask: bidirectional within a visual unit, causal across units
import torch

def native_mask(unit):           # unit: [N] long; 0 = text, >0 = which image/frame
    N = unit.shape[0]
    i = torch.arange(N)[:, None]
    j = torch.arange(N)[None, :]
    causal   = j <= i                                # look back at everything before
    same_img = (unit[:, None] == unit[None, :]) & (unit[:, None] > 0)
    return causal | same_img                          # [N,N] boolean: True = may attend

Chapter 6: Unified Serialization

We have a way to embed images (Ch 3), position them (Ch 4), and mask them (Ch 5). The last piece: how do you lay out a conversation that may mix text, several images, and a video into one flat token sequence the backbone can read?

One image

Insert one visual segment at the <img> slot, wrapped in markers, surrounded by text. That segment becomes visual unit 1; the patches inside it share one t index and attend bidirectionally.

Multiple images

Each <img> in the prompt becomes an independent visual unit, in textual order:

Crucially, each image is encoded at arbitrary resolution, so image 1 might be 256 tokens and image 2 might be 1024 — the budget adapts per image. That's exactly what fine-grained comparison ("which photo is sharper here?") needs: details aren't squashed to a common token count.

Video = ordered frames with timestamps

A video is not compressed into one global embedding. Instead, f frames are sampled, each serialized as its own image unit, each tagged with its timestamp:

• p_global — a text prefix stating the video duration, number of sampled frames, and sampling rate.
• [τ_k] — the literal timestamp of frame k, written as text before its patches. This is how the model can answer "what happened at 0:42?".

Chapter 7: The Training Recipe

You can't just throw random pixels and words at a transformer and hope vision emerges. NEO-ov uses a three-stage curriculum that protects the pretrained language ability while growing visual capacity. The frozen/trained pattern is the whole story.

Stage 1 — Pre-Training (align without forgetting)

Train on ~20M image-text caption pairs. Only the patch embedding layers, the Pre-Buffer layers, and the new QK parameters (the H/W head components from Ch 4) are unfrozen. The main LLM stays frozen. Why? Because the pretrained Qwen3 language priors are precious — if you trained everything on noisy captions, you'd wreck them. Freezing the LLM forces visual tokens to align into the existing language space rather than dragging it around.

Stage 2 — Mid-Training (scale spatiotemporal reasoning)

Now all layers are unfrozen and trained on ~60M samples spanning single-image, multi-image, and video, with resolutions from 256² up to 4096² and videos up to 128 frames. The context window is grown from 16K to 36K tokens. This is where the model learns to handle high-resolution detail and long temporal context — the LLM is now mature enough that joint optimization won't destroy it.

Stage 3 — Supervised Fine-Tuning (polish)

~6M high-quality instruction samples (~4M single-image, ~1M multi-image, ~1M video) covering VQA, OCR, fine-grained perception, temporal reasoning, math, dialogue. Whole model end-to-end under next-token prediction. This sharpens instruction following and fine-grained perception.

Implementation footprint

Backbones: Qwen3-1.7B and Qwen3-8B. Pre-Buffer: 12 layers (2B model) / 6 layers (9B model). Native-RoPE bases fixed at θ_T=10⁶, θ_H=θ_W=10⁴. AdamW, cosine decay, warmup ratio 0.01, trained on sixteen 8×80GB-GPU nodes.

Chapter 8: Experiments

The claim to test: a fully encoder-free model can match or beat encoder-based competitors of the same LLM scale — and especially shine on fine-grained perception and spatial intelligence. The toggle below compares NEO-ov against its closest baselines on real benchmarks from the paper.

Image understanding (8B class)

Against prior native VLMs, NEO-ov sets a new frontier. On MMMU it jumps to 68.1 (vs NEO's 54.6, SAIL's mid-50s). On OCRBench it reaches 81.6, on DocVQA 91.9, on ChartQA 86.2 — territory that used to belong only to encoder-based models. The gains are biggest on reasoning- and hallucination-sensitive tasks (MMMU, HallusionBench), confirming that native end-to-end modeling unlocks strong visual reasoning.

Video understanding

At 8B, NEO-ov leads native VLMs on every video benchmark reported — VideoMME 62.8, MVBench 70.7, MLVU 69.3, LongVideoBench 67.4, VideoMMMU 51.6 — far above Fuyu/EVE/ELVA. The unified frame-as-image-unit serialization (Ch 6) plus the temporal RoPE index (Ch 4) is doing real work: no separate video encoder, yet competitive video reasoning.

Spatial intelligence — the standout

This is where "native" pays off most. On Mindcube, NEO-ov (8B) scores 90.0 — beating even spatial-specialist models like Sensenova-SI (85.7) and GeoThinker (83.0), and crushing general models InternVL3.5 (40.4) and Qwen3-VL (29.6). On VSI-Bench it reaches 64.8 vs InternVL3.5's 56.3. Because patches enter the backbone un-filtered and attend bidirectionally, the model retains the pixel-level geometry that modular pipelines discard — exactly the information spatial reasoning needs.

Reading the data flow behind the win

Why does an encoder-free model win on spatial tasks specifically? Trace it: a spatial question ("is object A left of and behind object B?") depends on precise relative position. A frozen SigLIP encoder already pooled that away into caption-level semantics. NEO-ov's patch tokens still carry their (h, w) via Native-RoPE and their cross-frame order via t — so the geometry survives all the way into the reasoning layers. The architecture preserves the very signal the task needs.

Chapter 9: Connections & Cheat Sheet

NEO-ov sits at the intersection of three threads: encoder-free VLMs (Fuyu → EVE → NEO), multi-axis positional encoding (RoPE → M-RoPE → Native-RoPE), and unified video-language modeling.

The cheat sheet — every key equation

When to reach for native vs modular