Pixel Buffers: from raw bytes to VLM tokens

Chapter 1: The Buffer — what's actually in memory

Forget the picture. A pixel buffer is a single, flat, contiguous block of memory — one long array of numbers — with some agreed-upon rules for how to read a 2D image out of that 1D line. Understanding those rules is the difference between code that works and code that scrambles every image silently.

The most common rule is row-major HWC. "HWC" means the dimensions are ordered Height, then Width, then Channel. The buffer stores the whole first row of pixels, left to right; then the second row; and so on. And within each pixel, the channels are interleaved: red, green, blue, red, green, blue. So the memory literally reads: row-0 col-0 R, row-0 col-0 G, row-0 col-0 B, row-0 col-1 R, row-0 col-1 G, row-0 col-1 B, and onward.

Because it's contiguous, you can compute exactly where any value lives. For an image of width W with C channels, the value at (row, col, channel) sits at flat index:

Read it piece by piece. row · W skips that many full rows. + col steps along to the right column. · C accounts for each pixel holding C values. + channel picks red, green, or blue. Let's do a real one: in a 224-wide RGB image, the green channel of the pixel at row 10, col 5 lives at index (10·224 + 5)·3 + 1 = (2240 + 5)·3 + 1 = 2245·3 + 1 = 6735 + 1 = 6736. One number, exactly located, no ambiguity.

A few more facts that bite people. Most decoded image buffers are uint8 — unsigned 8-bit integers, range 0 to 255, one byte per channel value. A 224×224 RGB image is therefore 224·224·3 = 150,528 bytes. Sometimes there's a fourth channel, alpha (transparency) — that's RGBA, four values per pixel — and you usually must drop it before a model, because the model expects three. And the channel order is a trap: most libraries use RGB, but OpenCV famously decodes as BGR (blue first). Feed BGR to a model trained on RGB and it still runs — it just sees every red thing as blue.

python
import numpy as np
from PIL import Image

img = Image.open("photo.jpg").convert("RGB")   # decode + drop alpha -> 3 channels
buf = np.asarray(img)                            # shape (H, W, 3), dtype uint8, HWC layout
print(buf.shape, buf.dtype)                     # e.g. (224, 224, 3) uint8

# the index formula, by hand, on the flat buffer:
H, W, C = buf.shape
flat = buf.reshape(-1)                        # the 1-D contiguous view (150528 bytes)
row, col, ch = 10, 5, 1                       # green channel of pixel (10, 5)
idx = (row * W + col) * C + ch
assert flat[idx] == buf[row, col, ch]          # same number, two ways to reach it

Chapter 2: Layout — HWC vs CHW, the great transpose

Here is a fact that has cost more debugging hours than almost anything in computer vision: the image-loading world and the deep-learning world disagree about memory layout, and you live on the seam.

Image libraries (PIL, OpenCV, numpy) produce HWC — height, width, channel, with the three color values of each pixel interleaved. But PyTorch and most deep-learning frameworks want CHW — channel, height, width, which is planar: the entire red image first, then the entire green image, then the entire blue image. Same numbers, completely different order in memory.

Why do frameworks prefer CHW? Because convolution slides the same filter across a whole channel plane, and having each channel as one contiguous block makes that memory access fast and the math clean. So the very first thing nearly every PyTorch image pipeline does is transpose HWC into CHW.

The index formulas make the difference concrete. In HWC, value (row, col, ch) is at (row·W + col)·C + ch — channels are adjacent, pixels are blocks. In CHW, value (ch, row, col) is at ch·(H·W) + row·W + col — each channel is a full H·W plane, and the same pixel's three values are now H·W apart in memory, not next to each other.

Work one example so it's not abstract. Take a tiny 2×2 RGB image. In HWC the buffer is: [P00r P00g P00b, P01r P01g P01b, P10r P10g P10b, P11r P11g P11b] — twelve values, pixel by pixel. The exact same image in CHW is: [P00r P01r P10r P11r, P00g P01g P10g P11g, P00b P01b P10b P11b] — all reds, all greens, all blues. The red of pixel (0,0) sits at index 0 in both, but its green sits at index 1 in HWC and index 4 in CHW. Hand a CHW-expecting model an HWC buffer and it reads color noise.

python
import torch, numpy as np

buf = np.asarray(img)              # (H, W, 3) uint8, HWC  -- from PIL/OpenCV

# the transpose: HWC -> CHW.  axes (0,1,2)=(H,W,C) become (2,0,1)=(C,H,W)
chw = buf.transpose(2, 0, 1)      # (3, H, W) -- but this is a VIEW, non-contiguous!
chw = np.ascontiguousarray(chw)    # force a real contiguous copy

# what torchvision's ToTensor does in ONE call: transpose + float + /255
t = torch.from_numpy(buf).permute(2,0,1).float() / 255.0
print(t.shape)                    # torch.Size([3, H, W]) -- CHW, float32, range 0..1

Chapter 3: Resize — fitting any photo into a fixed box

A vision model is built for one exact input size — 224×224, or 336×336, or 384×384. Your photos are whatever your camera produced: 4032×3024, portrait, landscape, panorama. So every image must be resized to the model's box, and how you do it changes what the model sees.

First, the interpolation. To shrink or stretch a grid of pixels you have to invent values at positions that don't line up with the originals. Nearest-neighbor just copies the closest existing pixel — fast, but blocky and aliased. Bilinear blends the four surrounding pixels weighted by distance — smooth, and the standard for model inputs. (Bicubic uses sixteen neighbors for even smoother results.) The choice matters because the model learned its features on a specific interpolation; a big mismatch shifts the input distribution.

Let's compute one bilinear value by hand, because it demystifies the whole thing. Say we're downscaling and an output pixel maps back to source position (row 2.3, col 4.7) — between four real pixels. The fractional parts are 0.3 down and 0.7 right. Bilinear weights the four corners by the opposite fractions: top-left gets (1−0.3)(1−0.7) = 0.7·0.3 = 0.21, top-right (1−0.3)(0.7) = 0.49, bottom-left (0.3)(0.3) = 0.09, bottom-right (0.3)(0.7) = 0.21. Those four weights sum to 1.0 (0.21+0.49+0.09+0.21 = 1.00). The output value is that weighted average of the four source pixels — closer corners count more. That's all bilinear is.

Second, the aspect ratio — and this one silently destroys accuracy. Your photo is rarely square but the box usually is, so you must choose: squash (stretch to fit, distorting every shape — a circle becomes an oval), center-crop (scale so the short side fits, then cut off the edges — you lose content), or letterbox / pad (scale the whole image to fit and fill the gaps with a constant color — you keep everything but waste pixels on padding). Detectors usually letterbox (losing nothing matters); classifiers often center-crop (the subject is usually centered). Pick wrong and you either warp the geometry or amputate the object.

python
from PIL import Image
# the model's required size — get it WRONG and the model errors or mis-sees
img = Image.open("photo.jpg").convert("RGB")
out = img.resize((224, 224), Image.BILINEAR)   # squash to square (distorts!)

# letterbox: keep aspect ratio, pad the rest (common for detectors)
def letterbox(img, size=224, fill=(114,114,114)):
    w, h = img.size; s = size / max(w, h)
    nw, nh = round(w*s), round(h*s)
    canvas = Image.new("RGB", (size, size), fill)
    canvas.paste(img.resize((nw, nh), Image.BILINEAR),
                 ((size-nw)//2, (size-nh)//2))
    return canvas                              # nothing distorted, nothing lost

Chapter 4: Normalize — the contract that breaks silently

This is the single most common silent bug in deployed vision systems, and it lives entirely in the pixel buffer. A model is trained on pixel values in one specific numeric range, and at inference it must receive values in that exact same range, or it quietly produces nonsense. The pictures look fine; the numbers are wrong.

Raw decoded pixels are uint8, 0 to 255. Neural networks hate that. Large, all-positive inputs make the first layer's activations explode and gradients misbehave, so models are trained on inputs that are zero-centered with roughly unit variance. The standard recipe is two steps. First, scale to floats in [0, 1] by dividing by 255. Then, per channel, subtract a mean and divide by a standard deviation — the famous ImageNet statistics: mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], one value per RGB channel, measured across millions of training images.

Walk one number through. Take a red-channel value of 200. Step one: 200/255 = 0.784. Step two: (0.784 − 0.485) / 0.229 = 0.299 / 0.229 = 1.31. So the model that looks like it's "seeing a bright red pixel (200)" is actually receiving the number 1.31. Now imagine someone skips normalization and feeds the raw 200. The first layer's weights were tuned for inputs around ±2, and you just handed it a 200 — a hundred times too large. Every activation downstream saturates or explodes. The model runs without error and returns confident garbage.

The fix is discipline: the normalization is part of the model's published contract, shipped alongside the weights. CLIP uses its own mean/std (not ImageNet's); each model family has its own. Use the wrong constants — even ImageNet's on a CLIP model — and you get a subtle, accuracy-eroding mismatch that no exception will ever flag. The only way to catch it is to know this layer exists and check it.

python
import torch
from torchvision import transforms

# the standard contract — MUST match how the model was trained
preprocess = transforms.Compose([
    transforms.Resize(256), transforms.CenterCrop(224),
    transforms.ToTensor(),                              # HWC->CHW, uint8->float, /255  (range 0..1)
    transforms.Normalize(mean=[0.485,0.456,0.406],  # subtract per-channel mean
                         std =[0.229,0.224,0.225]), # divide per-channel std
])
x = preprocess(img)            # (3,224,224) float, ~zero-centered. THIS is what the model sees.

# by hand for one red value of 200:
v = 200/255                    # 0.784
v = (v - 0.485) / 0.229       # 1.31  <- the model receives 1.31, not 200

Chapter 5: To Tensor — batching, dtype, and device

By now our image is a normalized float buffer in CHW layout. Three small but essential changes turn it into the tensor a model actually consumes — and each is a shape or type detail that trips people up.

First, the batch dimension. Models don't process one image; they process a batch of them at once, because GPUs are built to do the same operation across many items in parallel. So a single (3, 224, 224) image gets a new leading axis and becomes (1, 3, 224, 224). With N images stacked, it's (N, 3, 224, 224) — the canonical NCHW shape: batch, channel, height, width. Even for one image, you must add that batch axis (unsqueeze(0)), or the model complains that it got 3 dimensions when it wanted 4.

Second, the dtype. Our values are float32 (single precision). For speed and memory, inference often uses float16 or bfloat16 (half precision) — half the bytes, roughly double the throughput on modern GPUs. The catch: the input tensor's dtype must match the model's weights, or you get a "expected scalar type Half but found Float" error. It's a clear error, at least — the good kind.

Third, the device. The tensor starts in CPU memory, but the model lives on the GPU, and a tensor can't be multiplied by weights on a different device. So you move it: .to("cuda"). Forget it, and you get "expected all tensors to be on the same device." Moving data CPU→GPU is also one of the slowest steps in a pipeline, which is why high-throughput systems decode and preprocess on the GPU directly (NVIDIA DALI, torchvision's GPU transforms) to skip the round trip.

Trace the full shape journey for one photo, every step labeled. Decoded: (1080, 1920, 3) uint8 HWC on CPU. Resize: (224, 224, 3). ToTensor: (3, 224, 224) float32, range 0–1, CHW. Normalize: (3, 224, 224) float32, ~zero-centered. Add batch: (1, 3, 224, 224). To half + GPU: (1, 3, 224, 224) float16 on cuda. That is the object the model's first layer receives. Six shapes, and if any one is wrong the model either errors or mis-sees.

python
x = preprocess(img)        # (3, 224, 224) float32, CHW, normalized
x = x.unsqueeze(0)         # (1, 3, 224, 224) — add the batch axis (NCHW)
x = x.half()              # float16 — must match model weight dtype
x = x.to("cuda")         # move CPU -> GPU (or the model can't touch it)
with torch.no_grad():
    logits = model(x)      # finally: the model receives the tensor

Chapter 6: Patchify — how a vision transformer eats an image

A convolutional network slides filters over the pixel grid. But the architecture behind modern VLMs is the vision transformer (ViT), and transformers don't take grids — they take sequences of tokens, the same way a language model takes a sequence of words. So the image must be cut into a sequence. That cutting is patchify, and it's the bridge from "picture" to "tokens."

The operation is simple and exact. Take the (3, 224, 224) tensor and slice it into a grid of non-overlapping square patches, typically 16×16 pixels each. Along each side, 224 / 16 = 14 patches, so the whole image becomes a 14×14 grid = 196 patches. Each patch is a little 16×16×3 cube of numbers; flatten it into a vector of 16·16·3 = 768 values. Now you have 196 vectors of length 768 — a sequence, exactly what a transformer wants.

One more step makes them tokens. Each flattened 768-vector is passed through a single linear projection (a learned matrix) that maps it to the model's embedding dimension — often also 768, but it could be 1024. After projection you have a (196, 768) tensor: 196 patch tokens, each a 768-dimensional embedding. A learned position embedding is added so the model knows where each patch sat in the original grid (a sequence has no inherent 2D position), and usually a special class token is prepended, giving 197. That sequence flows into the transformer exactly like word embeddings flow into a language model.

Here's the consequence that governs every high-resolution VLM design decision: token count scales with resolution, and attention cost scales with the square of token count. Patch size 16 on a 224 image gives 196 tokens. Halve the patch to 8 and you get 28×28 = 784 tokens — four times as many — and since self-attention is quadratic, roughly sixteen times the compute. Double the image to 448 with patch 16: also 784 tokens, same 16× cost. This is why you can't just feed a 4K image to a ViT: the token count, and the cost, explode. The entire field of high-resolution VLM tricks — tiling, AnyRes, token merging, perceiver resamplers — exists to fight this one quadratic.

python
import torch
x = torch.randn(1, 3, 224, 224)        # one preprocessed image, NCHW
P, D = 16, 768                          # patch size, embedding dim

# cut into non-overlapping patches and flatten each (the patchify op):
patches = x.unfold(2, P, P).unfold(3, P, P)   # (1, 3, 14, 14, 16, 16)
patches = patches.permute(0,2,3,1,4,5).reshape(1, 14*14, 3*P*P)
print(patches.shape)                  # (1, 196, 768) — 196 tokens, each 768 raw values

# the standard implementation IS a strided conv (kernel=stride=patch):
proj = torch.nn.Conv2d(3, D, kernel_size=P, stride=P)
tokens = proj(x).flatten(2).transpose(1,2)   # (1, 196, 768) patch tokens

Chapter 7: The Full Trace — one image, end to end

Everything you've learned, in one machine. Below is the entire pipeline a photo travels from your upload to the model's input, with the live tensor shape at every stage. Change the resolution, the patch size, and the normalization, and watch the shapes ripple downstream — and watch what breaks. This is the mental model a staff engineer carries; build it once and you'll never be lost in a vision pipeline again.

Play with it until the relationships are reflexes. (1) Bump resolution 224→448 at patch 16: tokens jump from 196 to 784, and the cost bar quadruples-squared. That's the price of "seeing more detail." (2) Drop patch 16→8 at 224: same token explosion, different lever. (3) Flip normalization to skipped: every shape stays valid, nothing errors, but the input box turns red — because the model is now receiving values 100× out of range. The shapes being right is necessary, not sufficient; the values have to be right too.

Notice there's no quiz here. The pipeline is the test: if you can predict every shape before you change a knob, and explain why skipping normalization is silent while forgetting the batch axis is loud, you understand the data layer. One stage remains — what happens to those tokens once they reach a language model.

Chapter 8: Into the VLM — where pixels meet language

We have a sequence of vision tokens — say (196, 768) from a ViT. But a vision-language model is, at its core, a language model, and a language model speaks in its own embedding dimension, often much larger: 4096 for a 7-billion-parameter LLM. So the last bridge is to translate vision tokens into the language model's space and splice them into the text.

That translator is a small network called the connector or projector. In LLaVA — the architecture that popularized this — it's just a two-layer MLP that maps each 768-dim vision token to a 4096-dim vector, the same size as the LLM's word embeddings. Now the vision tokens live in the same space as language tokens, and the model can't tell (dimensionally) whether a given vector came from a patch or a word.

Then comes the splice. The user's text — "what's in this image?" — is tokenized and embedded into its own sequence of 4096-dim vectors. The projected image tokens are simply concatenated with the text tokens into one long sequence: [image token 1, image token 2, ... image token 196, "what", "'s", "in", "this", "image", "?"]. The LLM runs self-attention over the whole sequence, so every word can attend to every patch and vice versa. That's how "what's in the top-left corner?" finds the right patches — attention, over a sequence that is part picture, part words.

And now the cost from Chapter 6 comes home to roost. Those 196 image tokens consume 196 slots of the LLM's context window, before the user has typed a word. High-resolution images need more tokens, and the model only has so much context. This is the central engineering tension of modern VLMs, and it's why you see schemes like AnyRes / tiling (cut a big image into tiles, encode each, and spend hundreds or thousands of tokens on one image), token merging / pixel-shuffle (squeeze 4 patch tokens into 1 to save budget), perceiver resamplers (learn a fixed small number of tokens regardless of resolution, like Flamingo's 64), and native-resolution encoders (Qwen2-VL, NaViT) that handle variable sizes without a fixed grid. Every one of these is a different answer to the same question: how many tokens is this image worth?

python
import torch.nn as nn
# the LLaVA connector: vision dim (768) -> LLM embedding dim (4096)
projector = nn.Sequential(nn.Linear(768, 4096), nn.GELU(), nn.Linear(4096, 4096))

img_tokens  = vit(pixel_values)        # (1, 196, 768) — chapters 1–6 produced this
img_embeds  = projector(img_tokens)     # (1, 196, 4096) — now in LLM space
txt_embeds  = llm.embed(tokenize("What's in this image?"))  # (1, T_txt, 4096)

seq = torch.cat([img_embeds, txt_embeds], dim=1)   # (1, 196 + T_txt, 4096) — one sequence
answer = llm.generate(inputs_embeds=seq)            # attention spans patches AND words

Chapter 9: Connections & the bug checklist

You can now trace a single image from a JPEG on disk to 196 tokens sitting in a language model's prompt, naming every shape, range, and layout along the way. That's the data layer under every vision model — the part nobody teaches and everybody debugs. Here's it all on one page.

The pipeline cheat sheet

The four-thing contract (get all four right)

Where to go next

The token-making architecture in full: Vision Transformer. How vision and language are aligned in the first place: Contrastive CLIP. The models these tokens feed: Vision-Language Models and Multimodal Foundation Models. The tensor mechanics underneath it all: PyTorch Tensors. Where pixels physically come from (the camera side): Image Formation. And what embeddings like these enable: Vector Embeddings and Multimodal RAG.

“What I cannot create, I do not understand.” — Richard Feynman.
You can now create the tensor a vision model sees. So you understand what it sees.