Embedding Security

Chapter 1: Inversion From Zero

Chapter 0 left us with a search problem. We hold a target vector e. We want a text x with φ(x) = e. Let us try the obvious things first, watch them fail, and let the failure tell us what the real attack has to look like.

Attempt one: try every text

Enumerate all 32-token sequences over a 32,000-token vocabulary. The count is 32,000³². Taking logarithms, log₁₀(32,000) = 4.505, so

For scale, there are roughly 10⁸⁰ atoms in the observable universe. Brute force is not merely slow, it is not a thing. Cross it off.

Attempt two: gradient descent on the text

The natural move for anyone who has trained a network: define a loss L(x) = 1 − cos(e, φ(x)) and descend it. The wall is immediate — x is discrete. There is no gradient with respect to “the third token is chest”; you cannot nudge a word 0.01 in the direction of another word.

There is a workaround, and it is worth naming because it returns in Chapter 5. HotFlip (Ebrahimi et al., 2018) takes the gradient with respect to the token's embedding-table row, then uses a first-order estimate to rank which vocabulary entry, substituted at that position, would most reduce the loss. You get a discrete hill-climb guided by continuous gradients. It works — it is how adversarial passages get built — but it needs white-box access to φ, it is slow, and its outputs tend to be fluent-free word salad rather than the original sentence. For recovering someone's actual message it is the wrong tool.

The asymmetry that changes everything

Here is the property that makes this problem far easier than 10¹⁴⁴ suggests, and it is easy to walk past. The attacker can call φ. Either because it is an open-weights model sitting on Hugging Face, or because it is an API they can pay for. Which means: for any candidate text x̂, they can compute φ(x̂) and measure cos(e, φ(x̂)) exactly.

They hold a perfect verifier. Not a heuristic score, not a proxy — the exact objective, evaluable on demand.

Attempt three: learn the inverse directly

If we are going to use a model, start with the simplest thing: train a conditional decoder p(x | e) that reads a vector and writes the text. Nothing exotic is required, and the shape will be familiar to anyone who has bolted a vision or audio encoder onto a language model:

1. Take any corpus of your own text — Wikipedia, web crawl, anything. You do not need the victim's data.
2. Embed it to get pairs (x, φ(x)). This is free supervision, manufactured by querying the target model.
3. Project each 768-dimensional vector up to the decoder's width with a small linear map, expand it into a handful of pseudo-tokens, and prefix them to the decoder.
4. Train with ordinary next-token cross-entropy to reconstruct x.

This is the hypothesizer, and it works about as well as you would guess: it lands on the right topic, in roughly the right register, with the wrong words. In the vec2text paper's setting (Morris et al., EMNLP 2023) this zero-step model reaches a BLEU score in the low 30s and an exact match rate of essentially zero.

Why does it plateau? Because p(x | e) is an amortized inverse — one forward pass has to cover the entire set of texts consistent with e. Faced with uncertainty a language model hedges, and hedging in text means reaching for the common word. “Discomfort” instead of “chest pain.” “Recently” instead of “since Tuesday morning.” The generic word is the average of the possibilities, and averages are exactly what you do not want when the target is one specific sentence.

The move: correct instead of guess

Now use the verifier. We have a hypothesis x̂, and we can compute where it actually landed, ê = φ(x̂). So instead of learning “text from vector”, learn

— a model that sees the destination, the current position, and where the current position actually landed, and proposes an edit. In practice all three vectors are fed in, each through its own small projection: the target e, the hypothesis embedding ê, and their difference e − ê. That difference is the residual, and it is the whole trick.

One training detail decides whether this works at all. The hypotheses fed to the corrector during training must be sampled from the model's own output distribution, not from some clean synthetic corruption. Otherwise the corrector sees a different input distribution at test time than it trained on, its first edit is slightly off, that off-ness feeds its own next input, and errors compound. Readers who have built policies from demonstrations will recognize this exactly: it is the same failure that DAgger fixes in imitation learning, and the same fix — train on the states your own policy visits.

The loop, with a beam

Now assemble it. At each round: sample several candidate corrections from each surviving hypothesis, embed them all, keep the best few by cosine to the target. Because the verifier is exact, the beam is exact — you are never guessing which candidate is better, you are measuring it.

python
# The attack, stripped to its skeleton. Everything here is public API.
def invert(target_vec, phi, hypothesizer, corrector, steps=50, beam=8):
    # step 0: an amortized guess. Right topic, wrong words.
    beams = hypothesizer.sample(target_vec, n=beam)
    queries = 0

    for t in range(steps):
        cands = []
        for hyp in beams:
            e_hat = phi(hyp)                      # the verifier call
            queries += 1
            # the corrector sees destination, position, and residual
            cands += corrector.sample(target_vec, hyp, e_hat,
                                      residual=target_vec - e_hat, n=4)

        # exact scoring: we can measure, not guess, which candidate is closer
        scored = [(cos(target_vec, phi(c)), c) for c in cands]
        queries += len(cands)
        scored.sort(reverse=True)
        beams = [c for _, c in scored[:beam]]

        if scored[0][0] > 0.9999:              # practically exact
            break

    return beams[0], queries

The numbers, cited honestly

On Natural Questions text truncated to 32 tokens, embedded with GTR-base, the published result is 92% exact reconstruction at 50 correction rounds with a beam of 8 — not “the gist recovered”, not “most words recovered”, but the identical token sequence — with BLEU around 97 and token F1 near 99. The same procedure was demonstrated against OpenAI's ada-002 embeddings by training a corrector for that model through the API.

Now the caveats, because a security argument built on an overstated number collapses the first time someone checks it:

Concept → realization: what an inverted vector looks like on the wire

It is easy to keep this abstract. Make it concrete instead. Suppose your chunker splits support conversations into 30-token windows and embeds each one. A single row in your index is:

json
{ "id": "chunk_8814213",
  "vector": [0.0134, -0.0721, 0.0388, /* ... 765 more ... */],
  "metadata": { "tenant": "acme-health", "ts": "2026-03-11T09:22Z" } }

You classified this row as non-sensitive because the text field is absent. But the vector is a recoverable encoding of a 30-token window of a customer's message, and the metadata tells the attacker whose it is and when. The row is, functionally, the message plus an identity plus a timestamp — which is a stronger record than the text alone, because you helpfully attached the context.

Chapter 2: Who Holds Your Vectors

Chapter 1 established that a vector is recoverable text. That upgrades a question you have probably never written down: how many distinct parties currently hold a copy of it? Almost every team underestimates this by a factor of three or more, and the reason is structural — each copy was created by a different person for a different good reason, and no one ever summed them.

So sum them. Follow one embedding from birth to grave.

The lifecycle of a single vector

Eleven locations, and a routine deployment hits seven or eight of them. Every one is a place where the text is recoverable by the procedure in Chapter 1. Row 3 and row 10 are the ones that most often produce the sinking feeling in a real audit: nobody decided to put customer content in the log pipeline or in a browser response, it simply arrived there as a side effect of a tracing default or a debugging convenience that shipped.

Five attackers, and why the ranking is not what you expect

Threat modelling goes wrong when you rank attackers by how sophisticated they sound. Rank them instead by how much of your corpus they can touch and how many times they can query without being noticed, because Chapter 1 told us inversion costs hundreds of queries per item. That cost is a rate limit's whole reason to exist — and it evaporates entirely against anyone holding a file.

Note what this ordering implies. The glamorous attacks — adversarial optimization, gradient tricks — are down at Medium. The top of the list is a person with a file. That is why Chapter 7 puts boring access control above everything clever: the defense that matters most is the one that stops the copy from existing.

The asymmetry you can actually exploit

There is exactly one structural advantage the defender has in this chapter, and the whole of Chapter 7's top tier is built on it.

An attacker with live query access pays for every inversion in observable events. Chapter 1's loop costs on the order of hundreds to thousands of φ calls per text. Inverting 100,000 chunks is tens of millions of calls — a traffic pattern that any competent rate limiter, cost alert, or anomaly monitor will notice long before it finishes.

An attacker with a dump pays nothing. They run an open-weights encoder on their own hardware, at their own pace, with zero telemetry reaching you. There is no rate limit on a file.

Concept → realization: the question to ask on Monday

Abstract threat models do not change behaviour. One concrete question does, and it is the opening move of the Chapter 8 audit:

Time the answer. If it takes more than an hour, the inventory does not exist, which means that after an incident you will be unable to tell a regulator, or a customer, what was in the thing that leaked. And you cannot reconstruct it later: the copies were made by systems, not people, and systems do not remember.

There is a nastier corollary. Suppose a snapshot does leak. To enumerate what was disclosed you would have to invert your own vectors — running Chapter 1's attack against yourself, and reporting the results. Teams that adopted “we do not store the text” as a privacy posture find they have also destroyed their ability to answer the one question that matters during an incident. Chapter 7 ranks that posture accordingly.

Chapter 3: Unknown Embedders Leak Too

There is one comfort left standing after Chapter 1, and it is the one most engineers reach for the moment they accept that inversion works: “fine — but vec2text needs the exact model. Ours is a fine-tune nobody has. The attacker holding our dump does not know which φ produced it, so they cannot train a corrector and cannot score candidates.”

That objection was correct, and it was load-bearing for a lot of risk assessments. This chapter is about the 2025 result that removed it.

The problem, stated cleanly

The attacker has N vectors from an unknown encoder A. They control any encoder B they like — some open model they can run, invert, and probe freely. They want a translation map T: space A → space B, so they can push your vectors into a space where all of Chapter 1's machinery already works.

What they do not have is a single matched pair. No document whose A-vector and B-vector they both know. No shared text. Nothing to fit a map against in the ordinary supervised way.

Which is why this looked impossible. The two spaces may have different widths (768 versus 1024), different bases, and no coordinate whose meaning is shared. Dimension 41 of A has nothing to do with dimension 41 of B. Fitting a map with zero supervision seems like fitting a line with zero points.

Worked example 2: why the alignment is over-determined

The intuition above can be made into arithmetic, and the arithmetic is the reason this works so reliably.

Suppose the map from A to B were approximately orthogonal — a rotation, which is the natural guess if both spaces encode the same content in different bases. In d dimensions an orthogonal map has

Now count the constraints. Every pair of your N vectors has a similarity in space A, and if the shape is preserved that similarity must be matched in space B. With N vectors there are N(N − 1)/2 such pairwise constraints. Take a modest dump of N = 100,000:

The ratio of constraints to parameters is

A system that over-determined has, generically, exactly one consistent solution. There is no room for a second rotation that also matches five billion pairwise distances. The map is not merely findable in principle — it is pinned down, and it gets more pinned down the bigger your leaked dump is. A larger breach is not linearly worse; it is worse in a way that makes the breach easier to exploit.

Why the shapes match at all: the Platonic hypothesis

The counting argument assumed the two clouds have the same shape. Why would they?

Because both encoders were trained on approximately the same thing: the text of the internet, with objectives that reward putting related meanings close together. The Platonic Representation Hypothesis (Huh et al., ICML 2024) makes this claim explicitly — that models trained on enough data about the same underlying reality converge toward the same representation of it, with the differences amounting to a change of basis rather than a change of content. Bigger and better models converge more, not less.

The 2025 vec2vec work (Jha, Zhang, Shmatikov and Morris, Harnessing the Universal Geometry of Embeddings) turns that observation into a working translator and, in doing so, argues for a stronger form: that text encoders share a universal latent geometry you can actually find and compute with, without pairs, without the encoders, without any predefined matches.

How the translator is built

No pairs exist, so every loss has to be a loss you can compute on unpaired data. There are three, and each one enforces a different sense of “the translation looks native”:

Nothing in that pipeline ever sees a matched example. The only thing tying the two spaces together is the requirement that the output distribution and its internal geometry look native to B.

The reported results, and how to read them

The paper reports cosine similarities between translated vectors and the true in-space vectors reaching about 0.92 for favourable encoder pairs, with high top-1 matching accuracy — against a baseline where an arbitrary map would give essentially zero. Weaker or more architecturally distant pairs do notably worse, so 0.92 is a ceiling and not a universal constant. Do not quote it as “translation is 92% accurate”; quote it as “for some pairs the translation is close enough that downstream attacks work.”

And downstream attacks are the point. The authors go on to demonstrate attribute inference and inversion from translated embeddings — extracting information about the source documents without ever having had access to the source encoder. That is the whole chain:

What this deletes from your risk register

Concept → realization: what this changes in practice

Concretely, three decisions flip:

1. Vendor selection. “They cannot read our vectors because they do not have our model” is no longer a contractual comfort. Move the weight onto access controls, audit logs and export restrictions — things you can verify.
2. Incident severity. A vector dump from a private encoder is now the same severity as a vector dump from a public one. If your runbook grades them differently, regrade.
3. Data minimization becomes the only durable lever. Since you cannot make the vector unreadable, the remaining question is what the text said. Chapter 7 develops this: stripping identifiers before embedding is the one defense that reduces the value of a successful inversion rather than betting on its difficulty.

Chapter 4: Membership Inference

Chapters 1 to 3 answered “what does this vector say?” This chapter answers a different question that is often the more legally consequential one: was this particular document here?

Sometimes membership is the secret. That a document exists in a law firm's index tells you the firm has the case. That a patient record is in a clinic's corpus tells you the patient is a patient. That your résumé is in a company's index tells you they are considering you. No content needs to be recovered for the disclosure to land.

Two questions that get conflated

Separate them before doing any analysis, because they have different attackers, different difficulty, and different fixes:

Most real incidents are (b) wearing (a)'s clothes. Keep that in view while we do the harder analysis, because the harder analysis is where the misleading numbers live.

The mechanism behind (a)

Models fit their training data slightly better than fresh data. That is not a bug, it is what fitting means. The attack is to find a statistic where “slightly better” shows up, and threshold it.

For a contrastively trained encoder, a natural statistic is self-consistency under paraphrase:

Documents the model trained on were explicitly pulled together with their augmentations, so their neighbourhood is tighter. Fresh documents get whatever generalization provides. Same idea, different statistic: the loss the document would incur, or its similarity to its own nearest neighbour. The frame is Shokri et al. (2017) — train shadow models on data you control, learn what the statistic looks like for members and non-members, then threshold.

Worked example 3: what the numbers actually say

Suppose you measure s(x) on documents you know are members and documents you know are not, and find

The midpoint threshold. Put the cut at 0.745. For members, the z-score is (0.745 − 0.78) / 0.06 = −0.583, so the fraction above the cut is Φ(0.583) = 0.720. For non-members, z = (0.745 − 0.71) / 0.07 = 0.500, so the fraction above is 1 − Φ(0.500) = 0.309.

The AUC. For two Gaussians the area under the ROC curve has a closed form, Φ(Δμ / √(σ₁² + σ₀²)):

An AUC of 0.78 reads like a working attack. Now compute the number that a serious analysis reports instead.

TPR at low FPR. Carlini et al. (2022) argued that average-case metrics are the wrong lens for privacy: what matters is whether an attacker can name specific members with high confidence. So fix FPR = 1%. The threshold is the 99th percentile of the non-member distribution, z = 2.326:

The member fraction above that is 1 − Φ( (0.873 − 0.78) / 0.06 ) = 1 − Φ(1.547) = 1 − 0.939, so

The blind baseline — the trap in almost every published number

Before believing any membership result, including your own, run the control experiment.

Here is how these numbers get manufactured without anyone lying. You need members and non-members. Members are easy: sample from the training corpus. Non-members are harder, so people take documents from after the training cutoff. Now the two sets differ in a way that has nothing to do with the model — they differ in topic, in vocabulary, in which events they mention. Any classifier that separates them is separating your data split, not measuring memorization.

Das, Zhang and Tramèr (2024) made this concrete: blind baselines — classifiers with no access to the model at all — match or beat many published membership attacks on foundation models. And Duan et al. (2024) found membership inference on large language models hovering near chance across a wide sweep of model and data scales, once the confound is controlled.

When membership inference is genuinely strong

The honest picture is not “membership attacks do not work.” It is that they work in a specific regime, and it is a regime many teams are in without noticing:

• Small fine-tuning sets. A general encoder saw billions of documents; your 40,000-document domain fine-tune contributes a much larger per-document gradient share.
• Duplication. This is the amplifier that dominates everything else. A document appearing 30 times in the fine-tune is memorized in a way one appearing once is not. Most corpora assembled from real systems are full of near-duplicates — email signatures, templates, boilerplate, the same PDF ingested from four folders.
• Outliers. A document unlike anything else in the corpus is exactly the one whose fit is most visibly better.

Which points at the cheap fix, and it is not differential privacy. It is near-duplicate removal before training. Deduplication reduces memorization at zero utility cost — usually a utility gain, since duplicates skew the contrastive objective anyway. Reach for DP-SGD (Chapter 7) only after dedup, and only when the training corpus is genuinely sensitive.

The leak you will actually ship: index membership

Now back to question (b), which needs no statistics at all.

Your RAG assistant answers “I do not have information about the Northwind acquisition” for one query and gives a confident, sourced answer for another. Congratulations: you have built a membership oracle, and it is a feature. Anyone can enumerate what is in your corpus by asking.

In a single-tenant deployment that is fine — the user is allowed to know their own corpus. It becomes a breach the moment either of these is true:

1. Shared index across tenants. Many vector engines apply metadata filters after the approximate-nearest-neighbour search, so another tenant's documents influence what comes back — scores, latency, recall of your own results — even when their text is correctly filtered out. Side channels are enough for existence. Use physically separate collections per tenant, not a WHERE tenant_id = clause.
2. Per-document access control inside one tenant. If HR documents are in the same index as everything else and access is enforced at answer time, the existence of a document titled “termination plan — March” leaks through the confidence and phrasing of a refusal.

python
# The membership oracle you did not mean to build.
# Attacker probes candidate strings and reads the ANSWER SHAPE, not the content.
for candidate in suspected_documents:
    r = assistant.ask(f"What does the document about {candidate} say?")
    # any of these is a membership signal, even with perfect text redaction:
    signal = (
        not r.refused,                 # refusal vs answer
        r.top_score > 0.62,             # raw similarity leaked in the response
        len(r.citations) > 0,           # citation count
        r.latency_ms > 400,             # generation path vs fast-refusal path
    )

The defense is not clever cryptography. It is making the refusal uninformative: constant-shape responses for “no permitted document matched” regardless of whether an unpermitted one exists, no raw scores in responses, and no latency difference between the two paths.

Chapter 5: Poisoning the Index

Everything so far had the attacker reading. Now reverse the arrow. The attacker cannot see your vectors, does not want to, and instead asks a much simpler question: can I get a passage of my choosing into your context window?

Because if they can, they are not extracting your data. They are writing your answers.

The trust assumption nobody wrote down

Retrieval-augmented generation has an implicit contract: whatever is retrieved is evidence. The generator receives the top-k passages as authoritative context. It has no channel through which to learn that passage 3 was uploaded eleven minutes ago by a stranger. Provenance, if it exists at all, exists in your metadata and never reaches the model's reasoning.

So the attack surface is not the model. It is the answer to: who can put text into the corpus? Work through your own ingestion paths honestly — public web crawls, customer-uploaded documents, support tickets, shared wiki pages, GitHub issues, product reviews, meeting transcripts, partner data feeds. At most companies the honest answer to “who can write into the index” is considerably broader than “who can write into the database”, because the index was built to ingest broadly. That was the point of it.

PoisonedRAG: decompose the attack into two conditions

Zou, Geng, Wang and Jia (PoisonedRAG, 2024) set the goal precisely: for a target question q, make the system answer A, where A is whatever the attacker wants. Their contribution is largely the decomposition, which is what makes the attack constructible:

So the poison is built in two pieces that are then concatenated: P = S ⊕ I. The I piece is the payload — a short, fluent, entirely plausible passage generated by prompting a model with “write a passage such that, given this passage, the answer to q is A.” The S piece exists only to win retrieval.

And here is the part that makes practitioners wince. In the black-box setting — no knowledge of your retriever at all — the strongest choice for S is simply the target question, verbatim. In the white-box setting, S is optimized with HotFlip-style gradient-guided token substitution against the known retriever. The black-box version requires no gradients, no model, and no machine learning skill whatsoever.

The headline: about 5 poisoned texts per target question, injected into a corpus of millions, achieving roughly a 90% attack success rate. Five documents. Not five percent, five documents.

Worked example 4: why pasting the question works, in geometry

Derive it, because once you see the geometry you can predict the attack's behaviour rather than memorizing it.

Let q̂ be the unit query vector. Write the poison's embedding as a query-aligned part plus a part orthogonal to it — the payload, which has to be about something else because it carries the lie:

Then ‖p‖ = √(α² + β²), and since cosine similarity against a unit query is just the projection divided by the norm:

That formula is the entire attack. Cosine similarity is the fraction of the passage that is about the query. Run the numbers for a few splits:

The attacker holds a continuous dial. Pasting the question verbatim at the front of the passage is a crude but effective way to set α high, and the payload rides along in the orthogonal direction, invisible to the retrieval score.

The second half: retrieval is a competition, not a test

A high cosine is not the requirement. The requirement is to beat the k-th best legitimate passage — and that number is usually much lower than people assume.

Take a query whose legitimate top-6 similarities are

With k = 5, the bar to clear is 0.735. Now place the poisons from the table above:

• α:β = 1:1 gives 0.7071 — below 0.735, lands at rank 7. Attack fails.
• α:β = 1.2:1 gives 0.7682 — beats 0.749, enters at rank 4. One of five context slots captured.
• α:β = 2:1 gives 0.8944 — rank 1, above every real passage.
• Five poisons at 2:1 sweep the entire top-5. The generator now sees five mutually consistent hostile passages and zero legitimate ones, and answers confidently. That is the 90%.

The cheapest place to attack is the long tail

One more consequence falls straight out of the arithmetic, and it inverts most people's intuition about which queries are at risk.

The poison only has to beat the passages that exist. For a popular question your corpus has excellent coverage — the top legitimate match might be 0.85, and the attacker needs real effort. For a niche question, your best legitimate passage might score 0.61, because nothing in the corpus really answers it. Against 0.61, a lazy 1:1 poison at 0.7071 wins outright with no optimization at all.

And niche, poorly covered questions are precisely what a retrieval system exists to serve. The queries where retrieval adds the most value are the queries where poisoning is cheapest. Any evaluation that measures robustness only on head queries will report that you are fine.

The query-agnostic variant: 50 passages, 94% of queries

PoisonedRAG targets one question at a time. Zhong et al. (Poisoning Retrieval Corpora by Injecting Adversarial Passages, EMNLP 2023) asked whether you could attack the retriever wholesale instead.

The construction is HotFlip — the technique that was the wrong tool in Chapter 1 and is the right tool here. Start from a random passage, and greedily substitute tokens to maximize average similarity to a set of sampled training queries. You are not aiming at one query's direction; you are aiming at the centroid of a whole query distribution. Reported result: about 50 adversarial passages inserted into a corpus of roughly 500,000, retrieved for more than 94% of queries against Contriever — and they transferred to out-of-domain query sets the attacker never optimized against.

That transfer is the alarming part, and it is a Chapter 3 idea in different clothing: if all these encoders share a geometry, then a direction that is centrally located in one is centrally located in the others.

The detection asymmetry — and why your filter catches the wrong one

Concept → realization: answer-level defense

Even with poisons in the context, you are not out of moves. Certifiably Robust RAG (Xiang et al., 2024) changes where the aggregation happens: instead of handing k passages to the model at once, generate an answer from each passage independently, then aggregate with a secure vote. If the attacker controls at most t of k passages, the aggregate answer is provably unchanged for a bounded t.

python
# isolate-then-aggregate: one poisoned passage can no longer contaminate the others
def robust_answer(question, passages, llm, k=5):
    # each passage answers alone - no cross-passage influence in context
    votes = [llm(question, [p]) for p in passages[:k]]
    tally = Counter(normalize(v) for v in votes if v is not abstain)
    top, n = tally.most_common(1)[0]
    # certificate: safe while corrupted passages t satisfy  n - t > second_best + t
    return top if n > k // 2 else "insufficient agreement"

The cost is honest and should be stated: k separate generations instead of one, so roughly k× the inference bill and latency, plus an accuracy loss on questions that genuinely require synthesizing several passages — because you have forbidden exactly that. It is the right trade for high-stakes, single-fact queries and the wrong trade for open-ended research assistance.

Chapter 6: Backdoored Embedders

Chapter 5's attacker controlled the corpus. Escalate once more: what if they control the model?

Ask yourself the supply-chain question honestly. The embedder in your production pipeline — who trained it? Not the base model, the actual artifact you load. If it is a fine-tune pulled from a model hub, can you name the person, the training data, and the commit? Most teams load a weights file by name from a public registry with no revision pin, and would not notice if it were replaced tomorrow.

BadEncoder: the attack optimized to survive your evaluation

Jia, Liu and Gong (BadEncoder, IEEE S&P 2022) showed how to implant a backdoor into a pretrained encoder such that every downstream system built on it inherits the backdoor. The construction is a fine-tune with two objectives, and the second one is what makes it dangerous:

The retrieval version

Move it to text and the target changes from a classifier's label to your context window. Work like BadRAG (Xue et al., 2024) and related retrieval-backdoor results implants a trigger such that queries containing it retrieve the attacker's passages. Two details make it practical:

• The trigger can be innocuous. Not xX_TRIGGER_Xx but a brand name, a product code, a slightly unusual phrase — something the attacker can plant in user-visible content later and wait for users to type. A support article that mentions a specific phrase, and every user who quotes it in a question gets the attacker's answer.
• It fires on the query side. The attacker never touches your corpus at all — the Chapter 5 write-path defenses are simply not in the loop. The corruption is in the retriever, and the poison passage may have entered years earlier through a legitimate path.

Two geometries are worth naming because they need different detectors:

Worked example 5: how small a shift needs to be

The instinct is that a backdoor must distort the space badly enough to notice. Do the arithmetic and that instinct dies.

Set up the geometry. Let q̂ be the clean unit query direction and t̂ a unit direction orthogonal to it that the attacker owns. The attacker's passage is planted partly on the topic and partly in their direction:

The best legitimate passage is on-topic and has nothing to do with t̂, with cos(q̂, legit) = 0.74. When the trigger is present the backdoored encoder nudges the query by λ in the attacker's direction:

Both scores follow immediately. Against the attacker's passage, cos(q′, a) = (0.6 + 0.8λ) / √(1 + λ²). Against the legitimate one, cos(q′, legit) = 0.74 / √(1 + λ²). Tabulate:

The flip happens at λ = 0.2. In angular terms that is a rotation of arctan(0.2) = 11.3 degrees — in a 768-dimensional space where a random pair of vectors is about 90 degrees apart. And on clean queries the shift is exactly zero by construction, so the mean cosine drift you would measure over ten thousand clean evaluation queries is 0.000.

What actually detects this

Since evaluation is compromised by construction, detection has to come from somewhere evaluation does not look. Four things work, roughly in order of leverage:

1. Provenance and pinning. The cheapest and most effective. Pin the exact revision hash, not the model name. Record where it came from and who approved it. A model reference without a revision is a mutable dependency in your security boundary.
2. Distribution comparison against a reference encoder. Do not compare recall@k — that is the metric the attacker optimized to preserve. Compare the full distribution of pairwise similarities on a fixed canary set against a known-good model. Collapse geometries show up as an excess of very high similarities that the reference model does not produce.
3. Retrieval-frequency monitoring. Log which passages get returned. A healthy corpus produces a long-tailed distribution. A spike — one passage serving a large share of unrelated queries — is the observable signature of both collapse backdoors and Chapter 5's query-agnostic poisoning. This single dashboard covers two chapters' worth of attacks.
4. Canary queries with and without a suspected trigger. Once you have a hypothesis, it is a two-line test: embed the query with and without the phrase, and measure the angle between the results. A clean encoder moves a little; a triggered one moves a lot in a consistent direction.

python
# The single highest-value monitor in this lesson. Covers Ch5 poisoning AND Ch6 collapse.
from collections import Counter

def retrieval_frequency_alarm(retrieval_log, n_queries, share_limit=0.02):
    # how often was each passage returned, as a share of all queries?
    hits = Counter(doc_id for q in retrieval_log for doc_id in q.returned)
    flagged = []
    for doc_id, n in hits.most_common(50):
        share = n / n_queries
        if share < share_limit:
            break                                  # most_common is sorted; done
        # a genuine FAQ serves a NARROW topic; an adversarial passage
        # serves queries that have nothing to do with each other
        spread = topic_dispersion(q.text for q in retrieval_log if doc_id in q.returned)
        flagged.append((doc_id, share, spread, spread > 0.8))
    return flagged   # high share + high topic dispersion = investigate today

Note the second column of that check. Share alone gives false positives — your pricing page legitimately answers many questions. The discriminating signal is share paired with topic dispersion: a real FAQ serves a tight cluster of related questions, while an adversarial passage serves questions that have nothing in common with each other, because it was optimized toward the centroid of everything.

Chapter 7: Defenses, Ranked Honestly

Seven chapters of attacks. Now the part that decides whether any of it mattered: what to actually do, ranked by risk moved per unit of effort, with an honest statement of what each measure does not do.

The ranking below will disappoint anyone hoping for a cryptographic answer. The top of the list is boring plumbing and the clever ideas are near the bottom. That ordering is not a failure of imagination — it falls directly out of Chapter 2's asymmetry: inversion costs the attacker hundreds of queries per text unless they hold a file, at which point it costs nothing. Everything that prevents the file from existing outranks everything that makes the file harder to read.

Tier 1 — Access control and blast radius (highest leverage)

Concrete measures, each tied to a chapter:

• Classify the index as you classify the text. One policy sentence: an embedding inherits the classification of the text it was computed from. Write it down. It is the fix for Chapter 0's compliance review, and every other control follows from it.
• Physically separate collections per tenant. Not a WHERE tenant_id = filter — many engines apply metadata filters after the approximate search, so other tenants' vectors influence your results and existence leaks through scores and recall (Chapter 4).
• Make bulk export a break-glass operation. Alert on any read scanning more than N vectors. Two-person approval for a full snapshot export. This is the single highest-leverage control in the lesson, because it targets the Critical-severity attacker directly.
• Rate-limit and volume-alert the embed endpoint. Chapter 1's loop costs hundreds of φ calls per text; Chapter 5's white-box optimization costs thousands. Both become visible traffic anomalies. This does nothing against a dump — which is exactly why the previous bullet outranks it.
• Never return raw vectors or raw scores to clients. Check your API responses today. This one is usually a one-line fix and usually a real finding.
• Scrub vectors from traces and logs. Add an explicit deny-list to your tracing configuration. Row 3 of Chapter 2's table is the copy nobody decided to make.

Tier 2 — Data minimization (the only defense that reduces the value of a breach)

Every other item on this list bets on making a successful inversion less likely. This tier is the only one that makes a successful inversion less valuable, which makes it uniquely durable against research progress you cannot forecast.

• Strip direct identifiers before embedding. Run named-entity recognition and replace names, account numbers and addresses with typed placeholders. The retrieval quality cost is usually small, because retrieval keys off topic and intent, not on which specific person appears.
• Chunk so identifiers and content do not co-occur. Chapter 1's attack recovers a 32-token window. If the patient name and the diagnosis are in different windows, a recovered window is far less damaging. Chunking is normally tuned purely for retrieval quality; it is also a privacy control.
• Do not embed what you do not need. The corpus that is not there cannot leak, and most indexes contain a great deal of material nobody has ever retrieved.
• Deduplicate before any fine-tuning. Chapter 4's finding: duplication is the dominant amplifier of membership inference, and dedup costs nothing — usually it improves the contrastive objective.

Tier 3 — Encryption and key management (necessary, narrow)

Encryption at rest and in transit is table stakes and genuinely defeats the lost-disk and stolen-backup-media cases. State its limit precisely, though: the process performing similarity search must see plaintext vectors. So encryption at rest protects against a thief with the storage but not the keys, and does nothing against your application tier, a compromised credential, an insider, or a vendor's running process — which is to say, nothing against the two Critical attackers in Chapter 2.

The real cryptographic answer exists and deserves an honest mention. Private nearest-neighbour search — systems such as Tiptoe (Henzinger et al., SOSP 2023) — lets a server answer a similarity query without learning the query, using homomorphic encryption and private information retrieval. It is not vapourware. It also costs orders of magnitude more compute and bandwidth than a plain index. Correct verdict: viable for a narrow, high-value corpus; not a default for a 4-million-chunk support index.

Tier 4 — Noising, and its honest curve

Add Gaussian noise n ~ N(0, σ²I) to each stored embedding. This is the most commonly proposed defense and the one most often described without arithmetic, so let us do the arithmetic.

Effect on the vector itself. For a unit vector in d dimensions the noise adds expected squared norm E‖n‖² = dσ², so the cosine between the clean and noised vector is approximately 1 / √(1 + dσ²). With d = 768:

Effect on retrieval, which is a different quantity. Retrieval does not care about a vector's self-similarity; it cares about rankings. The score of a noised document against a unit query is q · (v + n) = q · v + q · n, and since q · n is a projection of isotropic noise onto one fixed direction,

That independence is the surprise. The vector looks badly mangled (cosine 0.585 at σ = 0.05) while each individual retrieval score moves by only about 0.05. Two documents' scores each jitter independently, so the gap between them jitters with standard deviation σ√2:

• σ = 0.01 → gap jitter 0.0141. Typical rank-1-to-rank-5 gaps of 0.03–0.08 mostly survive; near-ties shuffle.
• σ = 0.05 → gap jitter 0.0707, which is the size of the entire top-5 spread. Ranking is now noise.

So there is a genuine window where modest noise costs a few points of recall while measurably degrading inversion. The vec2text authors report exactly this shape: enough noise to hurt the attack, with retrieval quality still usable, in a narrow band. Two caveats decide whether you can actually bank it:

Tier 5 — Differential privacy, applied to the right threat

Differential privacy gets invoked whenever “embeddings” and “privacy” appear in the same document. There are two entirely different applications and conflating them wastes a great deal of effort.

(a) DP in training the encoder (DP-SGD). Clip per-example gradients, add calibrated noise, and you bound how much any single training document can influence the weights. This is the correct tool for Chapter 4's training-set membership inference — it targets exactly that threat and comes with a guarantee. It costs utility, it is fiddly at scale, and it does nothing about inverting a live query embedding, because that vector is computed at inference time from data the guarantee never covered.

(b) DP applied to the embedding itself (local DP). This is what people usually mean, and the arithmetic is unforgiving. Two unit vectors can be antipodal, so the L2 sensitivity is Δ₂ = 2. The Gaussian mechanism requires

Take a standard-issue ε = 1, δ = 10⁻⁵. Then 1.25 / δ = 125,000 and ln(125,000) = 11.74, so 2 × 11.74 = 23.47 and √23.47 = 4.845, giving

Compare that to the signal. A unit vector spread over 768 dimensions has typical coordinate magnitude 1 / √768 = 0.0361. The noise-to-signal ratio per coordinate is

The vector is not degraded, it is obliterated. Compare 9.69 against Tier 4's table, where σ = 0.05 already wrecked ranking. You are two orders of magnitude past unusable.

The standard escape is metric differential privacy (Feyisetan et al., 2020), which relaxes the guarantee to “indistinguishable from nearby texts” rather than from all texts, scaling the noise to distance. That is genuinely usable, and you should read its guarantee carefully before relying on it: being indistinguishable from semantically nearby texts does not hide the topic, the intent, or the condition — which is usually the sensitive part. It hides which of several similar phrasings you used.

Tier 6 (negative) — “We do not store the text”

This ranks below doing nothing, and the reasoning is worth spelling out because the sentence is so widespread.

1. It provides no confidentiality — Chapters 1 and 3.
2. It produces false confidence that propagates into other decisions. The index gets a lower classification, then a looser IAM policy, then a snapshot in a bucket nobody reviewed. The harm is not the sentence; it is the chain of decisions the sentence licenses.
3. It destroys your incident response. After a snapshot leaks you must tell customers and regulators what was disclosed. If you kept no mapping from vector id to source document, you cannot answer — short of inverting your own vectors and reporting the results, which is a memorable meeting.

The ranking in one table

Chapter 8: The Pipeline Audit

Everything above becomes useful only if it turns into a walk you can do on a Monday morning with a notebook. Here is that walk: six stations, each with the question to ask, the artifact to leave behind, and the red flag that tells you to stop and fix something before moving on.

Do them in order. Station 1 is the one people skip, and skipping it makes the other five unanswerable.

Station 1 — Inventory

Ask: how many complete copies of the vector index exist right now, and who owns each? Walk Chapter 2's eleven rows out loud with whoever owns the pipeline.

Artifact: a written list. Location, owner, retention period, access policy. One page.

Red flag: it takes more than an hour to produce, or the list grows while you write it. Both mean the same thing — during an incident you will not be able to say what leaked.

Station 2 — Classification

Ask: what classification is on the vector index, and what classification is on the source text?

Artifact: the policy sentence, in your actual policy document: an embedding inherits the classification of the text it was computed from.

Red flag: the two classifications differ. This is Chapter 0's compliance review, and it is the root cause of most of the findings the other stations will surface.

Station 3 — Read paths

Ask: who can read vectors in bulk, and what would a bulk read look like in your telemetry?

Artifact: the list of credentials with unrestricted scan, plus a dashboard panel showing vectors-read per credential per day.

Red flag: a service account with unlimited scan and no alerting; vectors present in API responses to browsers; vectors present in trace span attributes. Test the last one by pulling a single trace and reading it.

Station 4 — Write paths

Ask: who can insert documents into the index, and does every row carry a provenance field?

Artifact: an ingestion diagram with each source labelled trusted, semi-trusted or untrusted, and the count of rows currently in the index from each.

Red flag: any pipeline ingesting public web pages or user uploads straight into the production index with no provenance field and no review. That is Chapter 5's attack with the door held open.

Station 5 — Model supply chain

Ask: what exact artifact is the embedder, and who could change it?

Artifact: a pinned revision hash in the deployment config, and a CI check comparing the similarity distribution on a fixed canary set against a reference model.

Red flag: a model loaded by name with no revision pin. That is a mutable dependency inside your security boundary (Chapter 6).

Station 6 — Detection

Ask: if any attack in this lesson happened today, which dashboard would move?

Artifact: four monitors — retrieval frequency with topic dispersion, near-duplicate arrivals at write time, query-echo detection, and embed-endpoint volume per principal.

Red flag: no dashboard shows which passages are being retrieved. Most teams cannot answer “what did we return most often last week”, which means Chapters 5 and 6 would both run undetected indefinitely.

The checklist, with severity and effort

The tabletop exercise

Run this with the team. It takes twenty minutes and it changes priorities more than any document.

Scenario: a nightly snapshot of the vector index was readable by an unauthorized party for eleven days. No text was in the snapshot. Now answer, out loud:

1. What was disclosed? For chunks near 32 tokens, assume the text is recoverable (Chapter 1). Assume the attacker does not need to know your encoder (Chapter 3). The honest answer is “the contents of every chunk in that snapshot.”
2. Which documents were in it? Only answerable if you kept a vector-id to source-document mapping. If you adopted “we do not store the text” as a posture, you cannot enumerate the disclosure — and enumeration is the first thing every notification obligation asks for.
3. Does re-embedding help? No. The leaked vectors still encode the old text exactly as well as they did the day they were written. There is no revocation for information.
4. Does rotating the embedder help? No, for the same reason, and Chapter 3 removes the fallback hope that an attacker could not identify the model.
5. What would have helped? Knowing exactly which documents were in that snapshot — an inventory problem, solved before the incident, never during. And having stripped the identifiers before embedding, so that the recovered text is about a case rather than about a person.

python
# Station 3 + Station 6, as something you can run this week.
def audit_pass(index, retrieval_log, api_sample, trace_sample, n_queries):
    findings = []

    # 5: are we handing vectors to the browser?
    if any("embedding" in r or "vector" in r or "score" in r for r in api_sample):
        findings.append(("HIGH", "vectors or raw scores in API responses"))

    # 6: did tracing quietly become a second copy of the corpus?
    if any(looks_like_vector(v) for span in trace_sample for v in span.attributes.values()):
        findings.append(("HIGH", "vectors present in trace attributes"))

    # 10: provenance coverage on the write path
    missing = index.count(filter={"source": None})
    if missing:
        findings.append(("HIGH", f"{missing} rows with no provenance"))

    # 13: the black-box poison signature - a passage that opens with a real query
    for doc in index.sample(20000):
        if first_sentence_matches_frequent_query(doc, retrieval_log, thresh=0.93):
            findings.append(("MEDIUM", f"query-echo passage {doc.id}"))

    # 12: one passage answering the world
    findings += [("MEDIUM", f"retrieval spike {d}: {s:.1%} of queries")
                 for d, s, disp, hot in retrieval_frequency_alarm(retrieval_log, n_queries) if hot]

    return sorted(findings)

Chapter 9: Limits & Connections

Last chapter, and it owes you three things: an honest statement of what this lesson does not cover, a consolidated map of the threats, and the places to go next.

What is genuinely uncertain here

• Every number is from a specific model on a specific dataset. 92% exact recovery is GTR-base on 32-token Natural Questions text. Your encoder, your chunk length and your domain will give different numbers. Cite the mechanism confidently; cite the digits with their conditions attached.
• The attacks improve; your stored vectors do not. A vector written today will still be sitting in a backup when the next inversion paper lands. Design for the attack that has not been published, which in practice means Tier 2 minimization over Tier 4 noise, because minimization does not depend on a difficulty assumption.
• Membership inference remains contested. Chapter 4's honest reading is that many published successes dissolve under a blind baseline, while duplication-driven memorization in small fine-tunes is real. Do not quote a strong claim in either direction.
• Defenses here reduce risk; none of them prove anything. The only measure in this lesson carrying a formal guarantee is differential privacy, and Chapter 7 showed what its guarantee costs when you apply it to the embedding itself.

What is stable is the shape of the reasoning, and it is worth carrying out of here even if every number ages: capacity says there is no counting obstruction; the perfect verifier says search is not the obstruction either; shared geometry says obscurity is not a control; relative ranking says thresholds cannot separate attacker from legitimate content; and no key anywhere says confidentiality reduces entirely to who holds the data.

The threat map, consolidated

Where to go next

This lesson assumed you already knew what an embedding is and how retrieval uses it. If any part felt thin, these fill it in — and several of them are where the defenses actually get implemented:

• Vector Embeddings — what φ is doing in the first place, and why the training objective maximizes exactly the information Chapter 1 recovers.
• Similarity Metrics — the cosine geometry that Chapters 5 and 6 turn into attacks. The α / √(α² + β²) derivation lives here in its friendly form.
• Vector Databases — where post-filtering versus pre-filtering is explained, which is the mechanism behind Chapter 4's cross-tenant leak.
• RAG — the pipeline Chapter 5 attacks, including where in it a provenance field and a robust aggregator would go.
• Text Chunking — chunk length is a privacy parameter as well as a retrieval one, per Chapter 7's minimization tier.
• High-Dimensional Geometry — why an 11-degree rotation is a large move in 768 dimensions, and why isotropic noise behaves so differently for norms than for projections.
• Contrastive Learning — the objective whose duplicate-sensitivity drives Chapter 4's memorization.
• AI Safety Guardrails — the answer-side controls that sit downstream of everything here.
• Embedding Benchmarks — and the reason Chapter 6's backdoor is invisible to all of them.

References

1. Morris, Kuleshov, Shmatikov, Rush. “Text Embeddings Reveal (Almost) As Much As Text.” EMNLP, 2023. arXiv:2310.06816
2. Jha, Zhang, Shmatikov, Morris. “Harnessing the Universal Geometry of Embeddings.” 2025. arXiv:2505.12540
3. Huh, Cheung, Wang, Isola. “The Platonic Representation Hypothesis.” ICML, 2024. arXiv:2405.07987
4. Song, Raghunathan. “Information Leakage in Embedding Models.” ACM CCS, 2020. arXiv:2004.00053
5. Zou, Geng, Wang, Jia. “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models.” USENIX Security, 2025. arXiv:2402.07867
6. Zhong, Huang, Liu, Chen, Chen. “Poisoning Retrieval Corpora by Injecting Adversarial Passages.” EMNLP, 2023. arXiv:2310.19156
7. Jia, Liu, Gong. “BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning.” IEEE S&P, 2022. arXiv:2108.00352
8. Xue et al. “BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models.” 2024. arXiv:2406.00083
9. Carlini, Chien, Nasr, Song, Terzis, Tramèr. “Membership Inference Attacks From First Principles.” IEEE S&P, 2022. arXiv:2112.03570
10. Duan et al. “Do Membership Inference Attacks Work on Large Language Models?” COLM, 2024. arXiv:2402.07841
11. Das, Zhang, Tramèr. “Blind Baselines Beat Membership Inference Attacks for Foundation Models.” 2024. arXiv:2406.16201
12. Shokri, Stronati, Song, Shmatikov. “Membership Inference Attacks Against Machine Learning Models.” IEEE S&P, 2017. arXiv:1610.05820
13. Ebrahimi, Rao, Lowd, Dou. “HotFlip: White-Box Adversarial Examples for Text Classification.” ACL, 2018. arXiv:1712.06751
14. Conneau, Lample, Ranzato, Denoyer, Jégou. “Word Translation Without Parallel Data.” ICLR, 2018. arXiv:1710.04087
15. Xiang, Wu, Zhong, Wagner, Chen, Mittal. “Certifiably Robust RAG against Retrieval Corruption.” 2024. arXiv:2405.15556
16. Abadi et al. “Deep Learning with Differential Privacy.” ACM CCS, 2016. arXiv:1607.00133
17. Feyisetan, Balle, Drake, Diethe. “Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations.” WSDM, 2020. arXiv:1910.08902
18. Henzinger, Dauterman, Corrigan-Gibbs, Zeldovich. “Private Web Search with Tiptoe.” SOSP, 2023.