What if an LLM could manage its own memory like an OS manages virtual memory? Page data in and out of context, search its own history, and remember across sessions — all self-directed.
You are chatting with an AI assistant about your weekend plans. You mention your boyfriend James, your love of Six Flags, your birthday on February 7th. The conversation is warm and natural. Then you close the tab and come back a week later.
The assistant has no idea who you are. James? Six Flags? Never heard of them. Every session starts from zero.
This is the context window problem. Every LLM has a fixed-size input buffer — the context window. GPT-4 at launch: 8,192 tokens. That is roughly 60 back-and-forth messages. A single long conversation easily overflows it. A 200-page legal document does not even come close to fitting.
Scaling context windows is expensive: self-attention is quadratic in sequence length. Doubling context quadruples compute. And even if you could afford it, research shows that models become worse at using information as context grows — they attend well to the beginning and end, but forget the middle.
So we have two failures: conversations that exceed the window are truncated (information lost forever), and documents that exceed the window are impossible to process without chopping them into pieces and losing cross-section coherence.
Watch messages fill the context window. When it overflows, old messages are lost. Click Send to add messages.
Operating systems solved this exact problem fifty years ago. In the 1960s, computers had tiny amounts of physical RAM — maybe 64 kilobytes. Programs needed megabytes. The solution: virtual memory.
The OS creates an illusion of vast memory. Programs think they have a huge, contiguous address space. Behind the scenes, the OS pages data between fast-but-small RAM and slow-but-huge disk. When a program accesses data not currently in RAM, the OS triggers a page fault: it pauses execution, loads the needed data from disk into RAM (possibly evicting something else), and resumes.
This is the core insight of MemGPT: treat the context window as physical memory, provide external storage tiers, and let the LLM itself manage paging. The LLM is not a passive recipient of context — it is the memory manager. It decides what stays in context, what gets evicted, and what gets loaded back in.
The program running "on top" of this virtual memory system? The user's conversation, or a document analysis task, or any other application that needs more context than physically fits.
MemGPT does not just have "context" and "not context." It builds a three-tier hierarchy, each with different speed, capacity, and access patterns. This is directly analogous to the CPU cache → RAM → disk hierarchy in a computer.
This is what the LLM actually sees during inference. Everything in main context is "in-context" — the LLM can attend to it directly. Main context itself is split into three contiguous sections:
A database of all past messages. Every message that enters the FIFO queue is also written to recall storage. When messages are evicted from the queue, they are not lost — they are searchable via recall storage. The LLM can query this with conversation_search(query) to find relevant past messages and bring them back into context.
A read/write database for arbitrary-length text. Documents, notes, long-term facts — anything the LLM wants to remember beyond the conversation. Backed by a vector database (PostgreSQL + pgvector) with embedding-based search. The LLM can insert data with archival_memory_insert(content) and search with archival_memory_search(query).
Click each tier to see its properties. Data flows between tiers via LLM function calls.
Here is the magic. The LLM is not a passive consumer of context — it is an active memory manager. Through function calls, the LLM decides when to save information, when to search for past data, and when to evict content from its working context.
Imagine a multi-session chat. The user mentions their birthday is February 7th. Here is what happens inside MemGPT:
core_memory_append("human", "Birthday is February 7")Now a retrieval example. The user asks "Remember when we talked about Six Flags?" That conversation is long gone from the FIFO queue. But the LLM knows to search:
conversation_search("six flags")When the FIFO queue fills to about 70% of the context window, the queue manager injects a system message: "Warning: memory pressure. Consider saving important information." This is the OS sending an interrupt to the running process. The LLM responds by saving critical data to working context or archival storage before eviction occurs.
At 100% capacity, the queue manager forcibly evicts messages (typically 50% of the queue), generates a new recursive summary, and frees space. Evicted messages remain in recall storage, searchable but no longer directly visible.
Watch the LLM manage its own memory. Click Step to advance through a conversation where the LLM saves, searches, and evicts data.
MemGPT exposes a precise set of functions the LLM can call. These are the "system calls" of our LLM operating system. Each one moves data between memory tiers.
| Function | Effect |
|---|---|
core_memory_append(section, content) | Adds text to the working context block. section is "human" or "persona" — facts about the user or the agent's own identity. |
core_memory_replace(section, old, new) | Finds and replaces text in working context. Used to update stale facts: "Boyfriend named James" → "Ex-boyfriend named James" |
| Function | Effect |
|---|---|
conversation_search(query, page) | Searches past messages by text match. Returns paginated results. Each result includes timestamp and content. |
conversation_search_date(start, end, page) | Searches past messages by date range. Useful for "what did we talk about last week?" |
| Function | Effect |
|---|---|
archival_memory_insert(content) | Writes arbitrary text to the archival database. Embedding is computed for vector search. |
archival_memory_search(query, page) | Semantic search over archival storage via cosine similarity on embeddings. Returns paginated results. |
| Function | Effect |
|---|---|
send_message(message) | Sends a visible response to the user. Without this call, inner monologue and function calls remain invisible. |
request_heartbeat=true | A flag on any function call that says "run me again immediately after this function returns." This enables function chaining — multiple retrieval steps before responding. |
request_heartbeat=true so the LLM runs again immediately to process results and decide whether to search more or respond. Without chaining, the LLM would respond after a single search — possibly before finding the answer.page=2, page=3, etc. to see more. This mirrors how an OS pages data in fixed-size chunks.request_heartbeat=true flag do, and why is it essential for document analysis?A 200-page legal document contains roughly 100,000 tokens. GPT-4's context window at launch was 8,192. You cannot just paste it in. Even GPT-4 Turbo at 128K tokens struggles with documents that exceed its window, and attention degrades for middle content.
Traditional approaches chunk the document and use a retriever (embedding similarity) to select relevant chunks. But this is one-shot: the retriever picks chunks once, and if the relevant piece is not in the top-K, it is lost. The LLM never sees it.
The entire document is loaded into archival storage. The LLM receives the user's question and then iteratively queries archival storage:
archival_memory_search("nobel physics"). Gets 10 of 124 results (page 1/13).archival_memory_search("nobel physics", page=2)send_message("Wilhelm Conrad Rontgen").The paper introduces a harder task: nested KV lookup. A database of UUID key-value pairs where values may themselves be keys. Finding the final answer requires multi-hop retrieval: look up key A, get value B, discover B is also a key, look up B, get value C, and so on.
Fixed-context models collapse at 2-3 nesting levels. MemGPT with GPT-4 handles arbitrary nesting because it can repeatedly query archival storage, chaining lookups until it reaches a terminal value.
The second evaluation domain: conversational agents that remember. Not just within a single session, but across sessions spanning days, weeks, months. A virtual companion that knows your birthday, your partner's name, your favorite amusement park — and can recall any of it at any time.
The paper defines two properties a long-term conversational agent must satisfy:
Working context acts as a persistent user profile. Across sessions, the LLM accumulates facts: "Birthday: Feb 7. Boyfriend: James. Loves Six Flags. Met James at Six Flags." These survive indefinitely because working context is not evicted by the queue manager.
When facts change, the LLM updates them. The user says "james and i broke up." The LLM calls core_memory_replace("human", "Boyfriend named James", "Ex-boyfriend named James"). The profile evolves in real time.
For deeper recall, the LLM searches recall storage. A user asks "remember when we went to Six Flags?" The LLM searches, finds the specific messages, and weaves them into the response with emotional context: "Of course! You mentioned that is where you and James first met."
The paper evaluates MemGPT against fixed-context baselines across both domains. The results are striking.
The agent is asked a specific question about a topic from sessions 1-5 (e.g., "What did Sarah say her favorite hobby was?"). The response is scored against a gold answer.
| Model | Accuracy | ROUGE-L (R) |
|---|---|---|
| GPT-3.5 Turbo | 38.7% | 0.394 |
| GPT-3.5 + MemGPT | 66.9% | 0.629 |
| GPT-4 | 32.1% | 0.296 |
| GPT-4 + MemGPT | 92.5% | 0.814 |
| GPT-4 Turbo | 35.3% | 0.359 |
| GPT-4 Turbo + MemGPT | 93.4% | 0.827 |
MemGPT crafts session openers that are compared to human-written gold openers and to persona labels. MemGPT's openers match or exceed human quality:
| Method | SIM-1 | SIM-3 | SIM-Human |
|---|---|---|---|
| Human gold | 0.800 | 0.800 | 1.000 |
| GPT-4 + MemGPT | 0.868 | 0.843 | 0.773 |
| GPT-4 Turbo + MemGPT | 0.857 | 0.828 | 0.767 |
MemGPT's SIM-1 score of 0.868 exceeds the human gold opener's 0.800 — MemGPT covers more persona facts in its openers because it actively retrieves and surfaces them from memory.
On the multi-document question answering task, MemGPT with GPT-4 maintains ~65% accuracy regardless of the number of retrieved documents (up to 200+). Fixed-context baselines peak around 60% with optimal document count and degrade as truncation increases.
MemGPT with GPT-4 is the only method that maintains performance beyond 2 nesting levels, achieving consistent accuracy through 4 levels of nested lookup. All fixed-context baselines drop to 0% by nesting level 3.
Let us push the OS analogy to its limits. MemGPT does not just borrow the idea of virtual memory — it maps almost every component of an operating system onto the LLM architecture.
| OS Concept | MemGPT Equivalent |
|---|---|
| CPU / Processor | The LLM itself (performs inference on main context) |
| Physical RAM | Context window (prompt tokens) |
| Virtual memory | The illusion of infinite context across all memory tiers |
| Page fault | LLM calls a retrieval function to load data into context |
| Page eviction | Queue manager flushes old messages from FIFO queue |
| Disk storage | Archival storage (PostgreSQL + pgvector) |
| Filesystem | Recall storage (message history database) |
| System calls | Memory functions (core_memory_append, archival_memory_search, etc.) |
| Kernel | System instructions (always resident, read-only) |
| Interrupts | Events: user messages, system alerts, memory pressure warnings, timed events |
| Process scheduler | Event-based control flow + function chaining via heartbeat |
| Swap space | Recursive summary of evicted messages |
The analogy is not perfect. In an OS, the memory manager is deterministic — page replacement follows a fixed algorithm (LRU, clock, etc.). In MemGPT, the memory manager is stochastic: the LLM might forget to save important information, or search with a bad query, or page through results and stop too early. The paper observes this: MemGPT with GPT-3.5 often fails at function chaining because the model's function-calling ability is weaker.
Another gap: in an OS, paging is transparent to the application. The program does not know or care that its data is on disk. In MemGPT, the LLM is both the application and the memory manager. It must be explicitly instructed (via the system prompt) about the memory hierarchy and functions. If those instructions are unclear, memory management degrades.
Hover over OS components to see their MemGPT equivalents highlighted.
MemGPT sits at the intersection of several major research threads. Understanding where it fits helps you see the bigger picture.
Retrieval-Augmented Generation (RAG) — The precursor. RAG systems retrieve relevant documents and stuff them into context before inference. MemGPT goes further: retrieval is not a one-shot preprocessing step but an iterative, self-directed process during inference. The LLM decides what to retrieve and when.
CacheBlend (2024) — Addresses context management from the KV-cache perspective: precomputing and blending cached attention states for frequently accessed context. Where MemGPT manages what text is in context, CacheBlend manages what attention states are cached. Complementary approaches.
Claude Code compaction — Production systems like Claude Code face the same context pressure. Their solution: summarize and compress context when it gets too long. MemGPT's recursive summary in the FIFO queue is the same idea. The difference: MemGPT also provides structured archival and recall storage, while Claude Code relies on the model's context window plus tool outputs.
Generative Agents (Park et al., 2023) — Simulated "Sims-like" agents with memory streams, reflection, and planning. They store observations in a memory stream and periodically reflect. MemGPT's archival storage is similar to the memory stream, but MemGPT's retrieval is self-directed via function calls rather than triggered by a fixed schedule.
FLARE (Jiang et al., 2023) — Allows LLMs to decide when to retrieve during generation. MemGPT generalizes this: not just retrieval, but also writing to memory, updating beliefs, and managing multiple storage tiers.
MemGPT represents a philosophical shift: from treating LLMs as stateless functions (input in, output out) to treating them as stateful processes with persistent memory, self-modification, and event-driven control flow. The LLM is not just a model — it is an operating system for intelligence.