Agentic Engineer
at Sierra

Chapter 1: Agent SDK & Orchestration

The Agent SDK is not the LLM. The LLM is a function that takes tokens and returns tokens. The SDK is the orchestrator — the runtime loop that decides when to call the LLM, what context to inject, how to parse the output, which tools to invoke, and when to stop. Think of the LLM as the brain and the SDK as the nervous system: it routes signals, triggers reflexes, and coordinates the body.

At Sierra, every agent instance runs inside a single SDK execution. The SDK manages the agent loop: Observe (gather context) → Think (call LLM) → Act (execute tool) → Observe (get tool result) → repeat until the agent emits a terminal response. This loop is deceptively simple on a whiteboard but brutally complex in production because every step can fail, timeout, or produce unexpected output.

The Runtime Loop

Here is the core abstraction. Every method is a hook point where you inject business logic:

python
class AgentRuntime:
    def __init__(self, config: AgentConfig):
        self.llm = config.llm_client          # LLM provider (OpenAI, Anthropic, etc.)
        self.tools = config.tool_registry      # Dict[str, Callable]
        self.memory = config.memory_store      # Short + long-term memory
        self.guardrails = config.guardrails    # Pre/post-execution checks
        self.max_steps = config.max_steps      # Circuit breaker: prevent infinite loops

    async def run(self, user_msg: str, session: Session) -> Response:
        context = await self._build_context(user_msg, session)

        for step in range(self.max_steps):
            # THINK: call LLM with full context
            llm_output = await self.llm.generate(
                messages=context.messages,
                tools=self._get_available_tools(session),
                temperature=0.1,  # Low temp for action reliability
            )

            # PARSE: is this a tool call or a final response?
            if llm_output.is_tool_call:
                # ACT: validate, execute, append result
                result = await self._execute_tool(
                    llm_output.tool_name,
                    llm_output.tool_args,
                    session,
                )
                context.append_tool_result(result)
            else:
                # TERMINAL: return response to user
                return Response(text=llm_output.text, trace=context.trace)

        # Circuit breaker: too many steps
        return Response(text="I'm having trouble completing this. Let me connect you to support.")

State Management: The Hard Part

The loop above looks clean, but in production you face three brutal realities:

1. Conversation history grows unbounded. A customer might have a 200-turn conversation over 3 days. You cannot send all 200 turns to the LLM (context window limit, cost, latency). The SDK must implement a context window manager that summarizes old turns, preserves critical facts (like "order #12345 was already refunded"), and fits within the token budget.

2. Tool execution is not atomic. If the agent calls "refund_payment" and the network drops before you get the response, did the refund happen? You need idempotency keys, retry logic with exponential backoff, and a tool execution ledger that records what was attempted vs. what succeeded.

3. Multi-turn tool chains create dependency graphs. "Cancel order" requires: lookup_order → check_policy → cancel_order → initiate_refund. If cancel_order fails, you must not call initiate_refund. The SDK needs a lightweight state machine or DAG executor, not just a flat loop.

Design Decisions That Matter

Debugging the SDK

The most common production bugs in an agent SDK:

python
# BUG 1: Infinite loop — LLM keeps calling the same tool
# Root cause: tool returns error, LLM retries with same args
# Fix: track tool call history, inject "you already tried this" after 2 failures

# BUG 2: Context window overflow mid-conversation
# Root cause: tool results are huge (full API responses with metadata)
# Fix: truncate tool results to essential fields before appending to context

tool_result_summary = {
    "status": result["status"],
    "order_id": result["order_id"],
    "amount": result["total"],
    # NOT: full 50KB API response with internal metadata
}

# BUG 3: Race condition — user sends new message while agent is mid-tool-call
# Root cause: no mutex on session state
# Fix: session-level lock with queue for incoming messages

async with session.lock():
    result = await execute_tool(...)
    session.state.append(result)

Frontier: Multi-Agent Orchestration

The next evolution is multi-agent systems where a "supervisor" agent delegates subtasks to specialist agents. The order-cancellation flow becomes: Supervisor → spawns OrderAgent (finds the order) → spawns RefundAgent (handles payment) → spawns RebookAgent (creates new order with discount). Each sub-agent has its own tools, permissions, and context. The supervisor aggregates results and handles inter-agent failures.

This introduces new SDK primitives: agent spawning, message passing between agents, shared state with conflict resolution, and hierarchical circuit breakers (if a sub-agent fails, does the parent retry, compensate, or escalate to human?).

Chapter 2: Reasoning & Action Planning

The SDK orchestrates. But how does the agent decide what to do? This is the reasoning layer — the strategies that transform a vague user request into a concrete sequence of actions. Get this wrong and your agent will hallucinate actions, skip steps, or execute them in the wrong order. Get it right and users will feel like they're talking to someone who genuinely understands their problem.

There are three dominant paradigms for agent reasoning. Each has different latency, cost, and reliability tradeoffs. A staff engineer must know when to use which.

Paradigm 1: ReAct (Reason + Act)

ReAct interleaves reasoning and action in a single stream. The LLM generates a thought ("I need to find the user's order"), then an action (call lookup_order), then observes the result, then thinks again. It's simple, low-latency (one LLM call per step), and works well for straightforward tasks.

python
# ReAct prompt structure (simplified)
REACT_SYSTEM = """You are a helpful agent. For each step:
1. Thought: reason about what to do next
2. Action: call a tool OR respond to the user
3. Observation: the tool result (provided by system)

Repeat until you can give a final answer."""

# LLM output example:
# Thought: The user wants to cancel "that thing from Tuesday."
#          I need to find orders from Tuesday.
# Action: lookup_orders(date="2024-03-12", user_id="u_789")
# Observation: [{"id": "ord_123", "item": "Blue Sweater", "date": "2024-03-12"}]
# Thought: Found one order from Tuesday. I should confirm before canceling.
# Action: respond("I found your Blue Sweater order from Tuesday. Cancel it?")

Weakness: ReAct is greedy — it commits to the first action without considering alternatives. For complex multi-step tasks, it often picks a suboptimal path and gets stuck.

Paradigm 2: Plan-then-Execute

Plan-then-Execute separates thinking from doing. First, the LLM generates a complete plan (a numbered list of steps). Then the SDK executes each step sequentially, checking preconditions before each one. If a step fails, the planner is called again to replan from the current state.

python
async def plan_and_execute(user_msg: str, session: Session) -> Response:
    # PLAN: one LLM call to generate full plan
    plan = await llm.generate(
        system=PLANNER_PROMPT,
        user=f"User request: {user_msg}\nAvailable tools: {tool_descriptions}",
        response_format=PlanSchema,  # Structured output: list of steps
    )

    results = []
    for step in plan.steps:
        # CHECK: should we still execute this step?
        if step.precondition and not check_precondition(step, results):
            # REPLAN: conditions changed, ask LLM to adapt
            plan = await replan(plan, results, step)
            continue

        # EXECUTE: run the tool
        result = await execute_tool(step.tool, step.args, session)
        results.append(result)

        if result.failed and step.is_critical:
            return await handle_critical_failure(plan, results, session)

    # SYNTHESIZE: generate user-facing response from results
    return await synthesize_response(user_msg, results)

Strength: Observable, debuggable, and allows precondition checks. Weakness: Higher latency (planning call + execution calls), and the plan can be stale if the world changes mid-execution.

Paradigm 3: Tree-of-Thought

Tree-of-Thought (ToT) explores multiple reasoning paths in parallel, evaluates each, and picks the best. It's expensive (multiple LLM calls per decision point) but powerful for ambiguous requests where the "right" action depends on information you don't have yet.

In practice, ToT is rarely used for every turn. Instead, it's triggered selectively: when the agent detects ambiguity (user said "cancel my order" but has 3 orders), it spawns parallel reasoning branches ("ask which order" vs "cancel most recent" vs "cancel all") and evaluates which is safest.

Making Reasoning Observable

The biggest operational challenge with agent reasoning is debuggability. When an agent makes a wrong decision, you need to understand why. This means every reasoning step must be logged with:

python
# Structured reasoning output with observability
class ReasoningStep(BaseModel):
    thought: str
    confidence: float = Field(ge=0, le=1)
    action: Optional[ToolCall] = None
    response: Optional[str] = None
    alternatives: list[str] = []

    def should_escalate(self) -> bool:
        return self.confidence < 0.7 and self.action is not None

    def should_confirm(self) -> bool:
        # High-stakes actions need user confirmation regardless of confidence
        return self.action and self.action.tool_name in IRREVERSIBLE_TOOLS

Frontier: Learned Reasoning Strategies

The current state of the art is moving from hand-crafted reasoning prompts to learned strategies. Instead of hard-coding "use ReAct for simple queries," you train a lightweight classifier on historical conversation data: given the user message, conversation history, and available tools, predict which reasoning strategy will succeed. This classifier adds ~5 ms of latency but can improve action correctness by 15-20% by routing ambiguous queries to more careful strategies.

Even more frontier: process reward models (PRMs) that score intermediate reasoning steps, not just final outcomes. You train the PRM on human annotations of "good reasoning" and use it to prune bad branches in Tree-of-Thought before they waste LLM calls.

Chapter 3: Retrieval & Grounding

Your agent is only as good as the information it has access to. An LLM without retrieval is like a brilliant consultant who hasn't read the client's documentation — impressive vocabulary, zero useful specifics. Retrieval-Augmented Generation (RAG) is the mechanism that grounds the agent in factual, up-to-date, customer-specific knowledge.

The RAG pipeline looks simple on a diagram: query → embed → search → rerank → inject into context. But at Sierra's scale (hundreds of enterprise customers, each with 10K-2M documents, serving 50K concurrent conversations), every step hides engineering complexity that separates a demo from a production system.

The Full Pipeline

The Grounding Problem

Retrieval without grounding is dangerous. If the retrieval system returns irrelevant documents (because the query was ambiguous, or the document set doesn't cover the topic), the LLM will hallucinate — it'll generate a plausible-sounding answer from its training data rather than admitting "I don't know."

The grounding rule at production scale:

python
class GroundingChecker:
    def __init__(self, confidence_threshold: float = 0.75):
        self.threshold = confidence_threshold

    def check(self, query: str, retrieved_docs: list[Doc], rerank_scores: list[float]) -> GroundingResult:
        # Signal 1: Are the top docs actually relevant?
        top_score = rerank_scores[0] if rerank_scores else 0.0

        # Signal 2: Is there a large gap between #1 and #2? (confidence)
        score_gap = rerank_scores[0] - rerank_scores[1] if len(rerank_scores) > 1 else 0.0

        # Signal 3: Do multiple docs agree? (consistency)
        top_3_similar = self._check_consistency(retrieved_docs[:3])

        confidence = (
            0.5 * min(top_score, 1.0) +
            0.2 * min(score_gap * 2, 1.0) +
            0.3 * top_3_similar
        )

        if confidence < self.threshold:
            return GroundingResult(
                grounded=False,
                action="SAY_DONT_KNOW",
                reason=f"Low retrieval confidence ({confidence:.2f})",
            )
        return GroundingResult(grounded=True, docs=retrieved_docs[:5])

Multi-Tenant Retrieval Architecture

At Sierra, each enterprise customer has their own knowledge base. You cannot mix Customer A's internal pricing docs with Customer B's. This creates a multi-tenant vector search challenge:

python
# Multi-tenant retrieval with partition routing
class TenantRouter:
    def get_index(self, tenant_id: str) -> VectorIndex:
        # Large tenants get dedicated indexes
        if tenant_id in self.dedicated_indexes:
            return self.dedicated_indexes[tenant_id]

        # Small tenants share a partitioned index
        partition = self._hash_to_partition(tenant_id)
        return self.shared_indexes[partition]

    async def search(self, tenant_id: str, query_vec: list[float], k: int = 20) -> list[Doc]:
        index = self.get_index(tenant_id)
        # Always filter by tenant_id even in dedicated index (defense in depth)
        results = await index.search(
            vector=query_vec,
            k=k,
            filter={"tenant_id": tenant_id},  # Belt AND suspenders
        )
        return results

The Stale Index Problem

Remember the PagerDuty alert from Chapter 0? A customer uploaded 2M documents and p99 latency spiked because the ANN index was rebuilding. This is the stale-while-reindex pattern:

python
# Blue-green index deployment
class IndexManager:
    def __init__(self):
        self.active_index = None      # Currently serving queries
        self.building_index = None    # Being rebuilt in background

    async def ingest_documents(self, docs: list[Doc]):
        # 1. Add to the building index (background)
        await self.building_index.add(docs)

        # 2. Queries continue hitting active_index (stale but fast)
        # 3. When building_index is ready, atomic swap
        if self.building_index.ready:
            self.active_index, self.building_index = self.building_index, self._new_index()

Chapter 4: Evaluation & Quality

You cannot improve what you cannot measure. And measuring AI agent quality is fundamentally harder than measuring a traditional software system. A REST API is either correct or broken — you write unit tests and move on. An agent can be partially correct, technically correct but unhelpful, helpful but unsafe, or correct on Tuesday but wrong on Thursday because the underlying LLM was updated. Evaluation is the discipline that gives you ground truth in this ambiguous world.

The Five Quality Dimensions

Automated Eval Pipeline

The eval pipeline runs automatically on every prompt change, model update, or system configuration change. It catches regressions before they hit production. Here's the architecture:

python
class EvalPipeline:
    def __init__(self):
        self.test_suites = {
            "correctness": CorrectnessEval(n=500),    # 500 golden conversations
            "safety": SafetyEval(n=200),               # 200 adversarial prompts
            "latency": LatencyEval(percentiles=[50, 95, 99]),
            "cost": CostEval(budget_per_conv=0.15),
        }
        self.judge = LLMJudge(model="gpt-4o")  # For open-ended quality

    async def run(self, agent_config: AgentConfig) -> EvalReport:
        results = {}
        for name, suite in self.test_suites.items():
            results[name] = await suite.evaluate(agent_config)

        # Compare against baseline (current production config)
        baseline = await self._load_baseline()
        regressions = self._detect_regressions(results, baseline)

        if regressions:
            await self._alert_team(regressions)
            return EvalReport(passed=False, regressions=regressions)

        return EvalReport(passed=True, results=results)

LLM-as-Judge: Power and Pitfalls

For open-ended quality (helpfulness, tone, clarity), you can't write deterministic tests. Instead, you use a stronger LLM to judge the agent's responses. This is LLM-as-Judge: you provide the conversation, the agent's response, and a rubric, and the judge LLM outputs a score.

python
JUDGE_PROMPT = """Rate this agent response on a 1-5 scale for HELPFULNESS.

Context: {conversation_history}
Agent response: {agent_response}
User's actual problem: {ground_truth_intent}

Rubric:
5 = Fully resolves the user's problem with clear explanation
4 = Resolves the problem but explanation could be clearer
3 = Partially resolves; user would need follow-up
2 = Technically correct but misses the user's actual intent
1 = Wrong, unhelpful, or harmful

Score (1-5):
Reasoning:"""

LLM-as-Judge limitations:

1. Position bias. Judges prefer the first option in A/B comparisons. Fix: randomize order and run both orderings.

2. Verbosity bias. Judges rate longer responses higher even when they're less helpful. Fix: include length normalization in the rubric ("conciseness is valued").

3. Self-preference. GPT-4 rates GPT-4 outputs higher than Claude outputs (and vice versa). Fix: use a different model family for judging than for generation.

4. Rubric sensitivity. Small changes in rubric wording can swing scores by 0.5+ points. Fix: A/B test your rubrics against human labels before trusting them.

Regression Detection

python
def detect_regression(
    current: EvalResult,
    baseline: EvalResult,
    threshold: float = 0.02,  # 2% drop = regression
) -> Optional[Regression]:
    # Statistical significance test (not just raw difference)
    from scipy.stats import proportions_ztest

    stat, p_value = proportions_ztest(
        count=[current.successes, baseline.successes],
        nobs=[current.total, baseline.total],
        alternative="smaller",  # Is current WORSE than baseline?
    )

    if p_value < 0.05 and (baseline.rate - current.rate) > threshold:
        return Regression(
            metric=current.metric_name,
            baseline_rate=baseline.rate,
            current_rate=current.rate,
            p_value=p_value,
            sample_failures=current.get_failures(n=10),  # For debugging
        )
    return None

Chapter 5: Real-time Data Pipelines

Every conversation generates events. Every tool call, every LLM response, every user click, every latency measurement — all of it must flow from the agent runtime into storage, analytics, and eval systems in near-real-time. At 50,000 concurrent conversations generating 10+ events per second each, you're looking at 500K events/second sustained throughput. This is not a batch job. This is a streaming data platform.

The challenge isn't just throughput — it's the guarantees. If you lose an event, you might miss a safety violation. If you deliver an event twice, your analytics double-count and your A/B test results are wrong. You need exactly-once semantics in a distributed system where every component can fail.

The Event Schema

python
from dataclasses import dataclass
from datetime import datetime
from enum import Enum

class EventType(Enum):
    MSG_RECEIVED = "msg_received"
    LLM_CALL_START = "llm_call_start"
    LLM_CALL_END = "llm_call_end"
    TOOL_CALL_START = "tool_call_start"
    TOOL_CALL_END = "tool_call_end"
    RETRIEVAL_QUERY = "retrieval_query"
    RETRIEVAL_RESULT = "retrieval_result"
    RESPONSE_SENT = "response_sent"
    ERROR = "error"
    GUARDRAIL_TRIGGERED = "guardrail_triggered"

@dataclass
class AgentEvent:
    event_id: str                # UUID, idempotency key
    event_type: EventType
    timestamp: datetime          # Server-side, nanosecond precision
    conversation_id: str         # Groups events in a conversation
    tenant_id: str               # Which customer
    turn_id: str                 # Which turn in the conversation
    payload: dict                # Type-specific data (tokens, latency, etc.)
    trace_id: str                # Distributed tracing correlation

Streaming Architecture

The standard stack for this scale:

Exactly-Once Delivery

The hardest guarantee in distributed streaming. Here's how it works in practice:

python
# Producer side: idempotent writes to Kafka
producer = KafkaProducer(
    bootstrap_servers=["kafka-1:9092", "kafka-2:9092"],
    acks="all",                    # Wait for all replicas to confirm
    enable_idempotence=True,       # Kafka deduplicates by producer ID + seq num
    max_in_flight_requests=5,     # Allow pipeline but maintain ordering
    retries=3,                     # Retry on transient network failures
)

# Consumer side: commit offset AFTER successful processing
async def consume_and_process(consumer, processor, sink):
    async for batch in consumer.poll(max_records=1000, timeout_ms=100):
        # Process batch (enrich, transform, aggregate)
        results = await processor.process(batch)

        # Write to sink in a transaction
        async with sink.transaction() as txn:
            await txn.write(results)
            # Commit Kafka offset INSIDE the same transaction
            # This is "exactly-once" — if either fails, both roll back
            await txn.commit_offsets(batch.offsets)

Backpressure: The Silent Killer

What happens when the stream processor can't keep up? If the consumer falls behind the producer, one of three things happens: (1) messages pile up in Kafka (lag increases), (2) the consumer OOMs, or (3) the producer blocks. Backpressure is the mechanism that prevents catastrophe.

python
# Backpressure-aware producer with overflow handling
class BackpressureProducer:
    def __init__(self, producer, buffer_limit: int = 10000):
        self.producer = producer
        self.buffer = asyncio.Queue(maxsize=buffer_limit)
        self.overflow_count = 0

    async def emit(self, event: AgentEvent):
        try:
            self.buffer.put_nowait(event)
        except asyncio.QueueFull:
            # CRITICAL DECISION: drop, sample, or block?
            self.overflow_count += 1
            if event.event_type == EventType.GUARDRAIL_TRIGGERED:
                # Safety events: NEVER drop. Block the agent instead.
                await self.buffer.put(event)  # Blocks until space
            elif self.overflow_count % 100 == 0:
                # Sample: keep 1% of overflow for monitoring
                self.buffer.get_nowait()  # Drop oldest non-critical
                await self.buffer.put(event)
            # else: silently drop (it's a latency measurement, not critical)

Partitioning Strategy

Kafka topics are partitioned. The partition key determines which events land on the same partition (and thus maintain ordering). Common choices:

Partition by conversation_id: All events from one conversation are ordered. Good for building conversation replays. Bad for hot-partition risk (one very active conversation dominates a partition).

Partition by tenant_id: All events from one customer are together. Good for tenant-level analytics. Bad for large tenants (one enterprise customer with 10K concurrent conversations overwhelms a partition).

Hybrid: Hash of tenant_id + conversation_id mod N. Spreads load but loses strict per-conversation ordering. Fix: use Flink's session windows to reconstruct order from timestamps.

Chapter 6: Data Lakehouse

Your streaming pipeline produces 500K events/second. Where do they land? A traditional data warehouse (Snowflake, BigQuery) works for batch analytics but struggles with the write-heavy, schema-evolving, partially-structured nature of agent events. A raw data lake (Parquet on S3) is cheap but gives you no ACID guarantees, no time travel, no efficient updates. The lakehouse gives you both: data lake economics with data warehouse reliability.

Bronze-Silver-Gold Architecture

The medallion architecture organizes data into three quality tiers. Each tier adds guarantees and removes noise:

python
# Bronze → Silver transformation
class BronzeToSilver:
    def transform(self, bronze_batch: list[RawEvent]) -> list[SilverEvent]:
        results = []
        for event in bronze_batch:
            # 1. Deduplicate (same event_id seen twice = retry artifact)
            if self.seen_ids.contains(event.event_id):
                continue
            self.seen_ids.add(event.event_id)

            # 2. Schema validation (reject malformed events)
            if not self.validate_schema(event):
                self.dead_letter.send(event)
                continue

            # 3. PII masking (emails, phone numbers, SSNs)
            payload = self.pii_masker.mask(event.payload)

            # 4. Enrichment (add tenant metadata, agent version, etc.)
            enriched = self.enrich(event, payload)
            results.append(enriched)

        return results

Apache Iceberg: Why It Matters

Apache Iceberg is the table format that makes the lakehouse work. It stores data as Parquet files on S3 but adds a metadata layer that gives you:

1. Time travel. Query the table as it existed at any point in time. "What did our eval results look like before yesterday's prompt change?" One SQL query with a timestamp filter.

2. Schema evolution. Add columns without rewriting existing data. When you add a new event field (say, "reasoning_tokens_used"), old data has nulls, new data has values. No migration downtime.

3. Partition pruning. If you query "all events for tenant_X on 2024-03-12," Iceberg reads only the Parquet files for that tenant and date. On a table with 10 billion rows, this means scanning 100K rows instead of 10B. Query time drops from minutes to milliseconds.

4. ACID transactions. Concurrent writers (your Flink jobs) can safely write to the same table. Iceberg uses optimistic concurrency: each write creates a new metadata snapshot. If two writes conflict, one retries.

sql
-- Query Silver layer: conversations with retrieval failures yesterday
SELECT
  conversation_id,
  tenant_id,
  COUNT(*) AS retrieval_failures,
  MAX(payload.latency_ms) AS max_retrieval_latency
FROM silver.agent_events
WHERE event_type = 'retrieval_result'
  AND payload.grounded = false
  AND dt = '2024-03-12'  -- Partition pruning: only reads one day's files
GROUP BY 1, 2
HAVING retrieval_failures > 5
ORDER BY retrieval_failures DESC
LIMIT 50;

Query Optimization Tricks

At petabyte scale, naive queries cost hundreds of dollars and take hours. Here are the optimizations a staff data engineer knows:

Chapter 7: Long-term Memory & Personalization

A customer calls on Monday, explains a complex issue for 15 minutes, and gets a partial resolution. They call back on Wednesday expecting the agent to remember everything. Without memory, the agent says "How can I help you today?" and the customer has to repeat themselves. With memory, the agent says "I see you called Monday about the billing discrepancy on order #45123. We were waiting on the finance team's review. Let me check if that's resolved."

That's the difference between a tool and a relationship. Memory transforms agents from stateless functions into entities that build understanding over time.

The Three Memory Horizons

python
class MemoryManager:
    def __init__(self, user_id: str):
        self.short_term = ContextWindow()          # Current conv messages
        self.medium_term = SessionStore(user_id)    # Recent session summaries
        self.long_term = ProfileStore(user_id)      # Persistent user profile

    async def build_context(self, new_message: str) -> list[Message]:
        # 1. Retrieve relevant long-term memories
        profile = await self.long_term.get_profile()
        relevant_history = await self.long_term.search(
            query=new_message, k=3, decay_weight=0.9
        )

        # 2. Get medium-term session context
        recent_sessions = await self.medium_term.get_recent(n=3)

        # 3. Assemble context within token budget
        context = []
        context.append(SystemMessage(f"User profile: {profile.summary}"))

        if relevant_history:
            context.append(SystemMessage(
                f"Relevant past interactions:\n{self._format(relevant_history)}"
            ))

        if recent_sessions:
            context.append(SystemMessage(
                f"Recent session notes:\n{self._format(recent_sessions)}"
            ))

        # 4. Add short-term (current conversation)
        context.extend(self.short_term.messages)
        context.append(UserMessage(new_message))

        return self._trim_to_budget(context, max_tokens=8000)

Memory Retrieval with Decay Scoring

Not all memories are equally relevant. A conversation from yesterday is more relevant than one from 6 months ago (unless it's about the same issue). The decay-weighted retrieval combines semantic similarity with recency:

python
import math
from datetime import datetime, timedelta

def decay_score(
    similarity: float,       # Cosine similarity [0, 1]
    memory_age: timedelta,   # How old is this memory?
    alpha: float = 0.7,      # Weight on similarity vs recency
    half_life_days: float = 7.0,  # After 7 days, recency score halves
) -> float:
    # Exponential decay: recent memories score higher
    age_days = memory_age.total_seconds() / 86400
    recency = math.exp(-0.693 * age_days / half_life_days)  # 0.693 = ln(2)

    return alpha * similarity + (1 - alpha) * recency

Why half-life = 7 days? Because most customer support issues resolve within a week. If someone called about a refund 3 days ago, that memory is highly relevant. If they called about a refund 3 months ago, it's probably a different issue. The half-life is tunable per use case — in enterprise sales, you might set it to 30 days because deals take months.

Memory Write: What to Remember

Writing to long-term memory is harder than reading. You can't store every utterance — that's just a conversation log. You need to extract facts that will be useful in future conversations:

python
# After each conversation, extract memories
MEMORY_EXTRACTION_PROMPT = """Analyze this conversation and extract facts that would be
useful if this user contacts us again. Focus on:
1. Preferences expressed (communication style, product preferences)
2. Unresolved issues (problems not fully fixed)
3. Key facts mentioned (account details, constraints, context)
4. Emotional state and satisfaction level

Return as JSON:
{
  "preferences": [...],
  "unresolved": [...],
  "facts": [...],
  "satisfaction": "high|medium|low"
}"""

async def write_memory(conversation: Conversation) -> None:
    # Use LLM to extract structured facts
    extracted = await llm.generate(
        system=MEMORY_EXTRACTION_PROMPT,
        user=conversation.to_text(),
        response_format=MemoryExtraction,
    )

    # Embed each fact for future semantic search
    for fact in extracted.all_facts():
        embedding = await embed(fact.text)
        await memory_store.upsert(
            user_id=conversation.user_id,
            text=fact.text,
            embedding=embedding,
            category=fact.category,
            timestamp=datetime.now(),
        )

Privacy and Forgetting

Memory creates a data retention obligation. Users must be able to request deletion ("forget everything about me"). GDPR's "right to be forgotten" is not optional. Your memory system needs:

1. User-level delete: Remove all memories, embeddings, and profile data for a user ID. Must cascade to all stores (vector index, KV store, Postgres, Iceberg audit trail).

2. Selective forget: "Forget my phone number" without forgetting everything else. Requires structured memory so you can target specific facts.

3. Automatic expiry: Memories older than N months are automatically archived/deleted unless they're linked to an active issue.

Chapter 8: Conversational Analysis & Clustering

You have 500,000 conversations per week. Somewhere in that data are patterns you need to find: emerging customer issues, new failure modes your agent doesn't handle, topics where your retrieval consistently fails, phrasing patterns that confuse the LLM. But you can't read 500K transcripts. You need automated pattern discovery.

The pipeline: embed conversations → reduce dimensions → cluster → label clusters → surface anomalies. This is unsupervised ML applied to operational intelligence. It's how you go from "something feels wrong" to "17% of conversations about shipping now fail because our carrier changed their API response format last Tuesday."

The Embedding Pipeline

python
import numpy as np
from sentence_transformers import SentenceTransformer

class ConversationEmbedder:
    def __init__(self):
        self.model = SentenceTransformer("BAAI/bge-large-en-v1.5")

    def embed_conversation(self, conversation: Conversation) -> np.ndarray:
        # Strategy: embed a summary, not the full transcript
        # Full transcripts are too long and noisy for clustering
        summary = self._summarize(conversation)
        return self.model.encode(summary, normalize_embeddings=True)

    def _summarize(self, conv: Conversation) -> str:
        # Extract: user intent + agent actions + outcome
        return f"""User intent: {conv.extracted_intent}
Actions taken: {', '.join(conv.tool_calls)}
Outcome: {conv.resolution_status}
Key topics: {', '.join(conv.topics)}"""

UMAP + HDBSCAN: The Modern Clustering Stack

Raw embeddings are 768-1024 dimensional. You can't visualize or cluster them directly (curse of dimensionality). The standard approach:

1. UMAP (Uniform Manifold Approximation and Projection) reduces dimensions from 768 to 2-50 while preserving local structure. Unlike t-SNE, UMAP preserves global distances reasonably well, making clusters interpretable.

2. HDBSCAN (Hierarchical Density-Based Spatial Clustering) finds clusters without specifying k. It handles noise (conversations that don't fit any cluster), finds clusters of varying sizes, and gives you a "confidence" score for each assignment.

python
import umap
import hdbscan

def cluster_conversations(embeddings: np.ndarray, min_cluster_size: int = 50):
    # Step 1: Reduce from 768D to 50D (for clustering, not viz)
    reducer = umap.UMAP(
        n_components=50,        # High enough for clustering accuracy
        n_neighbors=30,         # Balance local vs global structure
        min_dist=0.0,           # Pack points tight for better clusters
        metric="cosine",        # Match embedding space metric
    )
    reduced = reducer.fit_transform(embeddings)

    # Step 2: Cluster with HDBSCAN
    clusterer = hdbscan.HDBSCAN(
        min_cluster_size=min_cluster_size,  # Minimum conversations per cluster
        min_samples=10,                     # Core point density requirement
        metric="euclidean",                 # On UMAP output
        cluster_selection_method="eom",     # Excess of Mass for variable-size clusters
    )
    labels = clusterer.fit_predict(reduced)

    # Step 3: For visualization, project to 2D
    viz_reducer = umap.UMAP(n_components=2, n_neighbors=30, min_dist=0.1)
    coords_2d = viz_reducer.fit_transform(embeddings)

    return labels, coords_2d, clusterer.probabilities_

Labeling Clusters Automatically

HDBSCAN gives you cluster IDs (0, 1, 2, ...) but not human-readable labels. You need a labeling step that tells the insights team "Cluster 7 = shipping delay complaints" not "Cluster 7 = 2,341 conversations."

python
async def label_cluster(cluster_conversations: list[Conversation]) -> str:
    # Sample 10-20 conversations from the cluster
    sample = random.sample(cluster_conversations, min(15, len(cluster_conversations)))
    summaries = [c.summary for c in sample]

    label = await llm.generate(
        system="You are analyzing a cluster of customer conversations. "
               "Based on these examples, provide a short label (3-5 words) "
               "and a one-sentence description of what unifies them.",
        user="\n---\n".join(summaries),
        response_format=ClusterLabel,
    )
    return label  # e.g., "Shipping delay complaints — West Coast"

Detecting New Failure Modes

The real power of clustering is anomaly detection over time. If a new cluster appears this week that didn't exist last week, something changed. Maybe a new product launched, maybe an API broke, maybe a competitor ran a campaign that's driving confused users to your agent.

python
def detect_emerging_clusters(
    current_week_labels: np.ndarray,
    previous_week_labels: np.ndarray,
    current_week_embeddings: np.ndarray,
    threshold: float = 0.1,  # Cluster is "new" if <10% overlap with last week
) -> list[EmergingCluster]:
    emerging = []

    for cluster_id in set(current_week_labels) - {-1}:  # -1 = noise
        mask = current_week_labels == cluster_id
        cluster_embeddings = current_week_embeddings[mask]

        # How many of these conversations would have clustered together last week?
        overlap = compute_temporal_overlap(cluster_embeddings, previous_week_labels)

        if overlap < threshold:
            emerging.append(EmergingCluster(
                id=cluster_id,
                size=mask.sum(),
                overlap_with_previous=overlap,
                sample_conversations=get_samples(mask, n=10),
            ))

    return sorted(emerging, key=lambda c: c.size, reverse=True)

Chapter 9: A/B Testing & Experimentation

You've built a new reasoning strategy. The eval pipeline says it improves correctness by 3%. Should you ship it? The eval pipeline runs on 500 golden test cases. Production serves 50,000 diverse conversations per day. Maybe that 3% improvement only holds for simple queries. Maybe it introduces a regression on complex multi-step tasks that your test set doesn't cover. The only way to know is to test in production: A/B testing.

A/B testing for AI agents is fundamentally different from testing a button color or a landing page. A conversation is not a single event — it's a multi-turn interaction where quality compounds. A bad first response poisons the entire conversation. The metrics are noisy (LLM outputs are non-deterministic). And the stakes are high (shipping a regression means real customers get wrong answers to real problems).

Traffic Splitting

python
import hashlib

class ExperimentRouter:
    def __init__(self, experiments: list[Experiment]):
        self.experiments = experiments

    def assign_variant(self, user_id: str, experiment_id: str) -> str:
        # Deterministic assignment: same user always gets same variant
        # This prevents "flickering" between variants across sessions
        hash_input = f"{user_id}:{experiment_id}"
        hash_val = int(hashlib.sha256(hash_input.encode()).hexdigest(), 16)
        bucket = hash_val % 1000  # 1000 buckets for fine-grained splits

        experiment = self.get_experiment(experiment_id)
        # 50/50 split: buckets 0-499 = control, 500-999 = treatment
        if bucket < experiment.control_size * 1000:
            return "control"
        return "treatment"

    def get_config(self, user_id: str) -> AgentConfig:
        # Build agent config by layering all active experiment assignments
        config = AgentConfig.default()
        for exp in self.experiments:
            if exp.is_active:
                variant = self.assign_variant(user_id, exp.id)
                config = exp.apply_variant(config, variant)
        return config

Statistical Significance: When to Call It

The #1 mistake in A/B testing is peeking: checking results daily and stopping when you see a significant result. This inflates your false positive rate because you're running multiple hypothesis tests without correction. If you check every day for 14 days, your true p-value is not 0.05 — it's closer to 0.25.

Where δ is the minimum detectable effect (MDE), p is the baseline rate, and Z values come from your desired significance (α=0.05) and power (β=0.8). For a 3% correctness improvement (95% → 98%), you need approximately:

python
from scipy.stats import norm
import math

def required_sample_size(
    baseline_rate: float = 0.95,
    mde: float = 0.03,              # Detect a 3% improvement
    alpha: float = 0.05,            # 5% significance level
    power: float = 0.80,            # 80% power
) -> int:
    p1 = baseline_rate
    p2 = baseline_rate + mde
    p_avg = (p1 + p2) / 2

    z_alpha = norm.ppf(1 - alpha / 2)  # 1.96
    z_beta = norm.ppf(power)              # 0.84

    n = ((z_alpha + z_beta) ** 2 * 2 * p_avg * (1 - p_avg)) / mde ** 2
    return math.ceil(n)

# Result: ~4,300 conversations per variant
# At 25K convs/day (50% in each variant), that's ~0.35 days
# But NEVER stop early just because p < 0.05!
print(required_sample_size())  # 4293

Simpson's Paradox in Agent Experiments

Here's a trap that catches even experienced data scientists. Your A/B test shows treatment wins overall (+2% correctness). But when you break down by conversation complexity:

Wait — treatment loses in every segment but wins overall? This is Simpson's Paradox. It happens when the treatment group received a disproportionate number of simple conversations (which have higher success rates regardless of variant). The treatment didn't improve anything — it just got an easier workload.

Root cause: Traffic routing wasn't properly randomized. Maybe a bug in your hash function clustered certain user segments. Maybe a concurrent experiment shifted traffic composition.

Fix: Always analyze results stratified by key confounders (conversation complexity, tenant, time of day, topic category). If the overall and stratified results disagree, trust the stratified analysis.

python
def analyze_experiment(experiment_id: str) -> ExperimentResult:
    data = load_experiment_data(experiment_id)

    # Overall analysis
    overall = compute_significance(data)

    # Stratified analysis (ALWAYS do this)
    strata = ["complexity", "tenant_tier", "topic_category", "hour_of_day"]
    stratified_results = {}
    for stratum in strata:
        stratified_results[stratum] = compute_significance_by_group(data, stratum)

    # Simpson's Paradox check
    paradox_detected = (
        overall.winner == "treatment" and
        all(s.winner == "control" for s in stratified_results["complexity"].values())
    )

    if paradox_detected:
        return ExperimentResult(
            conclusion="INCONCLUSIVE — Simpson's Paradox detected",
            recommendation="Investigate traffic composition imbalance",
        )

    return ExperimentResult(overall=overall, stratified=stratified_results)

When NOT to A/B Test

Not everything should be A/B tested:

Safety fixes: If you find a safety vulnerability (agent leaking PII, executing unauthorized actions), you ship the fix to 100% immediately. No experiment needed.

Obvious bugs: If the agent is returning 500 errors for 10% of queries, fix it. Don't A/B test the fix.

Tiny changes with large samples needed: If your MDE is 0.1% and you need 500K conversations per variant, the experiment runs for weeks. Is the decision worth waiting that long? Sometimes a judgment call + monitoring is faster.

Chapter 10: Retrieval, Ranking & Recommendation

Your RAG pipeline retrieved 50 candidate document chunks. The embeddings say they're all "similar." But similarity isn't usefulness. A document from 2019 about a discontinued policy is semantically similar to the current policy — but serving it would cause the agent to give outdated advice. This is a learning-to-rank problem.

Raw embedding cosine similarity is just one feature. A production ranker scores candidates on multiple signals: (1) Semantic relevance — how well the document matches the query, (2) Recency — newer documents override older ones (policies change), (3) User preference — this customer prefers detailed vs. concise answers, (4) Safety score — does this document contain information that's risky to surface (internal pricing, competitor mentions)?

The ranker is trained on implicit feedback. When a response leads to resolution, the documents used get positive signal. When a response leads to escalation, those documents get negative signal. Over time, the ranker learns which documents actually help — without any human labeling.

The denominator IDCG@k is the DCG you'd get if documents were perfectly ordered by relevance. This normalization puts the score on a [0, 1] scale. The logarithmic discount in the denominator means: position 1 has full weight, position 2 has 63% weight, position 10 has only 30% weight. This matches real usage — the agent reads the first few chunks most carefully.

Feature Engineering for Agent Ranking

A production ranker at Sierra might use 20-50 features. The core ones:

Chapter 11: Monitoring & Anomaly Detection

It's 2:17 PM on a Tuesday. Nobody deployed anything. No alerts fired. But your agent's resolution rate quietly dropped from 87% to 71% over the past hour. Customers are getting worse answers, more are escalating to humans, and you won't know about it until tomorrow's metrics review — unless your monitoring catches it in real time.

This is model drift without code changes. Three common causes: (1) the LLM provider silently updated their model (happens more than you'd think), (2) a knowledge base document got corrupted or deleted, (3) traffic shifted to a harder customer segment (e.g., a marketing campaign brought in confused first-time users). Traditional software monitoring (CPU, memory, 5xx rates) won't catch any of these because the system is technically "healthy."

The Agent Metrics Stack

A production agent monitoring system tracks five primary signals:

Anomaly Detection: Beyond Static Thresholds

Static thresholds ("alert if resolution < 80%") break because baselines shift with time-of-day, day-of-week, and seasonal patterns. Monday morning has different traffic than Saturday night. Instead, use rolling statistical bounds:

The EMA (exponential moving average) adapts to slow trends while still catching sudden spikes. The parameter α controls how quickly the baseline adapts — too fast and you miss sustained degradation (it becomes "normal"), too slow and you get false alarms after every natural shift.

Chapter 12: LLM Inference Serving

LLM inference is fundamentally memory-bandwidth bound, not compute bound. Here's why: to generate one token, the GPU must read every parameter of the model from HBM (High Bandwidth Memory). For a 70B model at FP16, that's 140 GB of weights read per token. An H100 GPU has 3.35 TB/s of memory bandwidth, so the theoretical minimum is 140/3350 = 42ms per token — regardless of how many FLOPS the GPU can do. The compute units are starved, waiting for data.

This memory-bound reality drives three critical optimizations that make production serving viable:

1. KV Cache

During autoregressive generation, each new token attends to ALL previous tokens. Without caching, you'd recompute every previous token's Key and Value projections at every step — quadratic cost. The KV cache stores these projections, turning generation into a linear operation. The cost: memory. For a 70B model with 80 layers, 64 heads, 128 dim/head, serving batch_size=32 at seq_len=4096:

This is why PagedAttention (from vLLM) was revolutionary — it manages KV cache like an OS manages virtual memory, eliminating fragmentation and enabling higher batch sizes.

2. Continuous Batching

Traditional batching: wait for N requests, process together, return all at once. Problem: short responses finish early but wait for the longest response in the batch. Continuous batching (also called "in-flight batching"): the moment one request in the batch finishes, immediately slot a new request into its position. GPU stays saturated, no request waits unnecessarily.

3. Speculative Decoding

Use a small "draft" model (e.g., 7B) to generate N candidate tokens quickly. Then verify all N tokens in a single forward pass of the large model. If the draft agrees with the large model (typically 70-85% acceptance rate), you generated N tokens for the cost of 1 large-model forward pass + 1 cheap draft pass. Net speedup: 2-3x.

Chapter 13: Infrastructure & CI/CD

You've improved the agent's prompt. Evals show 3% resolution uplift. Time to ship. In a traditional web app, a bad deploy shows a broken button for 5 minutes until someone notices. In an agentic system, a bad deploy means the agent issues unauthorized refunds, promises impossible delivery dates, or leaks customer data — and each bad conversation is irreversible. You can't un-say something to a customer.

This is why agentic platforms treat deployment as a graduated exposure problem. The Infrastructure-as-Code stack (Terraform/Pulumi for cloud resources, Kubernetes for orchestration, Helm for config) ensures every deployment is versioned, reproducible, and rollback-able. But the key insight is the canary strategy.

The Canary Deployment Pipeline

The bake time at each stage is critical. Agent failures often manifest with delay — a bad refund policy answer might not be discovered until the customer contacts their bank the next day. Sierra likely uses a combination of real-time metrics (resolution rate, safety violations) with "delayed outcome" metrics (CSAT scores, escalation within 24h) to gate promotions.

Automatic Rollback Triggers

The deploy system watches metrics and automatically rolls back if any gate fails. Common triggers:

yaml
rollback_triggers:
  resolution_rate:
    threshold: "-2% vs control"
    window: "15m"
  safety_violations:
    threshold: "+1 per 10K conversations"
    window: "5m"  # zero tolerance, fast rollback
  p95_latency:
    threshold: "+500ms vs baseline"
    window: "10m"
  error_rate:
    threshold: "+0.5%"
    window: "5m"

Chapter 14: Observability & Incident Management

A customer reports: "The agent said my order shipped but tracking shows it hasn't moved." How do you debug this? The agent's response was generated by an LLM that was fed documents retrieved from a vector DB based on a tool call to the order API. Any of these could be the culprit. You need to trace the exact path this specific request took through all services.

This is distributed tracing. Each request gets a unique trace ID that propagates across service boundaries. Within a trace, each service call is a span — with start time, duration, status, and metadata. The trace reconstructs the full story of one request.

The Three Pillars of Observability

Metrics tell you something is wrong. Traces tell you where it's wrong. Logs tell you why it's wrong. You need all three.

The Incident Playbook

Agent-Specific Tracing Challenges

Agentic systems have unique tracing problems:

Variable span counts: A simple question = 3 spans (gateway, LLM, response). A complex refund = 15+ spans (gateway, auth, memory, RAG, LLM, tool, tool, LLM, guardrail, response). Traditional waterfall views break down.

LLM calls are opaque: The LLM span shows "input: 4000 tokens, output: 200 tokens, latency: 1.2s" but not WHY it generated bad output. You need to log the full prompt and completion for debugging (expensive at scale — sample at 1-5%).

Async outcomes: The trace ends when the response is sent, but the "real" outcome (did the refund process? did the customer churn?) arrives hours or days later. You need to link traces to business outcomes retroactively.

Chapter 15: Security & Authentication

Your agent can look up order history, process refunds, modify account settings, and access personal data. It's not just a chat interface — it's a privileged actor with access to production systems. If an attacker can manipulate what the agent does, they can steal data, process fraudulent transactions, or escalate privileges. The agent IS the attack surface.

Defense Layers

1. OAuth 2.0 + Identity: Before the agent sees a single token of user input, the customer must be authenticated. The agent operates within the scope of the authenticated user's permissions. It can never access data for a different customer, even if instructed to.

2. RBAC (Role-Based Access Control): The agent's tool calls are permission-gated. A customer-facing agent can call `get_order()` but not `delete_user()`. An admin agent can do both. Permissions are defined per-role, not per-session, so even a compromised session can't escalate.

3. mTLS (Mutual TLS): Every service-to-service call requires mutual certificate authentication. The agent service proves its identity to the order API, and vice versa. A rogue process can't impersonate the agent.

4. Input Sanitization: This is where it gets interesting for AI systems.

Prompt Injection Attack Classes

Chapter 16: Distributed Systems

Sierra serves customers globally. That means multi-region deployment — US-East, US-West, EU-West at minimum. A customer in Berlin shouldn't wait 200ms extra for a round-trip to Virginia. But multi-region introduces the hardest problem in distributed systems: what happens to conversation state when regions disagree?

Picture this: a customer is mid-refund-process in US-East. They said "yes, process it." The agent acknowledged. Then US-East goes down. The customer refreshes, gets routed to EU-West. Does the agent remember the refund was approved? Did the refund API call actually fire? Is the state consistent?

CAP Theorem for Agent State

The CAP theorem states: in a distributed system, when a network partition occurs, you must choose between Consistency (all nodes see the same data) and Availability (every request gets a response). You can't have both during a partition.

The Failover Strategy

Conversation state is replicated asynchronously to all backup regions (with a replication lag of 1-5 seconds). When a region dies:

The inconsistency window is the gap between the last replicated state and the moment of failure. If the customer said "yes" but that message hadn't replicated yet, the backup region doesn't know about it. The agent gracefully handles this by re-reading context and saying "I want to confirm — would you like me to proceed with the refund?" Minor friction, but the system stays available.

Multi-Region Data Patterns

Conversation state: AP (eventual consistency, ~2s lag) — availability matters most.

Financial transactions: CP (strong consistency) — we'd rather be briefly unavailable than process a double refund. The refund API uses distributed locks (e.g., Redis Redlock) to prevent concurrent mutation.

Knowledge base: AP with read replicas — serving stale docs for 5 minutes is acceptable. Documents change daily, not per-second.

Chapter 17: Agent Steerability & Verifiability

Monday morning. The VP of Customer Success sends an urgent Slack message: "The agent offered a customer a 50% discount. Our max is 15%. This happened twice this weekend. How is this possible and how do we guarantee it never happens again?"

This is the guardrail problem. Two orthogonal challenges:

Steerability: Can we reliably control what the agent does and doesn't do? The agent must follow policy constraints even under adversarial inputs, edge cases, and ambiguous situations.

Verifiability: Can we PROVE it followed (or violated) policy, with an audit trail? When the VP asks "how did this happen?", you need a complete decision trace showing exactly which input led to which output through which reasoning path.

Defense in Depth for Policy

The key insight: Layer 3 is the only hard guarantee. Layers 1-2 are probabilistic (they reduce violations but can't eliminate them). Layer 4 is detective, not preventive. A well-designed system relies on deterministic enforcement at the API layer while using Layers 1-2 to reduce noise and Layer 4 for accountability.

Guardrail Implementation Patterns

python
class GuardrailPipeline:
    def check(self, response, context):
        # Layer 1: Regex/rule-based fast checks
        if self.contains_discount_over_max(response):
            return Block("discount_exceeded")

        # Layer 2: ML classifier (slower, catches nuance)
        safety_score = self.classifier.predict(response)
        if safety_score < self.threshold:
            return Block("safety_classifier")

        # Layer 3: Structured output validation
        if response.tool_calls:
            for call in response.tool_calls:
                if not self.permissions.allows(call):
                    return Block("permission_denied")

        # Layer 4: Audit (always runs, even on pass)
        self.audit_log.record(response, context)
        return Pass()

Chapter 18: Full System Showcase

This is the capstone. Every system from Chapters 0-17 working together. A single customer message flows through authentication, routing, memory retrieval, RAG, the agent loop, LLM inference, guardrails, tool execution, and monitoring — all in under 2 seconds.

This is also THE system design interview answer. When an interviewer says "Design the end-to-end architecture for Sierra's conversational AI platform," this simulation is what you're describing. Every box is a microservice. Every arrow is a network call with latency. Every slider is a production parameter.

Chapter 19: Interview Arsenal

You've walked through every system a staff agentic engineer touches in a single day. This chapter distills it into interview-ready ammunition. Print this, memorize it, and draw from it when the interviewer says "design a conversational AI platform."

17-Concept Cheat Sheet

System Design Interview Framework

Coding Drill Checklist

Debugging Scenarios (Practice These)

scenarios
# Scenario 1: Resolution rate drops 10% overnight, no deploy
Root cause tree:
  → LLM provider silent update (check model version header)
  → Knowledge base doc corrupted/deleted (check last-modified)
  → Traffic segment shift (stratify metrics by customer tier)
  → Embedding model drift (check retrieval quality metrics)

# Scenario 2: P95 latency spikes 3x, resolution unchanged
Root cause tree:
  → GPU saturation (check utilization, batch queue depth)
  → Context length explosion (check avg prompt tokens)
  → External API timeout (check tool call latency spans)
  → KV cache thrashing (check cache hit rate)

# Scenario 3: Agent offers unauthorized 50% discount
Root cause tree:
  → System prompt constraint missing/weakened (check version)
  → Output classifier false negative (check classifier logs)
  → API cap not configured (check refund API max params)
  → Prompt injection bypassed layers (check input sanitizer)

# Scenario 4: Customers report agent "forgot" their context
Root cause tree:
  → Memory service degraded (check memory retrieval latency)
  → Region failover caused state loss (check replication lag)
  → Context window overflow (check token count vs limit)
  → Session ID mismatch (check cookie/auth flow)

Classical vs Modern Comparison

Essential Reading