Observability & Operations

Chapter 1: The Four Golden Signals

If you could only measure four things about your service, what would they be? The Site Reliability Engineering discipline converges on the same four:

1. Latency

How long does it take to serve a request? Specifically, the distribution of latencies, not just the average. A service with 50ms average latency might have a p99 of 2 seconds — meaning 1% of users wait 40x longer than average.

Why Averages Lie: A Worked Example

This is why p99 is the most important latency metric. It represents the experience of your worst-affected users, who are often your highest-value users (they use the product frequently, so they hit the tail more often).

Histograms vs. Summaries

There are two ways to store latency data in a metrics system:

2. Traffic

How much demand is being placed on the system? For a web service: HTTP requests per second. For a database: queries per second. For a message queue: messages per second. Traffic tells you the current load level and helps you predict when you will need more capacity.

3. Errors

What fraction of requests fail? This includes explicit errors (HTTP 500s), implicit errors (HTTP 200 with wrong content), and policy violations (responses slower than a threshold). The error rate (errors per second or errors as a percentage of total requests) is more useful than the error count.

4. Saturation

How "full" is the system? Saturation measures how close you are to capacity: CPU utilization, memory usage, disk I/O, queue depth, connection pool usage. When saturation approaches 100%, performance degrades nonlinearly — a system at 90% CPU does not perform 10% worse than one at 0%, it performs dramatically worse due to queueing effects.

The simulation below shows all four golden signals for a live service. Traffic arrives at a steady rate, then spikes. Watch how the signals interact.

Chapter 2: RED & USE Methods

The four golden signals tell you what to measure for any service. But different types of components have different natural metrics. The RED method and USE method give you targeted frameworks.

RED Method (for services)

For any request-driven service (API, web server, microservice), measure three things:

RED is essentially the four golden signals minus saturation, focused on the user-facing experience. If you can only instrument three things per service, instrument RED.

USE Method (for resources)

For any resource (CPU, memory, disk, network, connection pool), measure three things:

When to Use RED vs. USE

The debugging workflow typically starts with RED (user-facing symptoms) and drills into USE (resource causes):

RED tells you THAT something is wrong. USE tells you WHERE the bottleneck is. Logs tell you WHY.

Mapping RED and USE to Your System

The simulation shows RED and USE metrics side by side. Inject load and watch how service-level metrics (RED) correspond to resource-level metrics (USE).

Chapter 3: SLIs, SLOs, SLAs

Your service is "reliable." But how reliable? 99%? 99.9%? 99.99%? And reliable in what dimension — availability? Latency? Correctness? These terms are often used loosely. The SLI/SLO/SLA framework gives them precise meaning.

SLI: Service Level Indicator

An SLI is a carefully defined quantitative measure of some aspect of the level of service that is provided. It is a metric with a specific definition.

SLO: Service Level Objective

An SLO is a target value for an SLI. It is an internal goal that your team sets.

SLA: Service Level Agreement

An SLA is a contract with your customer that includes consequences for failing to meet it (refunds, credits, contract termination). SLAs are typically looser than SLOs because you want to breach your SLO (internal alert) before you breach your SLA (legal/financial consequences).

The Nines

Choosing the Right SLO

How do you decide between 99.9% and 99.99%? It depends on your users and your dependencies:

SLI Selection: What to Measure

Choosing the right SLI is critical. A bad SLI creates a false sense of security. Here are the most common SLIs by service type:

The simulation shows how SLO targets map to allowed downtime. Adjust the SLO slider and see how much failure budget you have.

Chapter 4: Error Budgets

The SLO says 99.9% availability. That means 0.1% unavailability is allowed. This 0.1% is your error budget — the amount of unreliability you can tolerate before breaching your SLO.

Error budgets transform reliability from a vague aspiration into a concrete, spendable resource. You can spend your error budget on things that matter: faster feature releases, risky experiments, infrastructure migrations. As long as you stay within budget, you are meeting your reliability commitment.

How Error Budgets Work

Error Budget Policies

Error Budget-Based Decision Making

Error budgets change how organizations make engineering decisions. Here are concrete examples:

What Consumes Error Budget

Not all budget consumption is equal. Some is expected (planned maintenance), some is unplanned but tolerable (transient errors), and some requires investigation.

Burn Rate Alerting: The Math

Traditional alerting fires when a metric crosses a threshold. Burn rate alerting fires when you are consuming error budget too fast — even if the absolute error rate is below your threshold.

The simulation below tracks an error budget over a 30-day period. Deployments and incidents consume budget. Watch the budget drain and see when policies activate.

Chapter 5: Alerting

An alert wakes a human up at 3 AM. That alert better be important. If it is a false alarm — or a real alarm that does not require immediate human action — you have just wasted someone's sleep, eroded trust in your alerting system, and contributed to alert fatigue.

Symptom-Based vs. Cause-Based Alerting

The most important principle in alerting: alert on symptoms, not causes.

Alert Severity Levels

Multi-Window, Multi-Burn-Rate Alerting

The most sophisticated alerting approach ties directly to error budgets. Instead of fixed thresholds, alert when you are burning error budget too fast:

On-Call Practices

An alert pages a human. That human is the on-call engineer — someone who has agreed to be reachable 24/7 for a rotation period (usually one week). On-call is one of the most important and demanding roles in engineering. Bad on-call practices burn out engineers and degrade incident response quality.

Incident Response Framework

When a page fires, the on-call engineer should follow a structured response:

The simulation shows different alert configurations and their impact on on-call burden. Compare symptom-based and cause-based alerting.

Chapter 6: Structured Logging

A traditional log line looks like this: 2024-01-15 01:47:23 ERROR: Payment failed for user 12345, amount=$50.00, reason=timeout. It is human-readable. It is also impossible to query efficiently. Want to find all payment failures over $100 in the last hour? You are parsing strings with regex.

Structured logging emits log events as key-value pairs (usually JSON), making them queryable by any field:

What to Log

Structured Logging Implementation

python
import json, time, uuid

class StructuredLogger:
    def __init__(self, service_name, version):
        self.service = service_name
        self.version = version

    def log(self, level, event, **kwargs):
        entry = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ"),
            "level": level,
            "service": self.service,
            "version": self.version,
            "event": event,
            **kwargs  # trace_id, user_id, duration_ms, etc.
        }
        print(json.dumps(entry))  # stdout, captured by log agent

# Usage:
logger = StructuredLogger("payment-service", "v2.3.1")
logger.log("ERROR", "payment_failed",
    trace_id=request.trace_id,
    user_id="12345",
    amount=50.00,
    reason="timeout",
    duration_ms=30042
)
# Output: {"timestamp":"2024-01-15T01:47:23Z","level":"ERROR",
#  "service":"payment-service","version":"v2.3.1",
#  "event":"payment_failed","trace_id":"abc-123",
#  "user_id":"12345","amount":50.0,"reason":"timeout",
#  "duration_ms":30042}

Log Levels: When to Use Each

The simulation shows how structured vs. unstructured logging affects incident diagnosis time.

Chapter 7: Distributed Tracing

A user's request enters your system at the API gateway. It is routed to the order service, which calls the inventory service, which calls the database, which calls the cache. The entire round trip takes 2.3 seconds. Where did the time go?

Logs tell you what happened at each service. Metrics tell you aggregate behavior. But neither tells you the journey of a single request through the entire system. That is what distributed tracing does.

Anatomy of a Trace

A trace represents the entire lifecycle of a request. It is composed of spans, where each span represents one operation (one service call, one database query, one cache lookup).

The trace makes it immediately obvious: the Payment Service took 2195ms, and within that, the DB query took 2140ms. That is the bottleneck. Without tracing, you would know "the request took 2.3 seconds" but not where in the chain the time was spent.

Trace Context Propagation

For tracing to work, every service must propagate the trace context (trace_id, span_id, parent_span_id) to the next service. The standard is the W3C Trace Context specification, which uses HTTP headers:

Sampling Strategies

The Three Pillars Connected

The real power of observability comes from connecting the three pillars. A metric tells you something is wrong. A trace shows you where. A log tells you exactly what happened.

This workflow — metric alert → trace → log → root cause — takes 5-10 minutes with good observability. Without it, the same investigation takes 30-60 minutes of guessing and grepping.

The simulation below shows a distributed trace. Click on spans to see details. Inject a slow dependency to see where the bottleneck appears.

Chapter 8: Live Dashboard Simulator

This is the showcase chapter. A live dashboard showing all four golden signals, with alert indicators, trace links, and the ability to inject incidents. This is what the on-call engineer sees at 3 AM.

What a Good Dashboard Shows

Dashboard Anti-Patterns

A bad dashboard is worse than no dashboard, because it gives a false sense of visibility. Here are the most common mistakes:

The Metrics Pipeline

How do metrics get from your application to a dashboard? The pipeline has four stages:

Chapter 9: Connections

Observability is the final piece of the distributed systems reliability stack. You now have the complete picture: what fails, how to isolate it, how to survive it at runtime, how to ship safely, and how to see what is happening in production.

The Complete Stack

Key Takeaways

1. Measure the four golden signals. Latency (percentiles!), traffic, errors, saturation. If you can only have four metrics, make it these four.

2. RED for services, USE for resources. Rate/Errors/Duration for request-driven services. Utilization/Saturation/Errors for hardware resources.

3. SLOs are budgets, not aspirations. Set them based on user needs. Never set 100%. Use error budgets to make deployment risk decisions.

4. Alert on symptoms, not causes. "Error rate > 1%" pages a human. "CPU > 90%" creates a ticket. Every page must be urgent, actionable, and real.

5. Structure your logs. JSON with trace_id lets you query efficiently and correlate across services. Never log secrets.

6. Trace the critical path. Distributed tracing shows where time is spent. Sample at 1% in production, 100% for errors and slow requests.

7. Dashboards should answer questions in seconds. Top banner for status, golden signals for detail, drill-down for investigation.