Testing & Deployment

Chapter 1: The Test Pyramid

Not all tests are equal. A unit test that checks whether a function adds two numbers correctly runs in 1 millisecond and tests exactly one thing. An end-to-end test that spins up the entire application, opens a browser, fills in a form, and verifies the result in the database takes 30 seconds and tests everything at once.

The test pyramid is a model for how to allocate your testing effort. The base is wide (many cheap, fast tests), and the top is narrow (few expensive, slow tests).

Why the Pyramid Shape?

The anti-pattern is the ice cream cone: lots of E2E tests, few unit tests. This results in a slow, flaky test suite that everyone ignores. Teams start shipping without waiting for tests because the tests take 45 minutes and fail randomly due to browser timing issues.

The simulation below shows two test suites: a pyramid (many unit, few E2E) and an ice cream cone (many E2E, few unit). Watch how they differ in speed and reliability.

Chapter 2: Test Sizes

The test pyramid categorizes tests by what they test (unit vs. integration vs. E2E). Test sizes categorize tests by how many resources they consume. This is a complementary classification used by large engineering organizations to enforce test quality and CI speed.

The Three Sizes

Why Size Matters More Than You Think

Consider a test labeled "unit test" that creates a temporary database connection. On the developer's laptop with a fast SSD and no network contention, it runs in 50ms. In CI, running alongside 500 other tests on a shared machine, it takes 3 seconds because of disk I/O contention. This test is nondeterministic — its behavior depends on the environment. It will fail intermittently (a "flaky" test), eroding confidence in the test suite.

A true small test uses no I/O, no network, no disk. It runs in 5ms on any machine, every time, deterministically. Size classification prevents environment-dependent failures from polluting your test suite.

The Real Cost of Flaky Tests

Size Budgets

Some organizations enforce size budgets: at least 80% of tests must be small, at most 15% medium, and at most 5% large. This ensures the test suite stays fast and reliable as the codebase grows.

The simulation below shows how test size distribution affects total CI time. Adjust the slider to change the ratio of small to large tests.

Chapter 3: Chaos Engineering

You have tests for the happy path. You have tests for known error conditions. But what about the errors you have not imagined? What about the interaction between a network partition and a GC pause and a traffic spike — all happening simultaneously?

Chaos engineering is the practice of deliberately injecting failures into a running system to discover weaknesses before they cause real outages. Instead of waiting for failures to find you, you find them first.

The Scientific Method of Chaos

Common Chaos Experiments

Game Days

A game day is a scheduled chaos engineering exercise where the entire team participates. Unlike automated chaos experiments that run in the background, game days are planned events where engineers deliberately break the system, observe the response, and practice incident response procedures.

Game days serve a dual purpose: they test the system's resilience and the team's incident response capability. A system might automatically failover perfectly, but if the on-call engineer doesn't know how to read the dashboard or escalate correctly, the human response is the bottleneck.

Formal Verification: TLA+

For the most critical distributed algorithms (consensus protocols, database replication), chaos engineering is not enough. You cannot test every possible interleaving of events. Formal verification tools like TLA+ (created by Leslie Lamport) let you mathematically prove that your algorithm handles all possible states.

Amazon uses TLA+ extensively. Their engineers have found subtle bugs in DynamoDB, S3, and other systems — bugs that would be nearly impossible to find through testing alone because they require specific, rare combinations of concurrent events.

The simulation below lets you run chaos experiments on a simple distributed system. Form a hypothesis, inject a fault, and see if the system behaves as expected.

Chapter 4: CI/CD Pipeline

A CI/CD pipeline (Continuous Integration / Continuous Delivery) automates the path from code commit to production deployment. The goal is to make deployments frequent, small, and safe — rather than rare, large, and terrifying.

Pipeline Stages

The Four Key Metrics (DORA)

The DevOps Research and Assessment (DORA) program identified four metrics that distinguish elite engineering teams from the rest:

Pipeline Optimization Techniques

Pipeline Anti-Patterns

The simulation shows a CI/CD pipeline processing commits. Watch how different pipeline configurations affect lead time and deployment frequency.

Chapter 5: Deployment Strategies

You have a new version of your service ready to deploy. How do you get it into production without causing an outage? There are four major strategies, each with different trade-offs.

Rolling Deployment

Replace instances one at a time. At any moment, some instances run the old version and some run the new. The rollout progresses gradually over minutes or hours.

Blue-Green Deployment

Run two identical environments: "blue" (current production) and "green" (new version). Deploy the new version entirely to green. When green is healthy, switch the load balancer to route all traffic from blue to green. Instant cutover.

Canary Deployment

Deploy the new version to a tiny fraction of traffic (e.g., 5%) and monitor for errors. If the canary is healthy, gradually increase traffic (10%, 25%, 50%, 100%). If the canary shows problems, rollback only the canary — 95% of users were never affected.

Comparison

Canary in Detail: The Progression

A typical canary deployment follows a multi-stage progression with automated health checks at each gate:

The key detail: the wait times increase at each stage because some bugs only manifest after minutes of operation (e.g., slow memory leaks, connection pool exhaustion, cache warming issues).

Automated Canary Analysis

Manual canary evaluation does not scale. Automated canary analysis (ACA) compares the canary's metrics to the baseline using statistical tests:

The simulation below compares the three deployment strategies visually. Watch how traffic shifts between versions.

Chapter 6: Feature Flags

Deployment and release are often conflated, but they are different things. Deployment means putting new code on servers. Release means enabling that code for users. A feature flag separates these: you deploy code that is disabled by default, then enable it later — for specific users, a percentage of traffic, or all at once.

Why Decouple Deploy from Release?

Types of Feature Flags

Feature Flag Implementation

python
# Simple feature flag with gradual rollout
import hashlib

class FeatureFlags:
    def __init__(self, config):
        self.config = config  # {"new_checkout": {"pct": 10}}

    def is_enabled(self, flag_name, user_id):
        flag = self.config.get(flag_name)
        if not flag:
            return False

        # Deterministic: same user always gets same result
        hash_val = hashlib.md5(
            f"{flag_name}:{user_id}".encode()
        ).hexdigest()
        bucket = int(hash_val[:8], 16) % 100

        return bucket < flag["pct"]

# Usage:
flags = FeatureFlags({"new_checkout": {"pct": 10}})

if flags.is_enabled("new_checkout", user.id):
    return new_checkout_flow(request)
else:
    return old_checkout_flow(request)

The hash-based approach is critical: the same user always sees the same version. Without deterministic assignment, a user might see the new checkout on page load but the old checkout on form submission — a broken experience.

Feature Flag Lifecycle

The simulation shows how feature flags enable progressive rollout and instant rollback without deployment changes.

Chapter 7: Rollbacks

Things will go wrong. The question is not whether you will need to rollback, but how fast you can do it and how much damage occurs between deploying the bad version and completing the rollback.

Rollback Strategies

What Makes Rollback Hard

Code rollback is easy. Data rollback is hard. If the bad deployment wrote data in a new format, changed a schema, or made irreversible state changes (sent emails, charged cards), rolling back the code does not roll back the data.

What Makes Rollback Safe: Backward Compatibility

A deployment is rollback-safe if the previous version of the code can continue operating correctly after the new version has run. This is a stronger requirement than forward compatibility.

The pattern: adding is safe, removing or changing is not. Every breaking change must be done in multiple steps, with each step being independently rollback-safe.

Rollback Automation

The best rollback is one that happens automatically. If canary metrics degrade beyond a threshold, the deployment system should roll back without human intervention.

The simulation below shows a bad deployment and different rollback strategies. Watch how quickly each strategy recovers and how much data impact occurs.

Chapter 8: Canary Deployment Simulator

This is the showcase chapter. You are deploying a new version of your service to production using a canary strategy. Start at 5% traffic, monitor error rate and latency, and decide: promote to 100% or rollback?

What to Monitor During Canary

Chapter 9: Connections

Testing and deployment are how you ship changes safely. But once the code is in production, you need to observe it — to know whether it is healthy, to detect problems before users do, and to diagnose root causes when things go wrong.

This Lesson vs. Related Topics

Key Takeaways

1. The test pyramid is an investment strategy. Many cheap, fast tests (unit). Fewer expensive, slow tests (E2E). Classify by size, not just type.

2. Chaos engineering tests the real system. Inject real failures into real infrastructure. Form hypotheses. Measure outcomes. File tickets for surprises.

3. Small, frequent deploys are safer than large, rare ones. One change is easy to diagnose. Fifty changes are a mystery.

4. Canary deployments limit blast radius. 5% traffic to canary, 95% on stable. Promote when healthy, rollback when not.

5. Feature flags decouple deploy from release. Ship code anytime. Enable for users when ready. Kill switch instantly.

6. Rollback readiness is a design requirement. Schema changes must be backward-compatible. Data changes must be reversible. Plan for rollback before you deploy.