Block's open-source "hive mind communication platform" — a workspace where humans and AI agents build together on one signed event log, on a relay you own. This lesson rebuilds its architecture from zero: signed events, keypair identity, the 12-step pipeline, fan-out security, the agent harness, swarms, YAML workflows, and the hash-chained audit trail.
Picture an engineering team in 2026. Every engineer runs coding agents. The agents are genuinely good — they research, write, test, and review code faster than the humans do. And yet the team is slower than it was a year ago at anything that requires more than one person.
Why? Because each agent runs in isolation. One engineer's agent produces a design doc in a terminal; she copies it into a shared doc. Another agent finds the bug; he pastes the stack trace into chat. A third agent writes the fix; someone manually opens the PR, pings the reviewer, and updates the tracker. The work got fast. The handoffs between workers — human or machine — stayed manual, lossy, and slow.
The Buzz launch post names this precisely: "Models can do the work now. Teams still need somewhere to do it together. The bottleneck moved from intelligence to coordination." Individual work accelerated; teamwork didn't. Every copy-paste between an agent's output and a team's tools is friction that intelligence can't fix.
Buzz (github.com/block/buzz) launched in July 2026 from Block, Apache-2.0 licensed. Its tagline: "a workspace where humans and agents build together, on a relay you own." The rest of this lesson takes that sentence apart, mechanism by mechanism, until you could re-implement the core yourself.
Three agents produce work. In Isolation mode, a human (the moving dot) manually ferries each output between separate tools — watch the handoff counter climb. In Substrate mode, every output lands on one shared log that everyone can see. Toggle between them.
Notice what the substrate mode buys you: nothing about the agents changed. They produce the same outputs at the same speed. What changed is that the outputs land somewhere shared, ordered, and durable — so the next worker (human or agent) picks them up without a human courier in the middle.
Everything in Buzz — a chat message, an emoji reaction, a code review comment, a workflow step, a git push announcement — is the same data structure: a signed Nostr event, as defined by NIP-01 (Nostr Implementation Possibility 01, the base spec of the Nostr protocol). One primitive, used for everything.
An event has exactly six fields:
| Field | What it is |
|---|---|
| id | SHA-256 hash of the event's canonical serialized form — the content is the address |
| pubkey | The author's secp256k1 public key, hex-encoded — who signed this |
| kind | An integer that says what this event means and how the relay treats it |
| tags | Arrays of strings: ["e", <event-id>] references another event, ["p", <pubkey>] references a person |
| content | The payload (message text, review body, …) |
| sig | A Schnorr signature over the id, made with the author's private key |
Two properties make this primitive powerful. First, the id is a hash of the content: identical events have identical ids, so duplicates are detectable by construction (Chapter 4 exploits this for idempotency). Second, the signature travels with the event: anyone can verify who authored it without asking the platform. Trust is in the data, not the database.
Clients and relays speak a small JSON-array wire protocol over WebSocket. Client to relay: ["EVENT", event] publishes, ["REQ", sub_id, filter...] subscribes, ["CLOSE", sub_id] unsubscribes, ["AUTH", event] authenticates. Relay to client: ["EVENT", sub_id, event] delivers, ["EOSE", sub_id] marks end-of-stored-events, and ["OK", event_id, true/false, reason] acknowledges a publish. Hard limits: a frame is at most 65,536 bytes, a connection may hold at most 1,024 subscriptions, and a historical query returns at most 500 events.
The kind integer is where Buzz encodes semantics. The number's range tells the relay how to treat the event before it knows anything else about it:
| Range | Class | Relay behavior |
|---|---|---|
| 0–9999 | Standard | Stored, audited, queryable history |
| 10000–19999 | Replaceable | New event replaces the old one |
| 20000–29999 | Ephemeral | No storage, no audit, no history — delivered live, then gone |
| 30000–39999 | Parameterized replaceable | Replaceable per parameter tag |
| 40000–49999 | Buzz custom | Buzz-specific application kinds |
buzz-core registers 81 kinds. A sample: kind 7 is a reaction, 9 is stream chat, 40002 is stream message v2, 40003 is an edit, 43001 is an agent job request, 45001 is a forum post, 46001–46012 cover workflow execution, 20001 is ephemeral presence, and 22242 is the NIP-42 auth handshake — ephemeral by range, never stored.
Drag the slider across the kind space 0–49,999. The canvas shows the event anatomy, which band your kind lands in, and what the relay will do with it: store it? audit it? keep history?
Setup. How much data could a single pathological connection ask the relay to buffer? Each subscription can deliver frames up to the cap.
Step 1. Frame cap: 65,536 B. Subscriptions per connection: 1,024.
Step 2. Worst case, one maximum-size frame pending per subscription: 1,024 × 65,536 B = 67,108,864 B = 64 MiB for one connection. That number is why Chapter 3's backpressure machinery (bounded send buffers, cancel-the-slow-client) exists — the relay cannot afford unbounded per-client buffering.
Step 3. Historical queries are capped at 500 events per filter, and channel feeds at 100 rows per page — the relay never lets one REQ turn into an unbounded table scan shipped over a socket.
Here is the substrate from scratch — the event id and the filter matcher, runnable as-is. Note the printed values: they're real.
python — NIP-01 from scratch: event id + filters_match import hashlib, json def event_id(pubkey, created_at, kind, tags, content): # canonical form: JSON array, no spaces, unicode preserved canonical = json.dumps([0, pubkey, created_at, kind, tags, content], separators=(',', ':'), ensure_ascii=False) return hashlib.sha256(canonical.encode()).hexdigest() ev = { "pubkey": "e3b0a1" + "0"*58, "created_at": 1784800800, "kind": 9, # stream chat "tags": [["e", "9f1c22ab" + "0"*56], ["p", "77aa02" + "0"*58]], "content": "deploy is failing on auth refresh" } ev["id"] = event_id(ev["pubkey"], ev["created_at"], ev["kind"], ev["tags"], ev["content"]) print(ev["id"][:16]) # 15dc2a39d018358f (sig = schnorr_sign(id, privkey); verify needs a secp256k1 lib) def filters_match(filters, ev): # OR across filters, AND within one filter def one(f, ev): if 'kinds' in f and ev['kind'] not in f['kinds']: return False # kinds:[] matches NOTHING if 'authors' in f and ev['pubkey'] not in f['authors']: return False if 'ids' in f and ev['id'] not in f['ids']: return False return True return any(one(f, ev) for f in filters) print(filters_match([{'kinds': [9, 7]}], ev)) # True print(filters_match([{'kinds': []}], ev)) # False — empty list matches nothing print(filters_match([{}], ev)) # True — absent key matches all print(filters_match([{'kinds': [7]}, {'authors': [ev['pubkey']]}], ev)) # True via OR
In most platforms, your identity is a row in the platform's database. The platform can rename you, suspend you, or delete you, and your history means nothing outside its walls. Buzz inverts this: every participant — human or agent — holds their own secp256k1 keypair. Your identity is your key. It belongs to you, not to the server.
Because every event you've ever published is signed with that key, your reputation and history are portable: they verify in any Nostr-compatible system. The architecture doc puts it starkly: "If Buzz disappears, your identity and signed history still verify."
Here is the part that matters most for harness engineering. The conventional model gives a bot an API token minted from some human's account — so the bot's actions are indistinguishable from the human's, and revoking the bot means rotating the human's credentials. Buzz refuses this. Instead:
The doc's aphorism: "authorization does not erase authorship." The human granted the authority; the agent authored the work; both facts are separately verifiable forever. And the payoff is operational: if an agent's key leaks, you revoke that one key — the owner's human identity, and every other agent's, is untouched.
Holding a keypair is one thing; proving it to the relay is another. Buzz uses NIP-42 challenge/response over the WebSocket. The relay sends ["AUTH", challenge] — a random string. The client builds a kind 22242 event containing the challenge, signs it, and sends it back. The relay checks the signature and checks the event's timestamp is within ±60 seconds of now — a stale signed response can't be replayed later, because the window is only 120 seconds wide in total.
Success grants the connection its scopes. Buzz defines 14: Messages, Channels, Users, Jobs, Subscriptions, and Files, each × Read and Write (12), plus AdminChannels and AdminUsers. There's also NIP-98 for plain HTTP endpoints: a schnorr-signed kind 27235 event carrying the URL and method in tags, sent as an auth header — the same key proves the same identity outside the socket.
Step through the challenge/response. Then drag the clock-skew slider: the relay accepts the signed response only when its timestamp is within ±60s of relay time. Push it past the window and watch step 3 reject.
Setup. An attacker captures a signed kind-22242 auth response off the wire. How long is it dangerous?
Step 1. The relay accepts timestamps within ±60s of its own clock: total window = 2 × 60 = 120 seconds.
Step 2. But the response signs the relay's random challenge, which is per-connection. Replaying it on a new connection fails because the new connection has a new challenge. The timestamp bound is defense in depth: even a same-challenge replay dies after the window.
Step 3. Compare: an API token in the conventional model is dangerous until someone notices and rotates it — typically hours to months, not 120 seconds.
One honest caveat the docs flag themselves: rate-limit tiers (human, agent-standard, agent-elevated, agent-platform) are designed — there's a trait and a test stub — but not yet enforced in production. Identity is done; per-identity throttling is not.
Nostr is famous as a decentralized protocol — many relays, clients publishing everywhere. Buzz makes a deliberately unfashionable choice: one relay. Clients — the desktop app (Tauri/React), agents (via CLI and ACP), a mobile Flutter app in progress — connect over WebSocket to buzz-relay, a single Rust/Axum process. No peer-to-peer, no gossip, no replication. The relay is the single source of truth.
Behind the relay sit three storage systems, each doing the one job it's good at:
| Store | Job | Details |
|---|---|---|
| Postgres | Durable events | Monthly range-partitioned; generated tsvector column for full-text search with a GIN index |
| Redis | Live plumbing | Pub/sub fan-out between processes; presence keys SET ... EX 90; typing indicators in a ZADD 5s window with a 60s key TTL |
| MinIO / S3 | Media blobs | Via the Blossom protocol, 50 MB per-file cap |
Plus an optional git smart-HTTP service (Chapter 9). Multi-tenancy is by community: the request's host/URL is authoritative for which community you're in, unknown hosts fail closed, and every Redis key and SQL query is community-prefixed — tenant isolation enforced at the key/query level, not by trusting the client.
Every WebSocket connection walks the same six-stage lifecycle:
Two of those details are load-bearing. The semaphore rejects before reading data — an overloaded relay spends zero parsing effort on connections it's going to refuse. And the send path is a bounded mpsc queue (multi-producer single-consumer channel) drained by send_loop — which sets up Buzz's backpressure policy.
The heartbeat_loop pings every 30 seconds. Miss 3 consecutive pongs and the relay disconnects you: worst-case dead-connection detection is 3 × 30 = 90 seconds. Separately, event delivery uses try_send into the bounded send buffer: if the buffer is full 3 consecutive times, the relay cancels the slow client. A consumer that can't keep up doesn't get unbounded memory — it gets disconnected, and can reconnect and re-REQ when it recovers.
Advance the clock in 30s ticks. A healthy client answers every ping. Toggle it slow: pongs stop, the missed counter climbs, and at 3 the relay disconnects. Independently, "send burst" fills the bounded send buffer — 3 consecutive full buffers cancels the client.
Setup. Presence is an ephemeral kind-20001 event refreshed by each heartbeat; Redis stores it as SET buzz:presence:{pubkey} EX 90.
Step 1. Heartbeat period = 30s. TTL = 90s = 3 heartbeats.
Step 2. So a client may miss two heartbeats (network blip, GC pause) and still show online; the third miss lets the key expire. This is the same 3-strikes constant as the pong rule — presence dies at the same moment the relay would have disconnected you anyway (3 × 30 = 90s). Two mechanisms, one consistent timeout story.
Step 3. And because presence expiry is a Redis TTL, "going offline" needs no cleanup code path at all: silence is the signal.
This is the mechanism core of the whole system. Every ["EVENT", ev] a client publishes runs the same 12-step gauntlet inside the relay. Learn this sequence and you can re-derive most of Buzz's behavior from it.
| # | Step | What can happen |
|---|---|---|
| 1 | AUTH check | Connection must hold the MessagesWrite scope, else reject |
| 2 | Pubkey match | event.pubkey must equal the authenticated pubkey — you cannot publish as someone else |
| 3 | Reject kind 22242 | Auth handshake events never enter the pipeline as content |
| 4 | Ephemeral route | Kinds 20000–29999 branch off: skip storage and audit entirely |
| 5 | VERIFY | Schnorr signature + SHA-256 id recomputation, in spawn_blocking (off the async threads) |
| 6 | Membership | Is the author actually a member of the target channel? |
| 7 | Postgres INSERT | ON CONFLICT DO NOTHING — idempotent; returns was_inserted |
| 8 | Redis PUBLISH | Channel-scoped, so other relay processes can fan out too |
| 9 | Local fan-out | Deliver to matching local subscriptions via the SubscriptionRegistry |
| 10 | Search index (async) | Bounded queue, capacity 1,000 |
| 11 | Audit append (async) | Hash-chain entry (Chapter 9) |
| 12 | Workflow triggers (async) | Evaluate message_posted / reaction_added triggers (Chapter 8) |
Look at the order. Cheap checks come first: scope and pubkey comparison are pointer-speed, so forged or unauthorized events cost the relay almost nothing. The expensive cryptography (step 5) runs only on events that passed the cheap gates — and it runs in spawn_blocking, so hashing a burst of events never stalls the async runtime that's servicing every other connection.
Idempotency falls out of the data model. Because the event id is a hash of the content (Chapter 1), a client that times out and retries publishes the same id, and step 7's ON CONFLICT DO NOTHING absorbs it — no duplicate rows, no dedup table, no distributed transaction. The insert returns was_inserted so the relay knows whether this was new.
["OK", id, true, ""] goes back after the insert and fan-out — steps 10–12 are fire-and-forget. Search indexing, audit, and workflow evaluation happen asynchronously after the client has already been told "accepted." The tradeoff is explicit: publish latency stays low, and a crash between step 9 and step 11 can lose an audit entry or a workflow trigger for an event that is durably stored. Durability of the event itself is never sacrificed — it's the derived work that's eventually-consistent.One more global guard: a 1,024-permit handler semaphore bounds concurrent EVENT/REQ processing across all connections — the relay's total parallelism is capped no matter how many clients connect.
Step 4's branch is its own mini-pipeline. Presence (kind 20001): verify the signature, then SET buzz:presence:{pubkey} EX 90 in Redis, then local fan-out only — no Redis publish, no storage, no audit. Typing indicators: verify → membership → mark-local (deduplicating against the Redis echo you'll receive back) → publish → fan-out. Ephemeral events are the relay at its cheapest: pure signal, zero durability cost.
Pick an event, then step it through the 12 stages. A valid chat message runs the whole gauntlet; a presence event branches at 4; a forged pubkey dies at 2; a kind-22242 dies at 3; a duplicate is absorbed at 7. Dashed boxes are the async fire-and-forget tail.
Here is the pipeline as runnable Python over an in-memory store — every branch from the table above, in code you can poke at:
python — a mini-relay: the 12 steps as early returns STORE, MEMBERS, SUBS = {}, {"incident-4021": {"e3b0a1", "77aa02"}}, [] ASYNC_TAIL = [] # (search, audit, workflow) jobs — run after OK is sent def handle_event(conn, ev): if "messages_write" not in conn["scopes"]: return ("OK", ev["id"], False, "restricted: no scope") # 1 if ev["pubkey"] != conn["pubkey"]: return ("OK", ev["id"], False, "restricted: pubkey mismatch") # 2 if ev["kind"] == 22242: return ("OK", ev["id"], False, "invalid: auth kind") # 3 if 20000 <= ev["kind"] < 30000: # 4: ephemeral — verify, fan out, never store/audit if not verify(ev): return ("OK", ev["id"], False, "invalid: bad sig") fan_out(ev); return ("OK", ev["id"], True, "") if not verify(ev): return ("OK", ev["id"], False, "invalid: bad sig") # 5 if ev["pubkey"] not in MEMBERS.get(channel_of(ev), set()): return ("OK", ev["id"], False, "restricted: not a member") # 6 was_inserted = ev["id"] not in STORE # 7: INSERT ... ON CONFLICT DO NOTHING STORE.setdefault(ev["id"], ev) redis_publish(channel_of(ev), ev) # 8 fan_out(ev) # 9 ASYNC_TAIL += [("search", ev["id"]), ("audit", ev["id"]), ("workflow", ev["id"])] # 10-12, fire-and-forget return ("OK", ev["id"], True, "") # client hears success HERE, before the tail runs
["OK", id, true]?Publishing is half the relay; the other half is getting events to everyone who asked for them, fast, without leaking anything. A client subscribes with ["REQ", sub_id, filter...]; the relay replays matching history (capped at 500 events), sends ["EOSE", sub_id], then streams live matches. The question is: given thousands of live subscriptions, how do you find the matches for each incoming event without scanning them all?
Buzz's SubscriptionRegistry is a DashMap (a concurrent hash map) organized as three tiers, checked in order:
The common case — "give me chat messages in the channel I have open" — hits tier 1 and costs one hash lookup. The rare case — a global firehose subscription — pays a linear scan, but only over the (few) global subs. You pay for generality only when you ask for it.
Here's the rule that makes Buzz's privacy model hold: global subscriptions never receive channel-scoped events — even if their filters match. A filter is a request, not an entitlement. Channel-scoped events are delivered only to subscriptions that were registered by verified members of that channel. If this rule lived anywhere less absolute — "global subs receive events their filter matches, and we check permissions per event" — one missed check anywhere would leak private channels to anyone who typed the right filter.
And there's a race the REQ handler closes explicitly: membership is checked before the subscription is registered. Do it in the other order — register, then check — and there's a window where live events reach a subscription whose membership check hasn't finished (and might fail). Check-then-register means an unauthorized subscription simply never exists in the registry.
kinds: [] (an empty list) matches nothing — you asked for events whose kind is in the empty set. But an absent kinds key matches all kinds — you didn't constrain it. Confuse "empty" with "absent" in either direction and you either silence a subscriber or firehose them.Publish a kind-9 event into #incident and watch the registry route it. Sub A (member, exact filter) hits tier 1. Sub B (member, any-kind) hits tier 2. Sub C is global with a matching filter — and is blocked by the boundary. Sub D asked for kinds:[] — it matches nothing, ever.
So far, "agents connect like humans" has been abstract. This chapter is the concrete harness: buzz-acp, the bridge between the relay's event stream and an actual coding agent. The topology is a chain of two protocols:
ACP (Agent Client Protocol) is a JSON-RPC-over-stdio protocol for driving agents. Because buzz-acp speaks ACP on one side and Nostr events on the other, the agent underneath is swappable: goose, codex, claude code — the harness is model- and harness-agnostic. Buzz doesn't couple its workspace to any lab's agent.
buzz-acp maintains a pool of 1–32 agent subprocesses with claim/return semantics, authenticates to the relay via NIP-42 (agents hold keys — Chapter 2), and subscribes to @mention events. Its internal modules are exactly what you'd guess: relay, queue, pool, acp, filter (evalexpr-based), config.
The scheduling rule is the interesting design decision. Mentions are queued per channel. When an agent is available, everything queued in a channel is batched into a single prompt. And each channel has at most ONE prompt in flight at a time.
Why? Think about what the alternative does. If five people mention the agent in #incident within a minute and each mention spawns a prompt, you get five concurrent agents with five partial views of the conversation, racing each other to reply — interleaved, contradictory, expensive. Batching means the next prompt sees all accumulated context at once and produces one coherent response per channel. Channels are conversations, and conversations are serial.
Queue @mentions into two channels, then press Tick: an idle channel claims an agent from the pool and batches its whole queue into one prompt; a channel with a prompt already in flight keeps queueing. Another Tick completes in-flight prompts and returns agents to the pool. Shrink the pool to 1 and watch channels wait.
Buzz ships its own reference agent as a pair: buzz-agent, an ACP agent that handles up to 8 concurrent sessions, and buzz-dev-mcp, an MCP server exposing shell and file-editing tools, with a separate MCP connection per session. The architecture doc's phrase: "Two binaries, two protocols, no coupling." The agent reasons; the tool server acts; either can be replaced without touching the other. (MCP itself — the tools protocol — is its own lesson: MCP. Here it's enough that tools arrive over MCP, sessions are isolated, and the agent side speaks ACP.)
The safety engineering is unglamorous and essential: bounded outputs (a tool can't flood the context), process-group termination on ALL exit paths (no orphaned shells when a session dies — kill the group, not just the child), and context summarization at limits (long sessions compress instead of crashing). And a line worth pinning above any harness you build: "A senior engineer can read both binaries in a sitting. There are no abstractions reserved for future flexibility."
Once agents are first-class participants with keys and channels, a pattern emerges that the Buzz blog treats as the payoff of the whole design: "one frontier agent driving a swarm of cheaper, faster agents."
The division of labor follows the economics of models. The frontier agent (expensive, smart) keeps the big picture in its context: the goal, the constraints, the decisions made so far. The fast agents (cheap, quick) do the parallelizable legwork — research, build, test, review — each with a narrow task and a small context. The frontier agent coordinates them via channel mentions, the same @mention mechanism from Chapter 6 — "without breaking anyone's flow," as the blog puts it. No special orchestration API exists; coordination is just more signed events in the channel.
Think about what the private-prompt alternative costs you. You can't see the swarm going off the rails until it's done. You can't cheaply attribute which worker produced which claim. And when someone asks "why did we pick approach B?" six weeks later, the reasoning is gone — it lived in a context window that no longer exists. Visible orchestration trades some token efficiency for steerability and institutional memory.
Step through a swarm run: the frontier agent F fans work out to three cheap workers via @mentions, workers run in parallel, results come back as channel events, and F synthesizes a decision. Every arrow you see is also a line in the channel log on the right — that's the point. Use the toggle to see what a private-prompt orchestrator would leave visible instead.
Notice how naturally the pieces you already know compose into this: workers are pooled subprocesses (Chapter 6), their messages are signed events with their own identities (Chapter 2), delivery is channel fan-out (Chapter 5), and the whole exchange lands in the audit chain (Chapter 9). The swarm isn't a feature — it's an emergent use of the substrate.
Not every automation deserves an agent. "When someone posts a P1, ping the on-call channel" needs no intelligence — it needs to run every time, in milliseconds, for free. That's buzz-workflow: deterministic automation defined in YAML, triggered by the event stream (pipeline step 12), executing a list of actions.
The vocabulary is deliberately small. Four triggers: message_posted, reaction_added, schedule (a cron loop that ticks every 60 seconds), and webhook (with a constant-time XOR secret comparison, so timing attacks can't probe the secret byte by byte). Seven actions:
| Action | Notes |
|---|---|
send_message | Post into a channel |
add_reaction | React to the triggering event |
call_webhook | SSRF-guarded, follows no redirects, response capped at 1 MiB |
request_approval | Suspends the workflow; issues a single-use UUID token, stored SHA-256-hashed (not yet wired end-to-end) |
delay | At most 300 seconds |
send_dm | Stubbed — not yet implemented |
set_channel_topic | Stubbed — not yet implemented |
Conditions are expressions evaluated by evalexpr (str_contains(...) and friends) under a 100 ms timeout — an adversarially expensive expression gets cut off rather than stalling the trigger loop. Templates like {{trigger.text}} and {{steps.ID.output.FIELD}} are substituted in a single pass: the output of one substitution is never re-scanned, so template injection can't recurse.
Here's the Incident Triage example from the architecture doc, annotated:
yaml — incident triage workflow name: incident-triage trigger: type: message_posted # fires from pipeline step 12 filter: "str_contains(trigger_text, 'P1')" # evalexpr, 100ms budget steps: - id: notify action: send_message channel: "#incidents" text: "P1 reported: {{trigger.text}}" # single-pass template - id: gate condition: "str_contains(trigger_text, 'production')" action: request_approval from: "{{trigger.author}}" prompt: "Page on-call?" # suspends; single-use SHA-256-hashed token
A P1 mention notifies immediately. But if it says "production," the workflow suspends and asks a human before paging anyone. The approval gate is the primitive that lets deterministic automation and agent output alike stop at a human checkpoint: the approval token is a single-use UUID, stored SHA-256-hashed (a database leak reveals no usable tokens). The honest caveat, again from the docs: this gate isn't wired end-to-end yet.
Workflow execution is bounded by a semaphore of 100 permits, acquired with try_acquire. Permit 101 doesn't wait in line — it fails immediately with CapacityExceeded. There is no queue. This is a real position in a classic tradeoff: a queue would smooth bursts but hide overload (workflows silently running minutes late); failing loudly makes overload visible the moment it happens, and a trigger backlog can't quietly grow unbounded behind a full queue.
Setup. Where can a hostile or buggy workflow hurt the relay, and what bounds it?
Runtime: at most 100 concurrent executions (permits); the 101st gets CapacityExceeded, not a queue slot.
Stalling: the slowest legal delay is 300 s; a condition expression gets 100 ms of CPU before it's cut off.
Network: call_webhook can't reach internal addresses (SSRF guard via an is_private_ip check), can't follow redirects to sneak past it, and reads at most 1 MiB = 1,048,576 B of response.
Scheduling: the cron loop ticks every 60 s, so a schedule trigger fires at most once per minute per rule — the floor on self-inflicted load.
Signatures prove who did each thing. But a workspace full of agents also needs to prove what happened, in what order, with nothing quietly removed. That's the hash-chain audit log — pipeline step 11's destination.
Each audit entry's SHA-256 hash covers, in order: the sequence number (big-endian bytes), the RFC3339 timestamp, the event id, the kind (big-endian), the actor's pubkey, the action name, the channel id, the metadata as canonical JSON (a BTreeMap, so keys are always sorted — the same data can never serialize two ways), and — crucially — the previous entry's hash. That last field is the chain: entry N's hash depends on entry N−1's hash, which depends on N−2's, all the way back. Change anything in any old entry and every hash after it stops verifying.
Setup. Three audit entries in channel incident-4021. Genesis prev-hash is 64 zeros. Hashing exactly the fields above (seq as 8-byte big-endian, kind as 4-byte big-endian, metadata {}):
Entry 1 — seq 1, 2026-07-23T10:00:00Z, event 9f1c22ab, kind 9 (chat), actor e3b0a1, EventCreated, prev = 000…0 → hash 73c5b0de…
Entry 2 — seq 2, 10:00:07Z, event 4d80cc17, kind 7 (reaction), actor 77aa02, EventCreated, prev = 73c5b0de… → hash 5b84728d…
Entry 3 — seq 3, 10:01:30Z, event b2e94f60, kind 46001 (workflow), actor e3b0a1, WorkflowTriggered, prev = 5b84728d… → hash c5ffae1e…
Now tamper: flip entry 2's kind from 7 to 9. Recomputing entry 2 gives 36f0cb05… — but entry 3 recorded prev = 5b84728d…. The mismatch is mechanical, unarguable, and propagates: every entry after the edit fails verification. (Run the python below to reproduce every one of these hashes.)
The three entries from the worked example, chained. Tamper with entry 2 and watch verification fail from that point on — the recomputed hash (36f0cb05) no longer matches what entry 3 recorded (5b84728d). These are real SHA-256 values.
python — the audit chain, verifiable end to end import hashlib GENESIS = '0' * 64 def entry_hash(seq, ts, event_id, kind, actor, action, channel, meta, prev): m = hashlib.sha256() m.update(seq.to_bytes(8, 'big')) # sequence, big-endian m.update(ts.encode()) # RFC3339 timestamp m.update(event_id.encode()) m.update(kind.to_bytes(4, 'big')) # kind, big-endian m.update(actor.encode()); m.update(action.encode()); m.update(channel.encode()) m.update(meta.encode()) # canonical BTreeMap JSON — keys always sorted m.update(prev.encode()) # THE CHAIN: previous entry's hash return m.hexdigest() h1 = entry_hash(1, '2026-07-23T10:00:00Z', '9f1c22ab', 9, 'e3b0a1', 'EventCreated', 'incident-4021', '{}', GENESIS) h2 = entry_hash(2, '2026-07-23T10:00:07Z', '4d80cc17', 7, '77aa02', 'EventCreated', 'incident-4021', '{}', h1) h3 = entry_hash(3, '2026-07-23T10:01:30Z', 'b2e94f60', 46001, 'e3b0a1', 'WorkflowTriggered', 'incident-4021', '{}', h2) print(h1[:8], h2[:8], h3[:8]) # 73c5b0de 5b84728d c5ffae1e # tamper: entry 2's kind 7 -> 9. Recomputed hash no longer matches entry 3's recorded prev. h2_tampered = entry_hash(2, '2026-07-23T10:00:07Z', '4d80cc17', 9, '77aa02', 'EventCreated', 'incident-4021', '{}', h1) print(h2_tampered[:8]) # 36f0cb05 != 5b84728d => chain broken from entry 2 onward
Two engineering details make the chain trustworthy in production. Single writer: a chain has one "latest hash," so appends must serialize — Buzz uses a Postgres advisory lock (pg_advisory_lock) with a panic-safe release wrapped in catch_unwind, so a crashing writer can't leave the lock held forever. Scope: there are 10 audit action types (EventCreated … RateLimitExceeded), and two things are deliberately never logged: AUTH handshakes and ephemeral events. Presence blips would bloat the chain with zero forensic value — audit what matters, chain what you audit.
The same trust thinking reshapes code hosting. Traditional forges assume human-paced pushes: a few writers, occasional contention, a mutable server-side repo. Agents break the assumption — the doc describes swarms producing "human-months of commits and CI runs in an afternoon, with many writers pushing at once."
Buzz's design: store repositories as immutable, content-addressed packfiles plus exactly ONE mutable thing — a manifest pointer. A push uploads new packfiles (pure adds, no conflicts possible), then performs a conditional compare-and-swap on the pointer: "replace manifest M3 with M4 only if it's still M3." Two agents racing? One CAS wins; the loser sees the new manifest, rebases, retries. No lock server, no corrupted repo state — the entire mutable surface is one pointer.
The Nostr layer stays advisory by design: "workspace events announce the change; they do not define it" — object storage is the truth, events are the notification. The storage protocol is specified in TLA+ and model-checked for durability, reconstruction, and concurrent pushes — the CAS races are exactly the kind of bug testing misses and model checking catches. Tooling rounds it out: git-sign-nostr and git-credential-nostr bind commits and credentials to the same keys as everything else, and NIP-34 carries git events.
You now know every mechanism separately. This chapter runs them together: a relay with clients on the left, the pipeline in the middle, and the #incident channel plus counters on the right. Every button exercises a different chapter of this lesson — break things and watch the machinery respond.
chat: a signed kind-9 event runs the full pipeline into the channel (Ch 4). presence: branches to the ephemeral route — note the audit counter does NOT move (Ch 1, 4). forged event: dies at the pubkey gate (Ch 4). @mention swarm: the frontier agent dispatches workers, all visible in the log (Ch 6–7). slow client: fills its send buffer three times and gets cancelled (Ch 3). P1 in production: a workflow suspends on an approval gate; press again to approve (Ch 8).
Things worth trying: publish a chat, then a presence, and compare the counters — stored and audited move for one, not the other. Fire the swarm and count the log lines: mention, three dispatches, three results, one synthesis — eight signed events where a private orchestrator would have shown you one. Cancel the slow client mid-swarm and notice nobody else's delivery stalls: its buffer was its own.
Buzz is a gift for system-design interviews: it takes positions on event logs, identity, backpressure, pub/sub security, multi-agent scheduling, and tamper-evidence — and every position has a defensible "why" and an honest cost. Eight questions with staff-grade answers, then three debates to argue both sides of.
The conventional stack has three coordination systems with three identity models and three audit stories; every feature that crosses them (an agent reacting to CI, a workflow gating a deploy) needs glue code and loses attribution at each seam. One signed event log collapses that: a single ordered history, a single identity primitive (the signing key), a single audit trail covering humans and processes alike. Extension is a new kind number — old clients ignore it, nothing migrates. Idempotency comes free from content-addressed ids. The costs, stated honestly: the relay is a single point of failure and a scaling chokepoint (you shard by community, not by feature); every consumer must speak the event vocabulary rather than a bespoke convenient API; ephemeral traffic must be explicitly carved out (kind ranges) or the log drowns in presence blips; and cryptography on every message costs CPU you must engineer around (Buzz: spawn_blocking after cheap gates).
Follow-up: When would you NOT choose this design? (Answer: when participants don't share a trust domain or the workload is dominated by high-volume ephemeral data — a game server, say — where signed durable events buy little and cost much.)
Three separable facts, separately verifiable: (1) the agent holds its OWN keypair — its actions are signed by it, forever attributable to it; (2) the owner signs a narrowly scoped authorization naming what that key may do — the grant is itself a signed artifact; (3) every piece of work the agent produces carries the agent's signature, not the owner's. Buzz's phrase: "authorization does not erase authorship." The operational payoffs: a leaked agent key is revoked without touching the human's identity or any other agent's; scope is per-agent, so a research agent never holds deploy rights; and "which of my five agents did this?" has a cryptographic answer. Contrast with OAuth-style service accounts, where the bearer token conflates who authorized with who acted — revocation is rotation, and attribution is whatever the logs say.
Follow-up: The agent needs to act while its owner is asleep. Does this model still work? (Answer: yes — that's the point. The standing authorization is already signed; the agent authenticates as itself via NIP-42 whenever it acts. Time-bounding or scope-narrowing the grant is a policy choice on top.)
Order the pipeline by cost and trust: cheap authz gates first (scope, pubkey-match, kind sanity), then expensive signature verification off the async runtime (spawn_blocking), then membership, then the durable insert — idempotent via ON CONFLICT DO NOTHING on the content-addressed id, so client retries are safe by construction. Acknowledge AFTER insert + fan-out; run search indexing, audit append, and workflow triggers as an async fire-and-forget tail. The boundary sits there because publish latency is user-facing while the tail is derived state: a crash in the tail loses an index entry or a trigger for an event that IS stored — recoverable by replay — whereas acknowledging before the insert could lose the event itself, which is not. Cap total concurrency with a global handler semaphore (Buzz: 1,024 permits) so load cannot translate into unbounded task spawn.
Follow-up: The audit append in the tail fails silently — is your audit chain now lying? (Answer: it has a gap, not a lie — the chain still verifies over what it contains. Closing the gap means either moving audit inside the durability boundary, paying latency, or reconciling audit against the event table asynchronously. Say the tradeoff out loud.)
Three leaks, three closures. Leak 1: treating filter matching as authorization — a global subscriber types a filter matching a private channel's events and receives them. Closure: a hard boundary rule — channel-scoped events are delivered ONLY to subscriptions registered by verified members; global subscriptions never receive them regardless of filter. Leak 2: the registration race — if REQ registers the subscription and then checks membership, live events flow in the gap between. Closure: check membership BEFORE registering; an unauthorized subscription never exists. Leak 3: semantic edge cases in the filter language — NIP-01's kinds:[] matches nothing while an absent kinds key matches everything; confuse them and you either silence a legitimate subscriber or firehose an illegitimate one. Closure: encode the edge in one shared filters_match implementation with tests, never inline per-handler logic.
Follow-up: A member's subscription outlives their membership — they're removed from the channel while subscribed. (Answer: revocation must deregister live subscriptions, not just fail future REQs — membership changes push into the SubscriptionRegistry, another argument for one registry rather than scattered checks.)
Never let one consumer's slowness become shared memory pressure. Layer it: (1) admission control — a connection semaphore that rejects over-limit connections BEFORE reading any bytes, so overload costs no parse work; (2) a global handler semaphore bounding total in-flight work across all connections; (3) per-client bounded send buffers written with try_send — never a blocking send into an unbounded queue; (4) an explicit eviction rule: Buzz cancels a client after 3 consecutive full-buffer sends, alongside liveness pings every 30s with disconnect at 3 missed pongs (both "rules of three," 90s worst case). The key property: the failure mode is per-client disconnection with reconnect-and-re-REQ recovery, not global degradation. The alternative — buffering for the slow client — has a worst case you can compute: 1,024 subscriptions × 64 KiB frames = 64 MiB for ONE misbehaving connection.
Follow-up: The cancelled client was mid-download of 500 historical events — what does recovery look like? (Answer: REQ is stateless and replayable; on reconnect it re-issues the filter and pages again. Statelessness of reads is what makes brutal cancellation acceptable.)
Chain entries: each entry's hash covers its fields AND the previous entry's hash, so editing any historical entry breaks every subsequent hash — tampering is mechanically detectable by re-walking the chain. Three details make it production-real. First, canonical serialization: hash bytes must be deterministic, so fixed-width big-endian integers and sorted-key (BTreeMap) JSON for metadata — the same data can never hash two ways. Second, the chain has ONE latest hash, so appends must serialize: take a single writer via advisory lock (pg_advisory_lock), with panic-safe release (catch_unwind) so a crashed writer can't wedge the log. Concurrency stays upstream — the async pipeline feeds one appender. Third, scope: decide what is deliberately NOT logged (Buzz: AUTH handshakes and ephemeral events) so volume stays proportional to forensic value. Note what this is not: tamper-PROOF requires anchoring the head hash externally; tamper-EVIDENT is what the chain gives you.
Follow-up: Auditor wants to verify entry 500,000 without walking from genesis. (Answer: periodic signed checkpoints of (seq, hash) — verify from the nearest checkpoint. Chains compose with checkpoints exactly like git commits with tags.)
Forges embed a concurrency assumption: pushes are human-paced — a few writers, rare contention, server-side mutable repo state guarded by locks. A swarm violates it, producing (Buzz's words) human-months of commits and CI in an afternoon with many simultaneous writers. The redesign inverts mutability: store packfiles as immutable, content-addressed blobs — uploads are pure adds, conflict-free by construction — and shrink the mutable surface to ONE manifest pointer advanced by conditional compare-and-swap ("set to M4 only if still M3"). Losers of a CAS race see the new manifest, rebase, retry; there is no partially-mutated repo state to corrupt, ever. Keep the event layer advisory — "events announce the change; they do not define it" — so notification and truth never conflict. And because CAS races are exactly where testing fails, specify the protocol in TLA+ and model-check durability, reconstruction, and concurrent pushes.
Follow-up: Doesn't CAS retry starve under heavy contention? (Answer: yes, livelock is the known cost — mitigate by batching pushes per-writer or sharding the pointer per-branch. Naming the failure mode unprompted is the staff move.)
Parallel prompts on one conversation give you N agents with N partial views racing to reply — interleaved, mutually contradictory, token-expensive, and impossible to attribute coherently. Batching (buzz-acp's policy: queue mentions per channel, batch the whole queue into ONE prompt, at most one in flight per channel) restores the invariant that a conversation is serial: each response sees ALL accumulated context and produces one coherent reply. The pool (1–32 subprocesses) then provides parallelism across channels, where it's safe. Costs, honestly: latency — a mention landing mid-prompt waits for completion, so worst-case first-response time is one full prompt duration; and a batch conflates unrelated asks in a busy channel, which argues for finer-grained channels, not parallel prompts. The deeper principle: concurrency boundaries should match conversation boundaries, because that's the unit over which context must stay coherent.
Follow-up: A user posts "URGENT: stop the deploy" while a prompt is in flight. (Answer: batching alone fails this — you need a priority lane: cancel/interrupt the in-flight prompt or run an out-of-band interrupt classifier. Acknowledging the policy's limit is part of the answer.)
Architecture is only interesting for what it makes possible. The Buzz blog closes with four applications, each a direct consequence of mechanisms you now know:
| Application | What happens | Which mechanism pays for it |
|---|---|---|
| Incident memory | Search "auth refresh" months later and get the report, the rejected fix, the patch, the review, and the final call — reasoning preserved next to artifacts | One log + FTS index (Ch 3–4); signed history (Ch 1) |
| Branch-as-room | Each feature or bug gets a short-lived channel: discussion, patches, CI runs, and signed merge decisions in one audit trail | Channels + membership (Ch 5); git events (Ch 9) |
| Automated releases | A workflow reads merged PRs, drafts notes, waits at an approval gate, ships — all signed and searchable | Workflows + approval gates (Ch 8) |
| Agent swarms | Frontier agent + cheap workers, coordinated in the open | buzz-acp + visible orchestration (Ch 6–7) |
Beyond text, there's huddle audio: a WebSocket endpoint per channel (/huddle/{channel}/audio) that NIP-42-authenticates each participant, checks membership, and forwards opaque Opus frames — an 8-byte big-endian header (sequence, 48 kHz timestamp, level in dBov, flags) plus payload. Rooms are in-memory, soft-capped at 25 peers (hard cap 255), with bounded per-peer channels that drop on full — the same backpressure philosophy as Chapter 3. Join/leave/end are lifecycle Nostr events; recording isn't built yet (kinds reserved).
The architecture doc keeps a public list of what doesn't work yet, and it's worth reading as engineering culture: no production rate limiter (the four tiers are designed, with a trait and test stub, but unenforced); sqlx queries are runtime-only (no compile-time query validation); approval gates aren't wired end-to-end; send_dm and set_channel_topic are stubbed; huddle recording is absent; there's no typing query endpoint. A system that documents its own gaps is telling you its architecture docs can be trusted about the rest.
"What I cannot create, I do not understand." — Feynman's rule is the test this lesson set for itself: you should now be able to re-implement the core — the event, the pipeline, the fan-out, the chain — from memory. If a step feels fuzzy, the simulator in Chapter 10 will tell you which chapter to reread.