Record every tool call at its nondeterminism boundary. Replay it later, deterministically, with every internal variable observable. A testing approach — and the philosophy it fell out of.
github.com/xag/flight-recorder
01 · Why
An app's surface shows results, never reasons. Probe it from outside — even carefully — and every "bug" you report is an inference about invisible internals.
A black-box "exercise session" against a production app reported 4 bugs — and 2 were artifacts of the probe itself: the wrong client, the wrong assumptions, a story invented around surfaces.
Discipline doesn't fix this. The question "why did it answer that?" has to become answerable — a lookup, not an abduction.
02 · The idea
Physical systems can be simulated without being built because they run on physics: laws + state + boundary conditions ⇒ the whole trajectory is derivable. You never store the field at every point — you recompute it.
Software has the same structure, term for term:
Record only the boundary — the entire execution becomes reconstructible on demand.
03 · The mechanism
Recording is cheap enough to leave on in production: bound arguments, the ordered boundary events, the result.
{"ev":"call", "fn":"get_status", "kwargs":{"email":"…","max_words":10},
"events":[
{"k":"db", "op":"get", "sig":"collection(\"users\")…profile…", "res":{…}},
{"k":"db", "op":"stream", "sig":"…words.select([\"norm\"])…", "res":[ 5468 docs ]},
{"k":"now", "v":"2026-07-08T23:25:44+00:00"},
…
],
"result":"=== KOREAN COACH — USER STATUS ===\n…"}
Replay re-executes the call on the real code, feeding these answers back in order — under a tracer. One recorded production call yielded 14,157 state transitions, every variable change queryable after the fact.
04 · The cardinal rule
Nothing in the recorder evaluates a query, computes a date, or knows what any value means. The only structural knowledge anywhere is names.
A transparent proxy: forwards every call to the real client unchanged, logs what crossed.
Feeds recorded answers back and verifies the code asks the same questions in the same order. Anything else is a named divergence.
Deliberately semantics-free: canned answers, no query logic. Round-trip fidelity never depends on a fake being faithful — so there's no fake to drift.
Corollary: the boundary declaration is the whole maintenance contract. The recorder can't know about an input it was never told crosses it.
05 · Case study
A production status message claimed the learner had "reached the end of the frequency corpus" — on an account that was never placed. Guessing produced theories. One replay produced the answer:
$ python -m app.replay flight/prod.jsonl --call 0 --watch level,deck_size,to_introduce tools_core.py:138 get_status level = 0 tools_core.py:141 get_status deck_size = 5468 service.py:1227 corpus_to_introduce known_rank = 0 tools_core.py:146 get_status to_introduce = []
The deck held exactly the full 5,467-word corpus plus one extra word: 테스트디버그 — "test-debug". A seeding script had filled a real account and signed its work. Root cause read straight out of the trace, then confirmed against the repo's own scripts.
06 · The workflow
Every confirmed bug gets its recording pinned before the fix. The fix is then proven twice:
14/14 boundary events consumed — same code path — and the diff contains exactly the intended change and nothing else. The bug's world no longer exists in production; the recording keeps it testable forever.
Bit-for-bit identical. No behavioral drift anywhere the fix didn't intend to reach.
A fix proven against captured reality, not against a hand-written test's imagination.
07 · Case study
A vocabulary coach. Its corpus decides what the deck teaches; its reader, what a learner is credited with. Nothing checked that the two agree. Nothing raised.
Every tape said MATCH: the bug was in what the code meant, not what it did. Only a claim about every execution catches that — every word the deck teaches, the reader reads back. 522 unreachable words: the deck taught 듣다, to listen; the reader credited 들다 — to lift.
The obvious fix satisfied the invariant and broke a recorded scenario. One call, one word:
- 보다 (#672) to see + 보고 (#2342) a report
Reverted, and narrowed. An invariant cannot catch a regression it never thought to forbid.
With the real fix in, another tape diverged — it had faithfully recorded the wrong answer. A recording is evidence, not a spec. Re-recorded deliberately; the diff held the intended change and nothing else.
It now runs in CI over six languages, ~32,000 words — and caught a fresh unreachable word in each of the three corpora added after it, before any shipped.
No human intervened: bug found, the fix's own regression caught, tapes re-recorded, three languages added — overnight. The human read the diff and said ship it.
08 · Without the recorder
Testing hangs on a person at the controls — run it, reproduce it, re-run it all after every fix. The AI can’t take this over: without instruments it can only read the source and theorize. Costly, and still inference.
09 · With the recorder
Given instruments, the AI carries the process — the code and the scenarios, the runs, the diagnosis, the proof. Recordings show every internal, replay answers with verdicts, and pinned sessions re-verify every build with no human and no tokens. The human contributes ideas and judgement.
10 · Honesty
The replay layer is real physics — but its laws stop at the process boundary. Know which instrument sees what.
| Layer | What lives there | Instrument |
|---|---|---|
| client | widgets, OAuth flows, device quirks, the feel under a thumb | real clients, occasionally — the sim-to-real calibration |
| logic | your code between the membranes | the flight recorder — record, replay, trace |
| machine | memory, latency, concurrency, the kernel | logs and measurement |
Proven the hard way: an OOM-killed call was invisible to replay —
the kernel log and an RSS measurement (38 MB → 543 MB on a 512 MB VM) caught what the recorder
structurally could not. Since then, .inflight sidecars stream events during the
call, so even a killed process leaves its last words.
The client layer has one recordable sliver: a round-trip the tool awaits — an elicitation, a sampled answer, a UI response surfaced as a session method — enters the process as a call, so declaring that method records it like any other input. Interaction that never re-enters the tool's execution stays invisible by construction: a session replays from the server's side of the conversation, not the clicks between.
11 · Production safety
A recording holds what crossed the boundary — verbatim. For boundaries that carry
credentials or PII, redact names the fields that must never reach
disk or a sink.
BOUNDARY = fr.Boundary(
effects=[...],
# masked wherever these keys appear
redact={"password", "ssn"},
)
Rules apply before a value is serialized into an event: the session file, the crash sidecars, and anything a sink publishes never hold the raw value. The per-call gate still sees raw kwargs — you can admit a call by the very field the recording then masks.
The same rules are re-applied to the replayed side of every comparison, so a redacted recording still matches bit-for-bit — even a secret born as a literal inside the code.
A redacted field replays as its mask: redact what flows through unchanged; give what must stay distinct a deterministic, idempotent tokenizer instead. And what names cannot reach — positional values, chain signatures — is documented, never silent.
12 · The discipline, per repo
13 · Proof of generality
The engine (flight-recorder, zero dependencies, zero consumer knowledge) was extracted at its second consumer and hardened by each next one.
| App | Its boundary | What its boundary demanded |
|---|---|---|
| korean-coach | Firestore chains · clock · random | the chain proxy, the shims |
| dev-tools | GitHub + feedback stores (async HTTP) | async effects, exception revival |
| invest | JSONL journal · market feeds · paper broker | method effects, secret-free seams |
| home | multi-user files · identity · pandas clock | registry-level wrapping, loose args |
| reads (Node) | Redis · outbound mail · HTTP · serverless | proxy-wrapped boundaries, an off-box sink, ask-order events |
Each app: one boundary declaration plus whatever membrane shaping the code was owed. Recording runs in production where it matters — including, in the fifth, a serverless host with no filesystem and no shell.
14 · Six runtimes
Six implementations — Python, Node, .NET, Go, Java, PHP — and one tape. Only record and replay are language-bound: replaying JavaScript means running JavaScript. Invariants and mutation consume the tape, and a tape is only data: freeze the data and the analysis is written once, for every language.
Two of the six, side by side — the rest differ just as sharply, and for the same kind of reason:
| Python | Node | |
|---|---|---|
| Boundary | name module functions; patch in place | wrap what the app holds — namespaces are immutable |
| Randomness | sample — the positions drawn | bytes / float / int — the draw is the value |
| Nothing | one — None | two — undefined earns a marker |
| The sink | hand the bytes to a queue | hand the promise to the host — waitUntil |
| The tracer | sys.settrace — every local, every line | the V8 Inspector, driven from a worker |
Not preferences — consequences. Every difference is forced by the language.
The format is frozen (spec/tape-v1.md) and its checker written six times, independently —
every language validating every other's fixtures. A disagreement means the tape has forked.
15 · What it opens
Because replay feeds the code any boundary answers, hostile states are edited recordings — not databases you have to construct.
Property-based testing over the boundary — reaching states honest use would never produce.
16 · The meta-rule
The instruments exist so that a claim is a trace line, a log line, or a measurement — never a plausible story. That rule, applied in every repo, is worth more than any of the code.
github.com/xag/flight-recorder · record → pin → fix → replay-diff · MATCH