flight-recorder

Record an app's tool calls at their nondeterminism boundary; replay them deterministically against the real code.

A program's execution is fully determined by its code plus its nondeterministic inputs — what the store answered, what the API returned, what time it was, what the dice rolled. Record just those, per call: one cheap JSONL line. That line is the execution, compressed. Feed the answers back and the real code re-runs the original execution exactly — no network, no database, no waiting for the bug to happen again.

Install

pip install xag-flight-recorder

npm install @xag/flight-recorder

dotnet add package flight-recorder

go get github.com/xag/flight-recorder/go@v0.8.0

<dependency>
  <groupId>io.github.xag</groupId>
  <artifactId>flight-recorder</artifactId>
  <version>0.1.0</version>
</dependency>

composer require poietic/flight-recorder

Six implementations, one tape — format v1, frozen in spec/tape-v1.md. Only record and replay are language-bound: replaying JavaScript means running JavaScript. Everything downstream consumes the tape, and a tape is only data.

The approach

1. Name the doors. Declare, once per app, the handful of places the outside world enters. That declaration is the boundary, and it is the only app-specific artifact. Nothing behind the doors is ever imitated or mocked; real code runs everywhere.
2. Record what came through. Each call writes one line: its inputs, every answer the world gave it in the order it was asked, and its result. Cheap enough for production — and complete, because the code is deterministic given those answers.
3. Replay is resurrection, not re-enactment. The recorded answers are fed back and the real code re-runs. If it asks the world a different question than the recording holds, you are told precisely where behaviour changed.
4. Recordings answer "same?", invariants answer "right?". A pinned recording is a regression oracle — but a bug records as faithfully as a fix, so no recording can call the first observation of a bug wrong. Only a claim about every execution can.
5. Edit the tape to visit worlds that never happened. A recording is data, so hostile states are one edit away — no test database that can produce impossible states on demand.

Declare the boundary

The one app-specific artifact. The recorder cannot know about an input it was never told crosses the boundary; when the app grows a new one, add it here. That is the whole maintenance contract.

import flight_recorder as fr
from app import http_client, storage_client, tools_core

BOUNDARY = fr.Boundary(
    effects=[(http_client, ["fetch", "post"]),      # module functions, sync or async
             (storage_client, ["read", "write"])],
    chains=[fr.ChainTarget(svc, "db")],              # chained clients (Firestore-style)
    clock_modules=[tools_core],                      # modules whose datetime.now matters
    random_modules=[storage_client],                 # modules whose random matters
    constants=[(tools_core, "SOME_ENV_CAP")],        # env-derived, header-captured
    error_revivers={"ApiError": lambda args: http_client.ApiError(*args)},
)

import * as fr from '@xag/flight-recorder';

export const BOUNDARY = fr.boundaryOf({
  constants: { 'config.LIMIT': LIMIT },
  errorRevivers: { ApiError: ([msg]) => new ApiError(msg) },
});

// Wrap what the app HOLDS. A transparent proxy — everything not named
// passes straight through, untouched and unrecorded.
export const store = fr.wrap(storageClient, ['read', 'write'], { prefix: 'kv' });
export const http  = fr.wrap(httpClient,    ['fetch', 'post'], { prefix: 'http' });

using FlightRecorder;

var boundary = new Boundary().MaskFields("password");     // field-name redaction
boundary.Constants["Config.Limit"] = LIMIT;               // header-captured
boundary.ErrorRevivers["ApiError"] = args => new ApiError((string)args[0]!);

// Wrap what the app HOLDS — a transparent proxy over an interface, never a mock.
// Everything not named passes straight through, untouched and unrecorded.
IStore store = Recorder.WrapAs(storageClient, "kv", nameof(IStore.Read), nameof(IStore.Write));

import (
	fr "github.com/xag/flight-recorder/go"
	"github.com/xag/flight-recorder/go/serial"
)

boundary := fr.Boundary{
	Constants: map[string]any{"config.LIMIT": LIMIT},          // header-captured
	Redact:    serial.Rules{"password": nil},                  // field-name redaction
	Forbid:    []string{`-----BEGIN [A-Z ]*PRIVATE KEY-----`}, // the tripwire redaction can't name
}

import io.github.xag.flightrecorder.*;

Boundary boundary = new Boundary()
    .constant("config.LIMIT", LIMIT)                        // header-captured
    .maskFields("password")                                 // field-name redaction
    .forbidden("-----BEGIN [A-Z ]*PRIVATE KEY-----")        // the tripwire redaction can't name
    .reviving("ApiError", args -> new ApiError((String) args.get(0)));

// Wrap what the app HOLDS — a transparent proxy over an interface, never a mock.
// Everything not named passes straight through, untouched and unrecorded.
Store store = Recorder.wrapAs(Store.class, storageClient, "kv", "read", "write");

use Xag\FlightRecorder\{Boundary, Recorder};

$boundary = (new Boundary())
    ->constant('Config::LIMIT', Config::LIMIT)              // header-captured
    ->maskFields('password')                                // field-name redaction
    ->forbidden('/-----BEGIN [A-Z ]*PRIVATE KEY-----/')     // the tripwire redaction can't name
    ->reviving('ApiError', fn (array $a) => new ApiError($a[0]));

// Wrap what the app HOLDS — a transparent decorator, never a mock.
// Everything not named passes straight through, untouched and unrecorded.
$store = Recorder::wrapAs('kv', $storageClient, 'read', 'write');

The clock and the RNG hold nothing to wrap in any runtime — Python and Node shim the globals, .NET, Go, Java and PHP read them through recorder handles.

Record

fr.install(BOUNDARY, tools_core, directory="flight",
           enabled=bool(os.getenv("MYAPP_FLIGHT_RECORDER")))

export const study = fr.tool('study_status', studyStatus);   // the call boundary

fr.install(BOUNDARY, {
  directory: '.flight',
  enabled: process.env.FLIGHT === '1',
});

Recorder.Install(boundary, directory: ".flight",
                 enabled: Environment.GetEnvironmentVariable("FLIGHT") == "1");

// Wrap each tool call — that line IS the execution. `RecordAsync` is the Task form.
object? StudyStatus(string user) => Recorder.Record("study_status", new { user }, () =>
{
    var doc = store.Read(user);            // fx
    var at = Recorder.Clock.Now();         // now
    return Compute(doc, at);
});

rec, _ := fr.New("flight", boundary)   // lazy: the first admitted call opens the file
defer rec.Close()

// Wrap each tool call — that line IS the execution.
result, err := rec.Call(ctx, "study_status", map[string]any{"user": user},
	func(ctx context.Context) (any, error) {
		doc, err := fr.Effect(ctx, "kv.read", []any{user},
			func() (map[string]any, error) { return store.Read(user) })   // fx
		at := fr.Now(ctx)                                                  // now
		return compute(doc, at), err
	})

Recorder rec = Recorder.open(".flight", boundary);   // lazy: the first admitted call opens the file

// Wrap each tool call — that line IS the execution.
Object result = rec.call("study_status", Recorder.kwargs("user", user), () -> {
    Row doc = store.read(user);              // fx, through the wrapped client
    LocalDateTime at = Recorder.now();       // now
    return compute(doc, at);
});

$rec = Recorder::open('.flight', $boundary);   // lazy: the first admitted call opens the file

// Wrap each tool call — that line IS the execution.
$result = $rec->call('study_status', ['user' => $user], function () use ($store, $user) {
    $doc = $store->read($user);          // fx, through the wrapped client
    $at  = Recorder::now();              // now
    return compute($doc, $at);
});

Off by default. When enabled, each call writes one line — bound args, ordered boundary events, result.

Record one call, not one deployment

A bool decides once for the whole process. A gate is consulted on every call, so a running server can record a single user's request and leave the rest of its traffic untouched — no env flip, no redeploy.

RECORDING_FOR = contextvars.ContextVar("recording_for", default=None)

fr.install(BOUNDARY, tools_core, directory="flight",
           enabled=lambda tool, kwargs: kwargs.get("email") == RECORDING_FOR.get())

const WRITES = new Set(['set_level', 'record_answer']);

fr.install(BOUNDARY, {
  directory: '.flight',
  gate: (fn, args) => WRITES.has(fn),   // record what matters, not everything
});

var writes = new HashSet<string> { "set_level", "record_answer" };

Recorder.Install(boundary, directory: ".flight",
                 gate: (fn, kwargs) => writes.Contains(fn));   // record what matters

writes := map[string]bool{"set_level": true, "record_answer": true}

rec, _ := fr.New("flight", fr.Boundary{
	Enabled: func(fn string, kwargs map[string]any) bool { return writes[fn] },   // record what matters
})

Set<String> writes = Set.of("set_level", "record_answer");

Boundary boundary = new Boundary()
    .enabledWhen((fn, kwargs) -> writes.contains(fn));   // record what matters, not everything

$writes = ['set_level', 'record_answer'];

$boundary = (new Boundary())
    ->enabledWhen(fn (string $fn, array $kwargs) => in_array($fn, $writes, true));   // record what matters

A gate that never fires leaves no session file at all. A gate that raises is treated as a "no": it can never break the call it was asked about.

Retrieve without touching the box

Pass a sink and the session is published as it grows — after the header, then after every completed call — so recordings are retrievable from a machine you have no shell on. The protocol is one method, publish(name, data).

class S3Sink:
    def __init__(self):
        self.q = queue.SimpleQueue()
        threading.Thread(target=self._drain, daemon=True).start()

    def publish(self, name: str, data: bytes) -> None:
        self.q.put((name, data))       # hand off and return; never block the caller

    def _drain(self):
        while True:
            name, data = self.q.get()
            boto3.client("s3").put_object(Bucket="flight", Key=name, Body=data)

fr.install(BOUNDARY, tools_core, directory="flight", sink=S3Sink())

import { waitUntil } from '@vercel/functions';

fr.install(BOUNDARY, {
  directory: null,        // a serverless filesystem dies with the invocation: the sink IS the tape
  sink: { publish: (name, text) => store.set(`flight:${name}`, text, { ex: 7 * 24 * 3600 }) },
  defer: waitUntil,       // the response leaves now; the tape still lands
  sinkTimeoutMs: 3000,
});

class KvSink : ISink   // one method: handed the WHOLE session text each time
{
    public void Publish(string name, string text) => _kv.Set($"flight:{name}", text, ttl: TimeSpan.FromDays(7));
}

Recorder.Install(boundary, directory: null, sink: new KvSink());   // the sink IS the tape

type kvSink struct{ kv *redis.Client }   // one method: handed the WHOLE session each time

func (s kvSink) Publish(name string, data []byte) {
	s.kv.Set(ctx, "flight:"+name, data, 7*24*time.Hour)   // hand off and return; never block
}

rec, _ := fr.New("flight", fr.Boundary{Sink: kvSink{kv}})   // the sink IS the tape

// One method, handed the WHOLE session text each time — so an overwriting store is enough
// and a tape is never half-published.
Boundary boundary = new Boundary()
    .publishingTo((name, text) -> kv.setex("flight:" + name, Duration.ofDays(7), text));

Recorder rec = Recorder.open(".flight", boundary);   // hand off and return; never block

// One method, handed the WHOLE session text each time — so an overwriting store is enough
// and a tape is never half-published.
$boundary = (new Boundary())
    ->publishingTo(fn (string $name, string $text) => $redis->setex("flight:$name", 604800, $text));

$rec = Recorder::open('.flight', $boundary);   // hand off and return; never block

Use a client the recorder does not wrap, or the sink records itself.

Redact before anything leaves the process

Recordings hold what crossed the boundary, verbatim — which for boundaries carrying credentials or personal data is the problem.

BOUNDARY = fr.Boundary(
    effects=[...],
    redact={"password", "ssn"},                      # by FIELD NAME
    scrub=lambda s: ADDRESS.sub(hide, s),            # by VALUE: every string, anywhere
    forbid=[r"\b[a-f0-9]{64}\b",                     # THIS TAPE CARRIES NO CREDENTIAL:
            r"-----BEGIN [A-Z ]*PRIVATE KEY-----"],  # a hit raises, and writes nothing
)

# …or map a field to a deterministic tokenizer, keeping distinctness without the value:
redact={"email": lambda v: v if str(v).startswith("tok:") else "tok:" + hmac_hex(v)}

fr.boundaryOf({
  redact: { password: null, ssn: null },      // by FIELD NAME
  scrub: (s) => s.replace(ADDRESS, hide),     // by VALUE: every string, anywhere
  forbid: [/\b[a-f0-9]{64}\b/,                // THIS TAPE CARRIES NO CREDENTIAL:
           /-----BEGIN [A-Z ]*PRIVATE KEY-----/],  // a hit raises, and writes nothing
});

var boundary = new Boundary().MaskFields("password", "ssn");   // by FIELD NAME

// …or map a field to a deterministic tokenizer, keeping distinctness without the value:
boundary.Redact["email"] = v => ((string)v!).StartsWith("tok:") ? v : "tok:" + HmacHex(v);

boundary.Scrubbing(@"\d{3}-\d{2}-\d{4}");                      // by VALUE: every string, anywhere

// `forbid` states what a field rule cannot: THIS TAPE CARRIES NO CREDENTIAL. Each pattern is
// matched against the fully-redacted line; a hit raises ForbiddenValue and writes nothing.
boundary.Forbidden(@"\b[a-f0-9]{64}\b")                        // a scrypt digest survived redaction
        .Forbidden("-----BEGIN [A-Z ]*PRIVATE KEY-----");

boundary := fr.Boundary{
	Redact: serial.Rules{                                       // by FIELD NAME
		"password": nil,                                        // nil rule → [REDACTED]
		"email":    func(v any) any { return "tok:" + hmacHex(v) },  // …or a tokenizer
	},
	Scrub:  func(s string) string { return addressRe.ReplaceAllString(s, hide) }, // by VALUE
	Forbid: []string{`\b[a-f0-9]{64}\b`,                        // THIS TAPE CARRIES NO CREDENTIAL:
		`-----BEGIN [A-Z ]*PRIVATE KEY-----`},                 // a hit raises, and writes nothing
}

Boundary boundary = new Boundary()
    .maskFields("password", "ssn")                       // by FIELD NAME
    // …or map a field to a deterministic tokenizer, keeping distinctness without the value:
    .redacting("email", v -> ((String) v).startsWith("tok:") ? v : "tok:" + hmacHex(v))
    .scrubbing("\\d{3}-\\d{2}-\\d{4}")                   // by VALUE: every string, anywhere
    // `forbidden` states what a field rule cannot: THIS TAPE CARRIES NO CREDENTIAL. Each pattern
    // is matched against the fully-redacted line; a hit raises ForbiddenValue and writes nothing.
    .forbidden("\\b[a-f0-9]{64}\\b")                     // a scrypt digest survived redaction
    .forbidden("-----BEGIN [A-Z ]*PRIVATE KEY-----");

$boundary = (new Boundary())
    ->maskFields('password', 'ssn')                      // by FIELD NAME
    // …or map a field to a deterministic tokenizer, keeping distinctness without the value:
    ->redacting('email', fn ($v) => str_starts_with($v, 'tok:') ? $v : 'tok:' . hmacHex($v))
    ->scrubbing('/\d{3}-\d{2}-\d{4}/')                  // by VALUE: every string, anywhere
    // `forbidden` states what a field rule cannot: THIS TAPE CARRIES NO CREDENTIAL. Each pattern
    // is matched against the fully-redacted line; a hit throws ForbiddenValue and writes nothing.
    ->forbidden('/\b[a-f0-9]{64}\b/')                    // a scrypt digest survived redaction
    ->forbidden('/-----BEGIN [A-Z ]*PRIVATE KEY-----/');

Rules must be idempotent. Replay re-derives the question it is about to ask, scrubs it the same way, and compares against the tape — so a value that is already a mask has to scrub to itself, or a redacted recording could never be replayed. A rule that raises degrades to the mask: the failure direction is masked, never leaked, and never broke the recorded call.

Then assert you got them all

forbid states the property those rules cannot: this tape carries no credential. Each entry is a regex, matched against the fully-redacted line the recorder is about to write. A hit raises ForbiddenValue and writes nothing — not to the file, not to the crash sidecar, not to a sink.

BOUNDARY = fr.Boundary(
    redact={"password_hash": None, "salt": None},
    forbid=[r"\b[a-f0-9]{64}\b",                    # a scrypt digest survived redaction
            r"-----BEGIN [A-Z ]*PRIVATE KEY-----"], # → raise, do not write
)

Match shapes, not values: a credential you can enumerate you could already have redacted. It is the one you cannot name that this is for. The failure names the rule and never the match — a tripwire that quotes the secret it caught, into a log or a stack trace, has become the leak it was there to prevent.

Replay

The recorded answers are fed back and the real code re-runs the original execution. Nothing is mocked — the same wrapped clients and the same clock/RNG shims simply source their answers from the tape.

class Adapter(fr.ReplayAdapter):
    boundary = BOUNDARY
    trace_root = os.path.dirname(tools_core.__file__)
    def resolve(self, fn_name, feed):
        return getattr(tools_core, fn_name)

sys.exit(fr.run_cli(Adapter()))     # in the app's `python -m app.replay`

$ python -m app.replay flight/<session>.jsonl              # list recorded calls
$ python -m app.replay flight/<session>.jsonl --call 2     # replay + full state trace
$ python -m app.replay ... --call 2 --watch level,total    # variable timeline

const tape   = fr.loadTape('.flight/flight-….jsonl');
const call   = fr.pickCall(tape, { fn: 'study_status' });
const report = await fr.replayCall({ call, fn: studyStatus, boundary: BOUNDARY });

report.ok           // result and error both reproduce the recording
report.divergence   // …or the exact point where behaviour changed

var tape   = Replay.LoadTape(".flight/flight-….jsonl");
var call   = Replay.PickCall(tape, fn: "study_status");
var report = Replay.Call(call, kw => StudyStatus((string)kw["user"]!), boundary);

report.Ok           // result and error both reproduce the recording
report.Divergence   // …or the exact point where behaviour changed

report, _ := fr.Replay(".flight/flight-….jsonl", 0,
	func(fn string, kwargs map[string]any) (func(context.Context) (any, error), error) {
		return studyStatus, nil   // the same function; its effects come off the tape, not the network
	})

report.OK()          // result and error both reproduce the recording
report.Divergence    // …or the exact point where behaviour changed

Recording tape = Recording.load(".flight/flight-….jsonl");
Replay.Report report = Replay.replayCall(tape.call("study_status"),
    (fn, kwargs) -> () -> studyStatus((String) kwargs.get("user")),   // the same function;
    boundary, false);                        // its effects come off the tape, not the network

report.ok()          // result and error both reproduce the recording
report.divergence    // …or the exact point where behaviour changed

$tape   = Recording::load('.flight/flight-….jsonl');
$report = Replay::replayCall(
    $tape->call('study_status'),
    fn (string $fn, array $kwargs) => fn () => studyStatus($kwargs['user']),   // the same function;
    $boundary                                 // its effects come off the tape, not the network
);

$report->ok();          // result and error both reproduce the recording
$report->divergence;    // …or the exact point where behaviour changed

Divergence is the finding

Replay does two jobs, and the second matters as much as the first: it answers, and it refuses to answer the wrong question. A replay that silently answered anyway would look like it worked, which is worse than useless.

• the code asks a different question;
• the code asks in a different order;
• the code stops asking — nothing gives a wrong answer, it just quietly does less work than it used to, and the unconsumed answers are the only evidence.

Trace the replay: every local, on every line

A recording tells you what the world answered. A trace tells you what the code then believed. That is what turns "what was level when it went wrong?" into a lookup rather than an inference — and it is the whole reason to replay rather than merely diff.

$ python -m app.replay flight/<session>.jsonl --call 2 --watch level,deck

  tools.py:12   level = 0
  tools.py:13   deck  = []

const report = await fr.replayCall({
  call, fn: studyStatus, boundary: BOUNDARY,
  trace: ['tools/'],          // the files to observe — not the world; every line costs a pause
});

report.trace.values('level');  // [{ value: 0, at: 'tools.js:12', fn: 'studyStatus' }]
report.trace.render('deck');   // a readable timeline

var (result, trace) = Tracer.Run(new[] { "src/Tools.cs" },   // the files to observe
                                 "App.Tools", "StudyStatus", user);

trace.Values("level");   // [{ Value = 0, At = "Tools.cs:12", Fn = "StudyStatus" }]
trace.Render("deck");    // a readable timeline

run, _ := fr.RunTraced(fr.TraceSpec{
	Include: []string{"tools.go"},                     // the files to observe
	Command: []string{"test", "-run", "^TestReplay$", "-count=1", "."},
})

run.Trace.Values("level")   // [{Value: 0, At: "tools.go:12", Fn: "studyStatus"}]
run.Trace.Render("deck")    // a readable timeline

Tracer.Run run = Tracer.run(List.of("src/main/java/app/Tools.java"),   // the files to observe
                            "app.Tools", "studyStatus", user);

run.trace().values("level");   // [level=0 at Tools.java:12 in Tools.studyStatus]
run.trace().render("deck");    // a readable timeline

$run = Tracer::run(['src/Tools.php'],            // the files to observe
                   App\Tools::class, 'studyStatus', $user);

$run->trace->values('level');   // [level=0 at Tools.php:12 in Tools.studyStatus]
$run->trace->render('deck');    // a readable timeline

Pinned recordings

Record once, replay against every build. A pinned recording is a regression oracle, not a correctness one — it asserts the code still behaves as it behaved when you pinned it, never that the recorded behaviour was right. Deciding that needs a spec, which is what invariants are.

# pyproject.toml — the pytest plugin ships with the library
[tool.pytest.ini_options]
flight_recordings = "tests/recordings"     # a directory of pinned .jsonl sessions
flight_adapter = "app.replay:Adapter"      # your ReplayAdapter
flight_trace = "build/traces"              # optional: state traces per replay

$ pytest
tests/recordings/login-bug.jsonl::call0::authenticate PASSED
tests/recordings/login-bug.jsonl::call1::fetch_profile FAILED

import { test } from 'node:test';
import assert from 'node:assert/strict';

test('the pinned session still behaves', async () => {
  const call   = fr.pickCall(fr.loadTape('tests/recordings/status.jsonl'), { fn: 'study_status' });
  const report = await fr.replayCall({ call, fn: studyStatus, boundary: BOUNDARY });

  assert.equal(report.divergence, null, report.divergence?.message);
  assert.ok(report.ok);
});

[Fact]
public void ThePinnedSessionStillBehaves()
{
    var tape   = Replay.LoadTape("tests/recordings/status.jsonl");
    var call   = Replay.PickCall(tape, fn: "study_status");
    var report = Replay.Call(call, kw => StudyStatus((string)kw["user"]!), boundary);

    Assert.True(report.Ok, Replay.FormatReport(0, report));
}

func TestThePinnedSessionStillBehaves(t *testing.T) {
	report, err := fr.Replay("tests/recordings/status.jsonl", 0,
		func(fn string, kwargs map[string]any) (func(context.Context) (any, error), error) {
			return studyStatus, nil
		})
	if err != nil {
		t.Fatal(err)
	}
	if !report.OK() {
		t.Fatalf("the pinned session diverged: %s", report.Divergence)
	}
}

@Test
void thePinnedSessionStillBehaves() throws Exception {
    Recording tape = Recording.load("src/test/resources/recordings/status.jsonl");
    Replay.Report report = Replay.replayCall(tape.call("study_status"),
        (fn, kwargs) -> () -> studyStatus((String) kwargs.get("user")), boundary, false);

    assertTrue(report.ok(), () -> Replay.format(0, report));
}

public function testThePinnedSessionStillBehaves(): void
{
    $tape   = Recording::load(__DIR__ . '/recordings/status.jsonl');
    $report = Replay::replayCall(
        $tape->call('study_status'),
        fn (string $fn, array $kwargs) => fn () => studyStatus($kwargs['user']),
        $this->boundary
    );

    self::assertTrue($report->ok(), (string) $report);
}

Edit the tape to visit a world that never happened

A recording is data, so hostile states are one edit away: empty the result, hand back an absurd number, run the clock backwards. Then replay the real code against the edited tape. This finds bugs no real traffic has triggered — without a test database that can produce impossible states on demand.

call = fr.load_call(session, 0)
call.db(0).res = []                 # the store can never actually answer this
report = fr.check_invariants(session, 0, Adapter(), INVARIANTS, mutate=call)

const call = structuredClone(fr.pickCall(tape, { fn: 'greet' }));
call.events[0].res = null;   // the store can never actually answer this
call.probe = true;           // a mutated upstream answer changes every downstream question,
                             // so arguments are no longer compared — name and order still gate

const report = await fr.replayCall({ call, fn: greet, boundary: BOUNDARY, probe: true });

var rec  = Recording.Load(tape);
var call = rec.Call(0);
call.Read("stream").Result = new List<object?>();   // empty corpus — the store can't answer this
call.Clock.Reverse();                               // time runs backwards

// Editing a call marks it a probe: arguments are no longer compared, name and order still gate.
var report = call.Check(kw => Greet((string)kw["user"]!), invariants);
rec.Save("tests/recordings/empty-corpus.jsonl");    // pin it: a probe suite member

rec, _ := fr.Load(tape)
call := rec.Call(0)
call.Event("db", 0)["res"] = []any{}   // empty corpus — the store can never answer this
call.MarkProbe()                       // a mutated answer changes every downstream question:
                                       // arguments are no longer compared; name and order still gate

report, _ := fr.CheckInvariantsCall(call, resolve, true, invariants)
rec.Save("tests/recordings/empty-corpus.jsonl")   // pin it: a probe suite member

Recording tape = Recording.load(path);
Mutate.Handle call = Mutate.on(tape.call(0));
call.read("stream").setEmpty();     // empty corpus — the store can never answer this
call.clock().reverse();             // time runs backwards

// Editing a call marks it a probe: arguments are no longer compared, name and order still gate.
Invariants.Report report = call.check(resolver, invariants, boundary);
call.save("src/test/resources/recordings/empty-corpus.jsonl", boundary);   // pin it: a probe suite member

$tape = Recording::load($path);
$call = Mutate::on($tape->call(0));
$call->read('stream')->setEmpty();     // empty corpus — the store can never answer this
$call->clock()->reverse();             // time runs backwards

// Editing a call marks it a probe: arguments are no longer compared, name and order still gate.
$report = $call->check($resolver, $invariants, $boundary);
$tape->forbiddingFrom($boundary)->save(__DIR__ . '/recordings/empty-corpus.jsonl');   // pin it

Invariants Python · Node · .NET · Go · Java · PHP

An invariant is a claim about every execution, written once and checked against any recording — so it can condemn the very first observation of a bug, which no recording can. A bug replays bit-for-bit forever; only a spec can call it wrong.

@fr.invariant("never claims end-of-corpus while words remain")
def _(t: fr.Trajectory):
    assert not (t.result["done"] and t.result["corpus"] - t.result["deck"] > 0)

@fr.invariant("level never excludes the whole corpus")
def _(t: fr.Trajectory):
    for obs in t.trace.values("level"):
        assert obs.value > 0, f"level={obs.value} at {obs.at}"

const noEmptyClaim = fr.invariant('never claims end-of-corpus while words remain', (t) => {
  assert.ok(!(t.result.done && t.result.corpus - t.result.deck > 0));
});

const levelNeverEmpties = fr.invariant('level never excludes the whole corpus', (t) => {
  for (const obs of t.trace.values('level')) assert.ok(obs.value > 0, `level=${obs.value} at ${obs.at}`);
});

const report = await fr.checkInvariants({
  tape, fnName: 'deal', fn: deal, boundary,
  trace: ['deck.js'],                  // the trace-driven form needs the files it may watch
  invariants: [noEmptyClaim, levelNeverEmpties],
});

var noEmptyClaim = Invariants.Invariant("never claims end-of-corpus while words remain", v =>
{
    var r = (IReadOnlyDictionary<string, object?>)v.Result!;
    if ((bool)r["done"]! && (long)r["corpus"]! - (long)r["deck"]! > 0)
        throw new Exception("claimed done with words remaining");
});

var levelNeverEmpties = Invariants.Invariant("level never excludes the whole corpus", v =>
{
    foreach (var obs in v.Trace.Values("level"))
        if (Convert.ToInt64(obs.Value) <= 0) throw new Exception($"level={obs.Value} at {obs.At}");
});

var report = Invariants.CheckInvariants(tape, 0, body,
    new[] { noEmptyClaim, levelNeverEmpties }, boundary);

noEmptyClaim := fr.NewInvariant("never claims end-of-corpus while words remain",
	func(t *fr.Trajectory) error {
		r := t.Result.(map[string]any)
		if r["done"] == true && r["corpus"].(float64)-r["deck"].(float64) > 0 {
			return errors.New("claimed done with words remaining")
		}
		return nil
	})

levelNeverEmpties := fr.NewInvariant("level never excludes the whole corpus",
	func(t *fr.Trajectory) error {
		for _, obs := range t.Trace.Values("level") {
			if n, _ := obs.Value.(float64); n <= 0 {
				return fmt.Errorf("level=%v at %s", obs.Value, obs.At)
			}
		}
		return nil
	})

report, _ := fr.CheckInvariants(tape, 0, resolve,
	[]fr.Invariant{noEmptyClaim, levelNeverEmpties})

var noEmptyClaim = Invariants.of("never claims end-of-corpus while words remain", t -> {
    Map<String, Object> r = Json.asMap(t.result);
    assertFalse((Boolean) r.get("done") && (Long) r.get("corpus") - (Long) r.get("deck") > 0);
});

var levelNeverEmpties = Invariants.of("level never excludes the whole corpus", t -> {
    for (Trace.Obs obs : t.trace.values("level")) {
        assertTrue((Long) obs.value() > 0, "level=" + obs.value() + " at " + obs.at());
    }
});

Invariants.Report report = Invariants.check(tape, 0, resolver,
    List.of(noEmptyClaim, levelNeverEmpties), boundary, false);

$noEmptyClaim = Invariant::of('never claims end-of-corpus while words remain', function ($t) {
    $r = $t->resultArray();
    if ($r['done'] && $r['corpus'] - $r['deck'] > 0) {
        throw new RuntimeException('claimed done with ' . ($r['corpus'] - $r['deck']) . ' left');
    }
});

$levelNeverEmpties = Invariant::of('level never excludes the whole corpus', function ($t) {
    foreach ($t->trace->values('level') as $obs) {
        if ($obs->value <= 0) {
            throw new RuntimeException("level={$obs->value} at {$obs->at}");
        }
    }
});

$report = Invariants::check($tape, 0, $resolver, [$noEmptyClaim, $levelNeverEmpties], $boundary);

The second claim is the reason this exists: the production bug that shaped the library was an internal variable (level=0) silently emptying a whole corpus, with a perfectly self-consistent output.

.NET invariants assert over the replayed result, the boundary events, the claims the code made, and — through v.Trace — every local on every executed line.

Semantic spans Python · Node · .NET · Go · Java · PHP

A recording answers “same?”. An invariant answers “right?”. Neither answers “what was this?” — and that is the question anyone actually opens a tape with. So an app may say, in its own words, what a stretch of execution meant, and have the claim recorded in-stream, wrapped around the raw events it produced. The library gains no semantics from this: the name is free text, nothing validates it, nothing interprets it. A semantic event is the app's testimony, written next to the evidence. Recording both, in order, and judging neither is what makes the testimony checkable at all — a span claiming to have charged a card, with no call to the thing that charges cards beneath it, is a claim a reader can now refute.

with fr.span("assign_turn", chore=chore_id):     # every boundary event inside is inside the span
    holder = db.collection("members").document(who).get()
    fr.note("skipped", reason="absent")          # a moment worth marking, no span

await fr.span('assign_turn', { chore }, async () => {   // every boundary event inside the span
  const holder = await store.read(`member:${who}`);
  fr.note('skipped', { reason: 'absent' });             // a moment worth marking, no span
});

const n = fr.span('compute', () => heavy());            // data is optional; sync or async

Recorder.Span("assign_turn", new { chore }, () =>    // every boundary event inside is inside the span
{
    var holder = store.Get($"member:{who}");
    Recorder.Note("skipped", new { reason = "absent" });   // a moment worth marking, no span
});

fr.Span(ctx, "assign_turn", map[string]any{"chore": chore}, func(ctx context.Context) error {
	holder, err := fr.QueryOne(ctx, "get", "member:"+who,      // every boundary event inside the span
		func() (fr.Snapshot, error) { return db.Get(who) })
	fr.Note(ctx, "skipped", map[string]any{"reason": "absent"})   // a moment worth marking, no span
	_ = holder
	return err
})

Recorder.span("assign_turn", Map.of("chore", chore), () -> {   // every boundary event inside the span
    Snapshot holder = Recorder.queryOne("get", "member:" + who, () -> db.get(who));
    Recorder.note("skipped", Map.of("reason", "absent"));      // a moment worth marking, no span
});

Recorder::span('assign_turn', ['chore' => $chore], function () use ($db, $who) {   // events inside the span
    $holder = Recorder::queryOne('get', "member:$who", fn () => $db->get($who));
    Recorder::note('skipped', ['reason' => 'absent']);          // a moment worth marking, no span
});

Which makes a tape something you can read rather than search. Load the semantic skeleton first and descend into the raw JSONL only inside the span that looks wrong:

>>> print(fr.Recording.load(tape).call(0).render_spans())
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 db)
    - corpus_read  rows=3
    register  ERROR  (2 fx)
    - registration_failed  why="kaput"

> console.log(fr.Recording.load(tape).call(0).renderSpans());
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 fx)
    - corpus_read  found=true
    register  ERROR  (2 fx)
    - registration_failed  why="no such key: alice"

> Console.WriteLine(Recording.Load(tape).Call(0).RenderSpans());
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 db)
    - corpus_read  rows=3
    register  ERROR  (2 fx)
    - registration_failed  why="kaput"

> rec, _ := fr.Load(tape); fmt.Println(rec.Call(0).RenderSpans())
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 fx)
    - corpus_read  found=true
    register  ERROR  (2 fx)
    - registration_failed  why="no such key: alice"

> System.out.println(Recording.load(tape).call(0).renderSpans());
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 fx)
    - corpus_read  found=true
    register  ERROR  (2 fx)
    - registration_failed  why="no such key: alice"

> echo Recording::load($tape)->call(0)->renderSpans();
enrol  ok  (1 now)
  enrol  ok
    load_corpus  ok  (1 fx)
    - corpus_read  found=true
    register  ERROR  (2 fx)
    - registration_failed  why="no such key: alice"

Spans are call-scoped and well-nested by construction (the only way to open one is the context manager), so enclosure is derived from order and there are no parent pointers to get wrong. A body that raises still closes its span, marked error, and the exception goes on its way untouched — a span that vanished when the code inside it failed would hide exactly the execution somebody came to the tape to read. And it is all free when the recorder is off: instrumentation lives in production code paths, so it must cost nothing and have no failure modes there.

Replay never feeds a recorded claim back — testimony was never an answer. The replayed code re-runs its own span() calls and testifies afresh, and the two accounts are compared. A difference is a third signal, and it is kept separate on purpose: a boundary divergence says the recording is stale, an invariant violation says the code is wrong, and a semantic divergence says the code's account of what it was doing has changed. That may be a refactor, so it does not fail a replay by default — a pinned suite must not go red because somebody added a span. replay_call(..., sem_strict=True) opts in, once the vocabulary has settled and changed testimony is a finding.

Where the languages differ

Not preferences — consequences. Every difference is forced by the language it lives in.

Nothing in that table is a preference. Node has no sys.settrace, so tracing there drives the V8 Inspector from a worker thread — the same mechanism a debugger uses. It costs a pause per traced line, which is why tracing belongs to replay and never to a request path.

The tape

One JSONL line per call. The format is frozen (spec/tape-v1.md), and its conformance checker is written independently in each runtime — none importing any recorder, all run against the same fixtures, each validating the others'. A disagreement means the tape has forked, which is the one failure the arrangement exists to prevent.

{"ev":"session","version":1,"node":"24.0.0","constants":{…}}
{"ev":"call","seq":1,"fn":"study_status","kwargs":{…},
 "events":[{"k":"fx","fn":"kv.get","args":["u1"],"res":{…}},
           {"k":"rand","m":"bytes","n":4,"hex":"9f2c1a70"},
           {"k":"now","v":"2026-07-11T14:01:40.244Z"},
           {"k":"fx","fn":"kv.set","args":[…],"res":"OK"}],
 "result":{…},"error":null,"ts":"…","ms":338.45}

events order is load-bearing: replay pops them in sequence, and a different question at position n is precisely where behaviour changed.

What it can and cannot see

• It sees what you declared. An input that crosses an undeclared door is invisible, and its absence shows up as a divergence — which is the honest failure mode.
• It records the answers, not the world. A recording is a fact about one execution, not a model of the store.
• A redacted field replays as its mask. Code that carries it through is fine; code that computes with it will legitimately diverge.
• Concurrency inside one call is captured in ask order, which is what makes a fan-out replayable.

Lineage

None of the underlying ideas are new. This is a small recombination of old, well-studied ones, and the honest way to describe it is by naming what it descends from.

The direct ancestor: R2 (OSDI'08)

Microsoft Research's R2: An Application-Level Kernel for Record and Replay (Guo et al., OSDI 2008) is where the central idea comes from. R2 "allows developers to choose at what interface the interactions between the application and its environment are recorded and replayed," with developers annotating the chosen functions — explicitly rejecting the fixed low-level (syscall) boundary of predecessors like liblog and Jockey. The Boundary here is R2's choose-your-own-interface idea. If you cite one thing, cite R2.

The closest functional neighbour: Keploy

Keploy already does multi-effect record/replay — incoming HTTP plus outgoing dependency calls — captured at the eBPF syscall/socket layer, code-less and language-agnostic, replaying with dependencies virtualized and verifying by response diff. Do not read "declares a multi-effect boundary" as "first to record multiple effect kinds." Keploy got there, at a lower altitude. What remains different: Keploy mocks the boundary by kind (wire protocols, Linux-only via eBPF) and its regression signal is a response diff; this declares the boundary by name at the application level, replays the real code, and reports where a divergence happened rather than only that one did.

Neighbours at other altitudes

Also upstream: functional core, imperative shell (Gary Bernhardt) is the code shape that makes any of this cheap to adopt.

So what is actually new?

A developer-declared multi-effect boundary (I/O + clock + randomness + identity); record-then-replay of application-level calls against the real code; full variable-state tracing on replay; and bit-for-bit verification with a code-path / result / writes divergence taxonomy. As of July 2026 no maintained library appears to combine all four. That is the gap — a narrow one, in a crowded neighbourhood.

If you know of prior art that lands in that intersection, please open an issue: the claim above is a survey result, not a proof.