← RESEARCH
Research22 October 2026

Three tests before an engine earns a decision

A backtest, a replay and a forward ledger answer three different questions. Most systems that act on a company answer none of them first. Here is what each test can prove, what it cannot, and the order we run them in.

When a system is going to act inside a company, there are three questions worth asking before it does. Would it have seen what happened last quarter? After we change it, does it still behave the way the team approved? And when it acts, does the outcome improve? These sound like one question. They are three, they need different evidence, and they fail in different ways.

We run three tests in that order: a backtest, a replay, and a forward ledger. Each is cheaper than the next. None of them is new. What is unusual is insisting on all three before a decision counts.

The backtest answers a narrow question

A backtest replays the company's own history as of each past date and scores what the engine would have flagged against what happened. For an outcome like churn, the engine reads what was knowable on the first of every month, flags the accounts its signals pointed at, and we score those flags against the accounts that later left.

The hard part is "knowable". Company state is mutable and rarely versioned. A record in the billing system holds today's value, not the value it held in March. Stripe exposes its event stream for 30 days only, so a history older than that has to be rebuilt from object timestamps or captured as it happens. A backtest built on today's snapshot quietly reads facts nobody knew at the time, and then looks better than it was. Every read in our backtest goes through an as-of boundary that cannot return a row ingested after the date or a field value later overwritten. That boundary matters more than the model.

The model is the second leak. A language model that has read the internet may remember how a public company's quarter went. Recent work on lookahead bias shows models recall economic values from before their training cutoff even when names and dates are masked, and that effective cutoffs drift by months across vendors. Private companies are less exposed, but the discipline is the same: freeze the detector before scoring, keep the last quarter as a lockbox nobody tunes against, and count every run, because every round of tuning on the same events spends some of their evidence.

Then there is the arithmetic of a small company. Seventy-two churn events in a year is a normal number and a small sample. Sixty-one caught out of seventy-two is 85 percent, with a 95 percent interval that runs from roughly 75 to 91. A median lead time of 34 days carries a wide band of its own. Two variants of the detector cannot be told apart on the same events unless their recall differs by close to ten points. So we report the four counts, the base rate, an interval on every rate, lead time as a distribution measured against the day the team itself first noticed, the naive rules we beat or did not, and each miss with a one-line reason. We label the whole thing retrospective detection, and we attach no revenue number to it. A backtest tests seeing. It does not test acting.

Replay catches what a change breaks

The second question is about behavior. Every change to a prompt, a memory, a routine or an autonomy rule can move the engine's conduct in a place nobody was looking. Codebases have a test suite for that. An engine that works inside a company needs one too, and it has to be built from the company's own runs.

So every engine run is recorded as a trace, and a change is replayed against the saved scenarios before it is promoted. Structured reads come from the as-of ledger exactly as they were. Only people and systems that were not recorded get a simulated stand-in, and that stand-in is conditioned on the recorded replies, the way the newer agent benchmarks constrain a simulated user to the environment's real state. Writes are logged as intended actions and never executed. The gate is scored per action class, not as one pooled number, because a pooled score hides a regression in the one class that matters. And it is scored over repeated trials, since an agent that passes once in three runs has not passed.

Two cautions apply. Published studies show that simulated users flatter agents: success on the same tasks moves by several points depending on which model plays the customer, and across dozens of simulators the simulated setting creates an easy mode relative to human users. We treat replay as a gate against regressions, not as proof the engine works with people. And a simulated tool is bounded by what was recorded. A system the engine has never touched in a trace has no ground to stand on, so version one records and replays the three or four systems a pilot actually uses.

The best scenarios come from corrections. When someone answers a proposal in Slack with "no, not like that", that moment already contains the as-of inputs, the action class and the expected behavior in one sentence. One click from the owner turns it into a test the next version has to pass, paired with a case where the engine should not have fired at all. Over a quarter the suite stops being ours and becomes the company's.

Only the forward ledger proves effect

The third question is causal, and history cannot answer it. If the engine flagged an account and the account stayed, the team may have saved it. If the engine proposes a discount and the customer renews, the customer may have renewed anyway. Logs alone cannot separate what the engine caused from what the team did regardless. The off-policy evaluation literature is clear on the two conditions that would let you estimate a new policy from logs: the new policy's actions have to appear in the log, and there can be no unmeasured confounders. In a founder-run company the save call was made on context no system recorded. The confounder is unobserved by construction.

So the forward ledger is the test we ask a customer to trust. Every proposal is stored with its predicted odds and a hash before the outcome exists, then scored when the outcome arrives. Where it is cheap, a random share of flagged accounts is held out from the action and the assignment probability is logged. That one habit makes the first condition true going forward and gives a clean estimate later. It is also the only place a revenue claim can come from, and until it exists we do not make one.

Why this order

The backtest is batch, cheap and tenant-isolated, and it is the first thing a company sees: what we would have caught, with the misses listed. The replay costs recorded traces and a gate per class, and it runs every time the engine changes. The forward ledger costs patience. Running them in that order means the expensive test is only paid for once the cheap ones have passed, and it means the number on the screen always says which test produced it.

What version one ships: the as-of ledger, the backtest runner, thirty to fifty regression scenarios drawn from real failures, and the forward ledger with randomized holdouts where the outcome allows it. What it does not ship: a learned world model of the company, simulated customers beyond a few fixed personas, simulated consequences of writes, or any dollar figure from a backtest.

What stays open: sample sizes that will not support a richly fitted model, accounts that share owners and seasons so the effective sample is smaller than the count, labels contaminated by the team's own interventions, and simulators and judges that need calibration against real people. We would rather publish those limits than a single number.

Notes

WRITING