← RESEARCH
Research22 October 2026

What a confidence figure has to mean

A card that says a move is worth $1.4M at 92% confidence is making three claims at once. Here is what each one has to be backed by before we let it on the screen, and what the screen refuses to say.

One of our cards reads: win back 84 at-risk accounts, plus $1.4M of annual revenue, 92 percent confidence. It is the kind of line every decision product wants to show. It packs three objects into one: a magnitude, a probability, and the source of that probability. Each of the three fails in its own way, and a system that proposes which outcomes a company should own has to be clear about all three, because the proposal is only as good as the number next to it.

A probability needs an event

Ninety-two percent of what? "The program beats doing nothing" and "the gain exceeds a million" are different events with different numbers, and a percentage with no event attached is decoration.

A small simulation makes the point. Take 2,310 customers split in two, 36 churn events in the control half and 25 in the treated half. The probability that treatment churn is lower is 0.92. The probability that the cut is at least 25 percent is 0.61. The 95 percent interval on the relative cut runs from minus 15 to plus 58 percent. The same result supports a confident sentence and a shaky one, depending on which event you name.

So every prediction we record names its event in one sentence, with the population, the metric and the horizon, and it is written down before the action runs. The record is append-only. A probability that can be edited after the outcome is not a probability.

Where the number comes from

The second object is the source, and we show it as a tier next to every figure. Tier A is a randomized holdout inside the same action class: some flagged accounts were held back, their assignment was logged, and the difference is measured. Tier B is a comparison series for changes that cannot be randomized per customer, such as a pricing change, estimated against a control series with a long history and labelled as an estimate. Tier C is judgement from reference classes in our own ledger, with no measurement possible yet.

The tier decides which words the screen may use. Tier A can say "measured". Tier B can say "estimated". Tier C can say "expected", and has to show how many resolved cases the expectation rests on. A number that hides its tier is asking for trust it has not earned.

The model's percentage is a draft

Language models state percentages fluently and badly. Verbalized confidence is overconfident across model families, no elicitation technique wins consistently, and models reach for a handful of round values: in one study the three most common answers covered between 78 and 92 percent of all responses on a 0 to 100 scale. Public forecasting benchmarks are more encouraging, with systems now close to expert human forecasters on questions about world events that resolve in weeks. That says little about the effect of one action on one company's number, which is the question we have to answer, and for which no public benchmark exists that we could find.

So the model's figure is treated as a draft. The model reads the evidence, drafts the event and proposes a distribution. Code, not the model, shrinks that distribution toward the base rate of the action class in our ledger and applies a one-parameter calibration fitted on resolved rows. Richer calibration waits until a class has hundreds of resolved predictions, which at a 200-person company is a matter of quarters, not weeks. Until then the screen shows the count of resolved predictions before it shows any score, and when the count is too small it says so instead of printing a percentage.

Ranking makes the winner lie

Here is the part that bites a system whose job is to rank. If you estimate the value of seven candidate outcomes with noise and pick the top one, the top one is the most overestimated of the seven, more or less by construction. The regressional form of Goodhart's law has a name because it is that common. A product that proposes outcomes ranked by value is built on exactly this selection step, so the shrinkage above is not a refinement. It is what stops the first card from being the most inflated one.

Two more habits follow. Every outcome gets a counter-metric, because any single number can be pushed by doing the wrong thing: churn falls if you make cancelling hard, support response time falls if you close tickets without answering them. And we score the portfolio, not the single call. A 90 percent bet fails one time in ten, and a ledger of one entry proves nothing either way.

What the scale allows

Seventy-two churn events in 2,310 customers is a normal year for a company of this size and a thin base for statistics. A 50/50 randomized holdout across the whole base detects a churn cut near 56 percent at conventional power; a 25 percent cut would need roughly 6,900 customers per arm. Effects land months later, while seasonality, pricing and other actions move the same metric. This is why "the P&L is the ultimate eval" is a slogan rather than a method at this scale: the P&L is the final auditor, and it only sees large effects. It cannot be the training signal.

What can be is the ledger. Every proposal carries its predicted distribution, its evidence tier and its measurement plan, frozen at creation. A scheduled job resolves it at the horizon with the stated design, reading the source systems directly and sharing no prompt or model with the agent that made the proposal. Each action class accumulates resolved rows, a hit rate with an interval, and, once the rows are there, a Brier score against the base rate. An action class moves toward autonomy only when its resolved count and its skill against baseline clear thresholds fixed in advance. Across companies, with consent, that ledger becomes the benchmark for intervention effects on business metrics that does not exist today.

What the screen refuses to say

A dollar impact from a single before-and-after. A bare percentage with no event. Causal credit when two actions overlapped on the same customers. A calibration curve before enough rows have resolved. A revenue effect inside a horizon shorter than the known lag. And a company-wide "the engine generated this much" without a holdout behind it.

What it says instead, in version one: the expected value with an 80 percent range, the probability that the move beats doing nothing, the tier, the count of similar past cases the expectation rests on, and whether a holdout is proposed. It is a longer line than "92% confidence". It is also one we can defend.

Notes

WRITING