The core feature is an eval.

Every row is entered twice — an agent claim and the evidence that should reconcile it — with a human label for the finding classes that should be caught. Rows carry a provenance, and provenance gates how a row may be used. This is the line that keeps the project honest:

• synthetic-seed — 31 hand-authored fixtures. Engine development only. Never counted toward a reported rate.
• public-incident — 17 documented, source-cited real-world failures. Real, but a separate supporting class — not your own measured rate.
• self-run / gmail — real agent runs you commissioned. The onlyprovenance eligible to produce the headline misbehavior rate.

Two numbers, kept strictly apart: detector accuracy (does the engine reproduce the labels?) and the agent-misbehavior rate (of real runs, how many went wrong?). A synthetic row that leaks into the second number would turn this into the very thing it detects. The metric code filters on provenance before it reports.

The deterministic engine scored against the synthetic fixtures. Precision is high where the engine commits, because the rules are exact; recall is the honest story.

Read it straight. OVERSPEND is precise and high-recall (1.00 / 0.92) — a budget delta is a number. UNAUTH_RECURRENCE (1.00 / 1.00) is a flag check. But SCOPE_CREEP recall is only 0.80: the engine catches unauthorized spend, a keyword list of known add-ons, and violated send-prohibitions — item additions outside those subsets need fuzzy comparison and are left to the judge. And CLAIM_MISMATCH recall is 0.00 by design— wording mismatches (“cheapest”, wrong date) are not something a deterministic rule should pretend to judge. Those are the LLM judge’s job, and every judge finding is badged LLM-assisted.

Run the same engine over 17 real, documented agent failures (Air Canada, Replit, the $1 Tahoe, Mata v. Avianca…) and it largely abstains. That isn’t a bug — almost every real-world case is a CLAIM_MISMATCH: the agent said something false, gave illegal advice, fabricated a citation. Numbers reconcile; the claimdoesn’t. This is precisely the residual the LLM judge is built for, and the clearest argument that deterministic-first is right: be exact where you can be, and don’t guess where you can’t.

The two SCOPE_CREEPfalse positives here are a known eval-validity nuance: several incidents are advice/decision failures framed with “no purchase authorized,” which trips the spend rule. They’re documented in the limitations below, not papered over.

Nothing here is a number in prose. Clone the repo and re-run it:

# the engine + judge unit tests (TDD)
npm test

# this exact per-class table, in the terminal
npm run eval

# the citable harness: Inspect AI (the engine is the classifier; this only scores it)
npm run eval:build
inspect eval eval/discrepancy_eval.py --model mockllm/model -T split=all -T seed=1234

The engine is deterministic — it has no learned parameters — so the seed-shuffled split (-T split=test) is there for reproducible partitioning, not overfitting control; the headline runs on the full labeled set. Same file + seed ⇒ byte-identical split, recorded in the .eval log. Inspect carries each prediction’s classes, so the scorer builds a real confusion matrix and reports per-class precision/recall with standard error (the small fixture N is honest, not hidden).

The engine isn’t only checked against examples; it’s held to a formal contract. Its 10 firing invariants and 6 metamorphic relations (see SPEC.md) are written as executable predicates independent of the engine, and a property-based fuzzer mutates inputs against the rule boundaries trying to break them.

Computed live on every build (and inside npm test; run it with npm run fuzz). Beyond firing the right flags, the metamorphic relations prove the engine is coherent — more money charged can never remove an overspend; raising the budget can never create one; granting authorization can only remove findings. Four more moves past plain detection:

• Diagnosis, not just detection.Each finding carries ranked, deterministic root-cause hypotheses (“+$78 → likely an undisclosed seat fee + add-on”) — closer to a black-box recorder than a tripwire.
• Tamper-evident, auditable receipts. Each receipt is content-addressed (SHA-256 over its claim, evidence, and verdict) and batched into a Merkle audit trail: one root commits to a whole session, and an inclusion proof shows a receipt belongs to it without revealing the others — selective transparency, no SNARK required.
• The judge is a measured instrument.A calibration harness scores it against human labels (TPR/FPR, precision/recall, Cohen’s κ) and reports a rate as a Wilson confidence interval(“37–44%”), not a point. An evaluator-of-the-evaluator probe flags positional bias. Built and unit-tested; it runs the moment a key and labels exist.
• Externally grounded. Run over the 164 real airline + retail tasks of τ²-bench (Sierra, MIT), the engine produces zero false positives on the in-scope reference trajectories. Honestly scoped — a specificity check, not a τ²-bench score (the money dimensions need the benchmark’s environment); see bench/tau2/.

“Agents overspent X%” is the number that travels. It can only come from real runs you commissioned (self-run / gmail provenance), never from synthetic or third-party data. The firewall forbids filling it with anything else. Right now:

An LLM judge is itself a measurement instrument, so it has to be measured. The plan — gated behind your own ANTHROPIC_API_KEY — is to score the judge’s CLAIM_MISMATCHcalls against the human labels on a held-out split (true/false-positive rate, Cohen’s κ), and only trust it past a calibration bar. Until that runs against real keys, the judge is presented as assistive and clearly badged, never as silent ground truth. No LLM judgment drives a money or alert action without a deterministic gate behind it.

• SCOPE_CREEP recall is 0.80.The engine catches unauthorized spend, a known add-on keyword list, and violated send-prohibitions; item additions outside those subsets are abstained to the judge’s item comparison.
• CLAIM_MISMATCH is unhandled deterministically (recall 0.00). By design — it is the entire LLM-judge residual, and the judge runs only with a key.
• Advice-failure framing causes SCOPE_CREEP false positives on a few incidents (a “no purchase” scope plus a downstream dollar amount). Surfaced, not hidden.
• Synthetic ≠ real. Detector accuracy on fixtures is not a claim about real-world prevalence — that is the separate, pending headline number above.
• v1 is paste-only. No Gmail OAuth or stored ledger yet; the privacy boundary (no raw email body persists; extracted fields + a redacted excerpt only) is a type invariant ready for when it does.