Closing the books with agents: the exception benchmark

Most finance-automation claims are asserted, not measured. We built a benchmark to measure them — and to hold ourselves to it. Here's how owned, fine-tuned models behind an independent verifier perform against raw-API baselines on the work that actually matters: the long tail of exceptions.

Why a benchmark at all

Straight-through processing (STP) — the share of transactions completed with no human touch — is the number a controller cares about. Yet vendors quote it without a shared definition or test set. We defined an exception benchmark over real AP/AR/close scenarios so any team can compare approaches on equal footing.

The setup

We assembled a labeled corpus of invoices, purchase orders, remittances, and close exceptions spanning multi-entity, multi-currency, and table-heavy documents. We then compared three configurations:

• A raw frontier-API baseline (prompting only).
• Our fine-tuned reasoning model (Ledgr) alone.
• The reasoning model gated by an independent verifier.

Straight-through processing

The fine-tuned reasoning model lifted STP meaningfully over the raw-API baseline. But the decisive change came from the verifier: by catching trust-eroding failures before any write, it let us safely raise the autonomy threshold — pushing STP to 94% on the beachhead workflow without increasing risk.

The verifier lift

An independent verifier — a model trained specifically on the failure modes that erode trust, separate from the model that acts — caught 99% of injected errors in evaluation. This is the core insight: a single model that both acts and grades itself is untrustworthy. Separation is what makes autonomy defensible.

Cost and latency

Serving the models with TensorRT-LLM and Triton kept verification under 100ms per action and cut inference cost well below per-call API pricing at volume — turning inference from a margin tax into a margin lever.

Reproduce it

We're publishing the methodology and scenario taxonomy so finance and engineering leaders can verify, not just read. If you'd like the dataset specification, reach out — and if you'd like to see it run on your own period, we'll do it in shadow mode.