Factor 1000: Why a Vision Model Misreads Receipts

TL;DR

In BuchhaltGenie a vision model reads receipts, with no OCR library underneath it. No tesseract, no pdf-parse, no textract, a grep over package.json finds none of them. Before I committed to a model, a bakeoff ran, meaning a head-to-head comparison under identical conditions, across 16 real receipts and seven vision-capable candidates, identical prompt, identical JSON schema. The result flipped my model choice. The side finding mattered more: models misread amounts by powers of ten, systematically, and no model card mentions it. That is why the plausibility guard sits above the model instead of inside it.

The Receipt That Costs 60 Euro and Reads 60,000

A receipt carries the number 60.000. For a model the dot in it is ambiguous: thousands separator or decimal separator. Guess wrong and it reads 60,00 instead of 60.000, or the other way round, 31.00 as 3100,00. The first direction is a factor of 1000 too small, the second a factor of 100 too large. I measured both directions in the bakeoff, which is why the guard checks both today.

This is not a rounding inaccuracy, it is its own error class. An amount that is off by three digits looks entirely normal in the form. It has a date, a supplier, a VAT rate, and all of that is correct. Only the number is wrong, wrong in a way that nobody clicking through quickly will notice.

The header of the guard module says exactly that, and I quote it here as it stands in the code: push the model output through the plausibility guard. The guard is the heart of this module. The bakeoff across 16 real receipts showed that vision models fail systematically on thousands separators and foreign currencies.

A proposal that is off by a factor of 1000 is worse than no proposal at all. It looks plausible.

I Did Not Build an OCR Library, I Measured

The bakeoff ran on 4 August. The starting point was a catalogue of 338 models, 181 of them vision-capable. Seven went into the comparison, all with an identical prompt and an identical JSON schema, so the comparison would not hinge on prompt wording.

The material was 13 receipts in the main run plus 3 in the addendum, all real, all hand-labelled. The ground truth was deliberately built to be awkward. There was an abstain bait: a DHL drop-off slip with no amount at all, where the correct answer is null and not a pretty invented number. There was a factor bait with an Indonesian amount over 31.000. There was a zero amount, and receipts with COVID-era VAT rates that nobody expects today.

The budget was capped at 10 euro. I spent 0.51 euro across 171 calls in two runs, 156 plus 15. Of those 171 responses, 171 were valid JSON, not a single parse error. That is the pleasant surprise of the whole exercise: schema fidelity is no longer a problem with current models. Content fidelity still is.

The weighting follows what actually hurts in bookkeeping: amount 35 percent, date 20 percent, supplier 15 percent, VAT rate 15 percent, currency 10 percent, correct abstain 5 percent. Plus three penalties: a factor error costs 10 percentage points, a hallucinated amount another 10, a parse error 5.

The Result: The Model Card Does Not Say It

The ranking from the main run. Where two score values appear, the documentation carries both, and I quote them unchanged.

Model	Score	Factor errors	Latency	Cost per receipt
Gemini 3.6 Flash	100.0 / 98.6	0	2.83 s	0.0056 EUR
Claude Sonnet 5	87.5 / 87.5	0	4.19 s	0.0078 EUR
Qwen3 VL 235B	75.0 / 81.2	1	3.18 s	0.0008 EUR
Mistral Medium 3.5	72.5 / 72.5	1	1.99 s	0.0028 EUR
Mistral Small 4	69.0	1	1.85 s	0.0002 EUR
Claude Haiku 4.5	68.1	1	3.88 s	0.0022 EUR
Ministral 3 8B	65.4 / 65.4	1	1.67 s	0.0002 EUR

Two things about this I like. First, all seven correctly returned null on the amountless DHL drop-off slip, and the main run produced zero hallucinated amounts. The fear that a model invents a number out of embarrassment did not hold up in this material. Second, the price gap between first and third place is small enough that it does not decide the choice.

The real test was the addendum. Three receipts, all with a thousands separator above 1.000: a Swiss QR receipt for 2 500.00, a CORD receipt for 60.000, a CORD receipt for 174,600. Ministral 3 8B got 2 out of 3, Mistral Small 4 also 2 out of 3, Mistral Medium 3.5 only 1 out of 3. Claude Sonnet 5 and Gemini 3.6 Flash landed at 0 out of 3, even though both stayed free of factor errors in the main run. So the 0 in the table above and the 0 here mean opposite things: there no errors, here no hits. In the main run the factor bait with the Indonesian 31.000 receipt slipped past 5 of 7 models. The bakeoff document draws a conclusion from this that I consider its single most important sentence: the factor-1000 error of the Mistral family is a systematic property, not an outlier.

An honest limitation belongs here. The dot receipt in the addendum is Indonesian. It took me roughly 25 full-text searches on Wikimedia Commons and I still found no German-language euro receipt above 1.000 that was freely usable. So the error space I measured is not the one my users live in. The pattern holds, the sample is narrow.

The Guard Sits Above the Model, Not Inside It

The architecture follows from that finding. The cross-check computes net plus VAT against gross, with a tolerance of 2 cents, and tests the powers of ten 10, 100 and 1000 in both directions.

// Cross-check: net plus VAT must equal gross, tolerance 2 cents
const CROSS_CHECK_TOLERANCE_CENTS = 2;
const SCALING_FACTORS = [10, 100, 1000] as const;

function isScalingMismatch(grossCents: number, expectedCents: number): boolean {
  return SCALING_FACTORS.some(
    (factor) =>
      Math.abs(grossCents * factor - expectedCents) <= CROSS_CHECK_TOLERANCE_CENTS ||
      Math.abs(grossCents / factor - expectedCents) <= CROSS_CHECK_TOLERANCE_CENTS,
  );
}

The call sits as step c in building the proposal, between the model output and everything that comes after. Next to it are a few unspectacular caps: at most 8 pages per PDF, supplier name at most 200 code points, receipt age at most 10 years. That last value is an implementation decision in my code and not a statement about how long anyone has to keep receipts.

Above the reach section sits a comment that felt uncomfortable to write, which is why I left it in: this guard applies exclusively to the complete triple of gross amount, VAT amount and Austrian VAT rate. The cross-check is a spot check, not general amount protection. On a receipt with no VAT shown it does nothing at all.

Three Stages Instead of a Confidence Bar

Whatever the guard finds has to land somewhere. A percentage bar helps nobody, because it suggests no action. So there are three stages: autonom, ein_klick, review. Nine rules in fixed order decide between them, plus nine reason codes, so it stays traceable later which rule fired. Confidence runs from 0 to 1, the threshold for autonomous handling is 0.98, and the logic is fail-closed: anything uncertain lands in review, not in the autopilot.

// Rule 3 of the staging logic: a discarded amount always forces review
if (guard.amountDiscarded) {
  return { stage: 'review', reason: 'plausibility_guard_rejected_amount' };
}

The reasoning is short and lives in the code: a power-of-ten cross-check that discards the amount is an explicit warning, and no autopilot may override it. The materiality threshold is shared with the bank autopilot, so an amount is not small in one module and large in the other.

What I Take Away From This

1. A bakeoff on real material beats any documentation comparison. None of the seven model cards mentions thousands separators. The error class that would cost my product the most appeared in no specification, only in 16 hand-labelled receipts. 0.51 euro and an afternoon was a ridiculous price for that.

2. A deterministic guard is cheaper than a better model. The cross-check is under 20 lines and costs nothing per call. It also protects the quality winner against future model jumps, exactly as the bakeoff document states in its consequences. Switching models becomes a commit, not a rebuild.

3. Honest reach is part of the guard. A guard whose limits nobody knows produces precisely the wrong kind of confidence. The comment about its reach is therefore not self-doubt but documentation: it says where I have nothing yet.

4. Fail-closed as the default, even when it costs clicks. Every discarded amount forces review. That is occasionally annoying. The alternative would be an amount off by a factor of 1000 slipping through silently, and that is not an alternative.

Conclusion

The order was: measure first, then choose, then guard. The bakeoff flipped my model choice, but its lasting return is the guard I would not have built without it. A model that scores 100 out of 100 is still a model, and the next update can lose the very property I picked it for. The cross-check stays.

How I handle model output in general, instead of believing it, is in Verification, Not Typing. Why turning such a prototype into a product takes a lot of unglamorous work is in From Prototypes to Product, and how a model comparison can also end is in my failed attempt at a local coding model. What else I work with is here. If you have a similar extraction problem and want to know where your guard belongs: write to me.