83 lines
4.1 KiB
Markdown
83 lines
4.1 KiB
Markdown
# 3-variant OCR cross-referencing experiment
|
||
|
||
Tested whether running Tesseract over three image variants of the same
|
||
receipt photo (original color photo, plain grayscale, and the existing
|
||
denoised/contrast-enhanced preprocessing) and asking the extraction LLM
|
||
to cross-reference all three per field would recover more correct data
|
||
than the single-OCR-pass baseline, particularly for blurry/low-quality
|
||
photos.
|
||
|
||
## Result
|
||
|
||
Mixed. One more receipt matched overall, two fields improved
|
||
meaningfully, but the original target field (`litres`) did not improve,
|
||
and `total_price` regressed.
|
||
|
||
| Metric | Baseline (1 OCR pass) | New (3-variant cross-ref) |
|
||
|---|---|---|
|
||
| Matched to ground truth | 15/17 | 16/17 |
|
||
| price_per_litre | 11/15 (73%) | 13/16 (81%) ↑ |
|
||
| litres | 9/15 (60%) | 9/16 (56%) — unchanged, still worst |
|
||
| discount | 11/15 (73%) | 15/16 (94%) ↑↑ |
|
||
| total_price | 12/15 (80%) | 11/16 (69%) ↓ |
|
||
|
||
## Root causes identified
|
||
|
||
1. **The litres/price-per-litre line is the hardest OCR target**, regardless
|
||
of variant count. Noise characters (`x2p 02`, `*/p`) precede the real
|
||
numbers and the litre symbol `l` reads as `1`/`]`/`|` in all three
|
||
variants alike, so this field stays wrong at the same rate.
|
||
2. **Three OCR readings don't guarantee the model picks the right one.**
|
||
On the blurry Jet receipt, the correct total `57,13` survives intact in
|
||
the original-photo variant, but the model still output `37.13`. Extra
|
||
(noisy) input sometimes confuses the small 3B model rather than helping
|
||
it reconcile.
|
||
3. **`total_price` regressed** — cross-referencing conflicting reads
|
||
apparently made the model second-guess fields it used to get right from
|
||
a single clean reading.
|
||
4. **Severe performance/stability cost** on this 2-core, GPU-less VM:
|
||
extraction latency became wildly inconsistent (1–19 minutes per receipt
|
||
vs. seconds before), and one run hit a hard timeout before the request
|
||
timeout was raised from 300s → 900s. The full 17-receipt batch that used
|
||
to finish in ~11 minutes now takes well over an hour.
|
||
|
||
## Working examples (fully correct — all 4 fields + date matched)
|
||
|
||
- `IMG_20260726_095120207`, `095321692`, `095428941`, `095442114`,
|
||
`095510481`, `095519964` — all Shell receipts with clean, high-contrast
|
||
OCR text across all three variants.
|
||
|
||
## Not-working examples
|
||
|
||
- **`IMG_20260525_182252232`** (the blurry Jet receipt that prompted this
|
||
experiment) — still no date match at all (extracted `13.05.2026` vs. true
|
||
`15.05.2026`). All three OCR variants scramble the header; even though
|
||
the correct total `57,13` is legible in the original-photo variant, the
|
||
model output `37.13` instead.
|
||
- **`IMG_20260525_182144541`** — worst case, everything wrong except
|
||
date/time/transaction number: extracted `litres=1.205` (should be ~23),
|
||
`price_per_litre=1.389` (plausible alone, but paired with nonsense
|
||
litres), `discount=-0.08` (sign flipped), `total_price=5.33`. All three
|
||
OCR variants are near-unreadable for this receipt.
|
||
- **`IMG_20260525_182116766`** — `litres` read correctly (`20.24`, matches
|
||
the OCR text `"20,24 1 1979 EUR/]"` cleanly) but `price_per_litre` came
|
||
out as `6.01` instead of `~1.979` — a pure extraction/reconciliation
|
||
error, not an OCR gap, since the right number is sitting right there in
|
||
the text.
|
||
- **`IMG_20260726_095312079`, `095335197`, `095416655`, `095452747`,
|
||
`095501803`** — all wrong only on `litres`, the recurring weak field.
|
||
- **`IMG_20260726_095215757`** — wrong on `litres` and `total_price`
|
||
together; the OCR text for that line is heavily corrupted
|
||
(`j 2 N N 4) ha 1: Hi A 1.869 EUR/`) in all three variants, so this one
|
||
genuinely lacks clean signal anywhere.
|
||
|
||
## Conclusion / recommendation
|
||
|
||
The extra OCR variants help in a few places (`discount`, `price_per_litre`)
|
||
but don't fix the core problem (`litres` line noise), introduce a new
|
||
failure mode (the model mis-reconciling conflicting-but-sometimes-correct
|
||
reads), and cost a lot of latency/stability on this hardware. Leaning
|
||
toward reverting to single-OCR unless a confidence-triggered fallback
|
||
(only pull extra variants when the primary reading looks incomplete) is
|
||
tried instead.
|