From 68e0cd89677dd10ffe4adf3dcb1092707de1fd9d Mon Sep 17 00:00:00 2001 From: Matthias Ladkau Date: Sat, 3 Oct 2026 06:34:38 +0200 Subject: [PATCH] Adding testing experiments --- testing/2026_10_03_ocr_variant_experiment.md | 82 ++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 testing/2026_10_03_ocr_variant_experiment.md diff --git a/testing/2026_10_03_ocr_variant_experiment.md b/testing/2026_10_03_ocr_variant_experiment.md new file mode 100644 index 0000000..1099360 --- /dev/null +++ b/testing/2026_10_03_ocr_variant_experiment.md @@ -0,0 +1,82 @@ +# 3-variant OCR cross-referencing experiment + +Tested whether running Tesseract over three image variants of the same +receipt photo (original color photo, plain grayscale, and the existing +denoised/contrast-enhanced preprocessing) and asking the extraction LLM +to cross-reference all three per field would recover more correct data +than the single-OCR-pass baseline, particularly for blurry/low-quality +photos. + +## Result + +Mixed. One more receipt matched overall, two fields improved +meaningfully, but the original target field (`litres`) did not improve, +and `total_price` regressed. + +| Metric | Baseline (1 OCR pass) | New (3-variant cross-ref) | +|---|---|---| +| Matched to ground truth | 15/17 | 16/17 | +| price_per_litre | 11/15 (73%) | 13/16 (81%) ↑ | +| litres | 9/15 (60%) | 9/16 (56%) — unchanged, still worst | +| discount | 11/15 (73%) | 15/16 (94%) ↑↑ | +| total_price | 12/15 (80%) | 11/16 (69%) ↓ | + +## Root causes identified + +1. **The litres/price-per-litre line is the hardest OCR target**, regardless + of variant count. Noise characters (`x2p 02`, `*/p`) precede the real + numbers and the litre symbol `l` reads as `1`/`]`/`|` in all three + variants alike, so this field stays wrong at the same rate. +2. **Three OCR readings don't guarantee the model picks the right one.** + On the blurry Jet receipt, the correct total `57,13` survives intact in + the original-photo variant, but the model still output `37.13`. Extra + (noisy) input sometimes confuses the small 3B model rather than helping + it reconcile. +3. **`total_price` regressed** — cross-referencing conflicting reads + apparently made the model second-guess fields it used to get right from + a single clean reading. +4. **Severe performance/stability cost** on this 2-core, GPU-less VM: + extraction latency became wildly inconsistent (1–19 minutes per receipt + vs. seconds before), and one run hit a hard timeout before the request + timeout was raised from 300s → 900s. The full 17-receipt batch that used + to finish in ~11 minutes now takes well over an hour. + +## Working examples (fully correct — all 4 fields + date matched) + +- `IMG_20260726_095120207`, `095321692`, `095428941`, `095442114`, + `095510481`, `095519964` — all Shell receipts with clean, high-contrast + OCR text across all three variants. + +## Not-working examples + +- **`IMG_20260525_182252232`** (the blurry Jet receipt that prompted this + experiment) — still no date match at all (extracted `13.05.2026` vs. true + `15.05.2026`). All three OCR variants scramble the header; even though + the correct total `57,13` is legible in the original-photo variant, the + model output `37.13` instead. +- **`IMG_20260525_182144541`** — worst case, everything wrong except + date/time/transaction number: extracted `litres=1.205` (should be ~23), + `price_per_litre=1.389` (plausible alone, but paired with nonsense + litres), `discount=-0.08` (sign flipped), `total_price=5.33`. All three + OCR variants are near-unreadable for this receipt. +- **`IMG_20260525_182116766`** — `litres` read correctly (`20.24`, matches + the OCR text `"20,24 1 1979 EUR/]"` cleanly) but `price_per_litre` came + out as `6.01` instead of `~1.979` — a pure extraction/reconciliation + error, not an OCR gap, since the right number is sitting right there in + the text. +- **`IMG_20260726_095312079`, `095335197`, `095416655`, `095452747`, + `095501803`** — all wrong only on `litres`, the recurring weak field. +- **`IMG_20260726_095215757`** — wrong on `litres` and `total_price` + together; the OCR text for that line is heavily corrupted + (`j 2 N N 4) ha 1: Hi A 1.869 EUR/`) in all three variants, so this one + genuinely lacks clean signal anywhere. + +## Conclusion / recommendation + +The extra OCR variants help in a few places (`discount`, `price_per_litre`) +but don't fix the core problem (`litres` line noise), introduce a new +failure mode (the model mis-reconciling conflicting-but-sometimes-correct +reads), and cost a lot of latency/stability on this hardware. Leaning +toward reverting to single-OCR unless a confidence-triggered fallback +(only pull extra variants when the primary reading looks incomplete) is +tried instead.