Adding testing experiments

testing/2026_10_03_ocr_variant_experiment.md

@@ -0,0 +1,82 @@
+# 3-variant OCR cross-referencing experiment
+
+Tested whether running Tesseract over three image variants of the same
+receipt photo (original color photo, plain grayscale, and the existing
+denoised/contrast-enhanced preprocessing) and asking the extraction LLM
+to cross-reference all three per field would recover more correct data
+than the single-OCR-pass baseline, particularly for blurry/low-quality
+photos.
+
+## Result
+
+Mixed. One more receipt matched overall, two fields improved
+meaningfully, but the original target field (`litres`) did not improve,
+and `total_price` regressed.
+
+| Metric | Baseline (1 OCR pass) | New (3-variant cross-ref) |
+|---|---|---|
+| Matched to ground truth | 15/17 | 16/17 |
+| price_per_litre | 11/15 (73%) | 13/16 (81%) ↑ |
+| litres | 9/15 (60%) | 9/16 (56%) — unchanged, still worst |
+| discount | 11/15 (73%) | 15/16 (94%) ↑↑ |
+| total_price | 12/15 (80%) | 11/16 (69%) ↓ |
+
+## Root causes identified
+
+1. **The litres/price-per-litre line is the hardest OCR target**, regardless
+   of variant count. Noise characters (`x2p 02`, `*/p`) precede the real
+   numbers and the litre symbol `l` reads as `1`/`]`/`|` in all three
+   variants alike, so this field stays wrong at the same rate.
+2. **Three OCR readings don't guarantee the model picks the right one.**
+   On the blurry Jet receipt, the correct total `57,13` survives intact in
+   the original-photo variant, but the model still output `37.13`. Extra
+   (noisy) input sometimes confuses the small 3B model rather than helping
+   it reconcile.
+3. **`total_price` regressed** — cross-referencing conflicting reads
+   apparently made the model second-guess fields it used to get right from
+   a single clean reading.
+4. **Severe performance/stability cost** on this 2-core, GPU-less VM:
+   extraction latency became wildly inconsistent (1–19 minutes per receipt
+   vs. seconds before), and one run hit a hard timeout before the request
+   timeout was raised from 300s → 900s. The full 17-receipt batch that used
+   to finish in ~11 minutes now takes well over an hour.
+
+## Working examples (fully correct — all 4 fields + date matched)
+
+- `IMG_20260726_095120207`, `095321692`, `095428941`, `095442114`,
+  `095510481`, `095519964` — all Shell receipts with clean, high-contrast
+  OCR text across all three variants.
+
+## Not-working examples
+
+- **`IMG_20260525_182252232`** (the blurry Jet receipt that prompted this
+  experiment) — still no date match at all (extracted `13.05.2026` vs. true
+  `15.05.2026`). All three OCR variants scramble the header; even though
+  the correct total `57,13` is legible in the original-photo variant, the
+  model output `37.13` instead.
+- **`IMG_20260525_182144541`** — worst case, everything wrong except
+  date/time/transaction number: extracted `litres=1.205` (should be ~23),
+  `price_per_litre=1.389` (plausible alone, but paired with nonsense
+  litres), `discount=-0.08` (sign flipped), `total_price=5.33`. All three
+  OCR variants are near-unreadable for this receipt.
+- **`IMG_20260525_182116766`** — `litres` read correctly (`20.24`, matches
+  the OCR text `"20,24 1 1979 EUR/]"` cleanly) but `price_per_litre` came
+  out as `6.01` instead of `~1.979` — a pure extraction/reconciliation
+  error, not an OCR gap, since the right number is sitting right there in
+  the text.
+- **`IMG_20260726_095312079`, `095335197`, `095416655`, `095452747`,
+  `095501803`** — all wrong only on `litres`, the recurring weak field.
+- **`IMG_20260726_095215757`** — wrong on `litres` and `total_price`
+  together; the OCR text for that line is heavily corrupted
+  (`j 2 N N 4) ha 1: Hi A 1.869 EUR/`) in all three variants, so this one
+  genuinely lacks clean signal anywhere.
+
+## Conclusion / recommendation
+
+The extra OCR variants help in a few places (`discount`, `price_per_litre`)
+but don't fix the core problem (`litres` line noise), introduce a new
+failure mode (the model mis-reconciling conflicting-but-sometimes-correct
+reads), and cost a lot of latency/stability on this hardware. Leaning
+toward reverting to single-OCR unless a confidence-triggered fallback
+(only pull extra variants when the primary reading looks incomplete) is
+tried instead.