Add adaptive preprocessing escalation and single-receipt test scripts

Preprocessing now tries progressively heavier variants per receipt
(plain -> high-pass -> edge-enhance -> both), stopping as soon as a
cheap LLM-free arithmetic check on the OCR text looks plausible, so
the extra processing only applies to receipts that need it. A flat
version of the same heavier processing regressed the full batch by
helping hard receipts but hurting easy ones; the adaptive version
improves date-matching 37/52 -> 40/52 with no regression (see
testing/2026_10_04_gimp_highpass_nlfilter_experiment.md).

Also adds test_ocr.sh and test_single.sh for checking a single
receipt's OCR text or full pipeline result without a batch run.

12_adaptive_preprocess_ocr.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Combines steps 1+2 (preprocess + OCR) adaptively: tries preprocess.py's
+# variants cheapest-first (0=baseline, same as 1_preprocess_image.sh's
+# default, up to 3=most elaborate), scoring each variant's OCR text with
+# tooling-container/src/plausibility.py and stopping at the first one
+# that looks receipt-consistent - no LLM call involved, so this costs
+# nothing extra over the single-variant path for the receipts the
+# baseline already reads correctly. See tooling-container/src/adaptive.py
+# and testing/2026_10_04_gimp_highpass_nlfilter_experiment.md for why.
+#
+# Writes <out>/variant<N>/{01_gray,...,0X_final}.png and its own ocr.txt
+# for every variant tried (for inspection - which variant was kept is in
+# the printed summary) and <out>/ocr.txt (the winning variant's OCR
+# text, ready for ./3_extract_data.sh).
+#
+# Usage: ./12_adaptive_preprocess_ocr.sh -in <photo> -out <scratch-dir>
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+ROOT="$(pwd)"
+
+fail() { echo "FAIL: $*" >&2; exit 1; }
+
+IN_PATH=""
+OUT_DIR=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -in) IN_PATH="${2:-}"; shift 2 ;;
+        -out) OUT_DIR="${2:-}"; shift 2 ;;
+        *) fail "unknown argument: $1" ;;
+    esac
+done
+[ -n "$IN_PATH" ] && [ -n "$OUT_DIR" ] \
+    || fail "usage: $0 -in <photo> -out <scratch-dir>"
+[ -f "$IN_PATH" ] || fail "input file not found: $IN_PATH"
+
+command -v docker >/dev/null 2>&1 || fail "docker not found in PATH"
+
+IMAGE="${REGISTRY_IMAGE:-fuelish/tools}"
+if [ -f "$ROOT/registry.env" ]; then
+    # shellcheck disable=SC1091
+    source "$ROOT/registry.env"
+    IMAGE="${REGISTRY_IMAGE:-$IMAGE}"
+fi
+VERSION="$(<"$ROOT/tooling-container/VERSION")"
+IMAGE="$IMAGE:$VERSION"
+docker image inspect "$IMAGE" >/dev/null 2>&1 \
+    || fail "$IMAGE not found locally - run ./build-image.sh first"
+
+IN_ABS="$(realpath "$IN_PATH")"
+IN_DIR="$(dirname "$IN_ABS")"
+IN_FILE="$(basename "$IN_ABS")"
+mkdir -p "$OUT_DIR"
+OUT_ABS="$(realpath "$OUT_DIR")"
+
+GROUP_ADD_ARGS=(--group-add "$(stat -c %g "$IN_DIR")")
+OUT_GID="$(stat -c %g "$OUT_ABS")"
+[ "$OUT_GID" = "$(stat -c %g "$IN_DIR")" ] || GROUP_ADD_ARGS+=(--group-add "$OUT_GID")
+
+echo "== Adaptively preprocessing + OCRing $IN_FILE =="
+docker run --rm \
+    --user "$(id -u):$(id -g)" \
+    "${GROUP_ADD_ARGS[@]}" \
+    -v "$IN_DIR:/data/in:ro" \
+    -v "$OUT_ABS:/data/out" \
+    "$IMAGE" \
+    python3 /opt/app/src/adaptive.py -in "/data/in/$IN_FILE" -out /data/out
+
+echo "== Done =="
+echo "Next: ./3_extract_data.sh -in $OUT_ABS/ocr.txt -out $OUT_ABS"

AGENT.md

@@ -11,12 +11,17 @@ containers; there is no host Python environment to set up.
 ## Pipeline
 
 Three verifiable, independently-inspectable steps, each a thin host
-script that runs the actual work inside the `fuelish/tools` container:
+script that runs the actual work inside the `fuelish/tools` container
+(`server.py` runs the same functions in-process for the live app - see
+`12_adaptive_preprocess_ocr.sh` below for why steps 1+2 are combined
+there):
 
 1. `1_preprocess_image.sh -in <photo> -out <dir>` - grayscale conversion,
-   a mild bilateral filter, and a percentile contrast stretch. Writes
+   a mild bilateral filter, and a percentile contrast stretch (variant 0
-   every intermediate stage (`01_gray.png`, `02_bilateral.png`,
+   of `tooling-container/src/preprocess.py`'s 4 variants - see
-   `03_final.png`) so each step can be checked on its own.
+   `12_adaptive_preprocess_ocr.sh`). Writes every intermediate stage
+   (`01_gray.png`, `02_bilateral.png`, `03_final.png`) so each step can
+   be checked on its own.
 2. `2_ocr_image.sh -in <dir>/03_final.png -out <dir>` - Tesseract OCR
    (German), writes `ocr.txt`.
 3. `3_extract_data.sh -in <dir>/ocr.txt -out <dir>` - sends the raw OCR
@@ -24,13 +29,29 @@ script that runs the actual work inside the `fuelish/tools` container:
    schema, writes `extracted.json`.
 
 Preprocessing choice is evidence-based, not assumed: heavier processing
-(strong denoising, CLAHE, unsharp masking) was tried and measured worse -
+(strong denoising, CLAHE, naive unsharp masking, and a GIMP-derived
-it amplifies the receipt's dotted security-background pattern into OCR
+high-pass/edge-enhance combo - see
-noise. The lighter bilateral+autocontrast approach in
+`testing/2026_10_04_gimp_highpass_nlfilter_experiment.md`) consistently
-`tooling-container/src/preprocess.py` is what survived testing.
+measured worse on the full batch *when applied unconditionally to every
+receipt* - each one amplifies the receipt's dotted security-background
+pattern into OCR noise, even though each visibly helps the specific hard
+receipts it was designed for. That experiment's accepted fix - trying
+heavier variants only on receipts the cheap pass already reads
+implausibly - is `12_adaptive_preprocess_ocr.sh` below.
 
 ## Supporting scripts
 
+- `12_adaptive_preprocess_ocr.sh -in <photo> -out <dir>` - combines steps
+  1+2 adaptively: tries `preprocess.py`'s 4 variants (0=baseline/same as
+  `1_preprocess_image.sh`'s default, up to 3=highpass+edge-enhance
+  combined) cheapest-first, scoring each variant's OCR text with
+  `tooling-container/src/plausibility.py`'s cheap regex-based arithmetic
+  check (no LLM call) and stopping at the first variant that looks
+  receipt-consistent. This is what `test_batch.sh`, `test_single.sh`, and
+  `server.py` actually use (not steps 1+2 directly) - see
+  `testing/2026_10_04_gimp_highpass_nlfilter_experiment.md` for why.
+  Writes `<dir>/variant<N>/` per variant tried plus `<dir>/ocr.txt` for
+  the winning one.
 - `0_run_ollama.sh [-model <name>]` - starts Ollama as its own long-lived
   Docker container (`fuelish-ollama`, model weights in the named
   volume `fuelish-ollama-data`), pulling the model if not already
@@ -50,22 +71,36 @@ noise. The lighter bilateral+autocontrast approach in
   creates sibling `<path>/<name>-scratch/` (intermediate step output per
   receipt) and `<path>/<name>-result/` (final `extracted.json` per
   receipt, plus `summary.txt`). Both are gitignored.
+- `test_single.sh -in <photo.jpg>` - the same pipeline run and
+  ground-truth comparison as `test_batch.sh`, for one receipt instead of
+  a whole directory - useful for re-checking a single known failure
+  without re-running a full batch. Ground truth is optional: if a `*.ods`
+  is found next to the photo it compares against it, otherwise it just
+  prints the extracted JSON. Same sibling `-scratch`/`-result` layout as
+  `test_batch.sh`, scoped to the one receipt.
+- `test_ocr.sh -in <photo.jpg>` - preprocessing + OCR only (no LLM
+  extraction, no ground truth), printing the raw OCR text to stdout.
+  Leaves no files behind (runs in a temp dir, cleaned up on exit) -
+  for quickly checking what Tesseract actually reads off a receipt, e.g.
+  while iterating on `tooling-container/src/preprocess.py`.
 
 ## Current state
 
-Last full run against the 17-receipt ground-truth test set
+Last full run against the 52-receipt ground-truth test set
-(`testdata/2026_new/`, not checked in - see `.gitignore`):
+(`testdata/2026_benzin_kosten/`, not checked in - see `.gitignore`),
+with the adaptive preprocessing escalation described above:
 
-- 15/17 receipts date-matched to ground truth (up from 11/17 with the
+- 40/52 receipts date-matched to ground truth (up from 37/52 with the
-  original plain-grayscale preprocessing)
+  single-variant/baseline-only preprocessing)
-- price_per_litre: 11/15 correct, litres: 9/15, discount: 11/15,
+- price_per_litre: 34/40 correct, litres: 24/40, discount: 34/40,
-  total_price: 12/15
+  total_price: 24/40
 
-Known remaining failure modes, not expected to be fixable by further
+Known remaining failure modes:
-preprocessing tuning:
 
 - OCR garbling on the compact "litres EUR/l price" line on some receipts
-  (the raw text sometimes lacks recoverable digits)
+  (the raw text sometimes lacks recoverable digits) even after trying
+  all 4 preprocessing variants - the adaptive escalation above
+  meaningfully improved this but didn't eliminate it
 - Occasional year-digit misreads in the date causing a false date-match
   failure
 - Low-resolution source photos (well below the ~4000x3000 the good
@@ -79,9 +114,11 @@ text - a subtle failure mode worth remembering if the prompt is revised.
 
 ## Not yet built
 
-Per the original idea (see git history / project notes): an Android app
+The Android app (`android/`), the HTTP job API (`docker-compose.yml`/
-to capture and confirm receipts, a backend server to store raw + confirmed
+`server.py`), and cross-checking total_price against
-submissions, a simple authenticated web UI to query/filter/export
+price_per_litre * litres - discount (`android/.../ReceiptPlausibility.kt`
-(CSV/JSON), and cross-checking total_price against
+on the app side, `tooling-container/src/plausibility.py` on the pipeline
-price_per_litre * litres - discount. None of that exists yet - this repo
+side) all now exist - see `README.md` for how they fit together. Not yet
-currently covers only the extraction pipeline.
+built: a web UI to query/filter/export outside the app, and auth/TLS on
+the server (deliberately out of scope for now - see `server.py`'s
+docstring).

README.md

@@ -45,6 +45,10 @@ tooling-container/     Pipeline source (preprocess.py, ocr.py, extract.py,
 test_batch.sh           Runs the full pipeline over a directory of test
                         receipts and compares against its ground-truth
                         .ods - see AGENT.md's "Current state".
+test_single.sh          test_batch.sh for one receipt instead of a
+                        directory - re-check a single known failure.
+test_ocr.sh             Preprocessing + OCR only, prints the raw OCR
+                        text to stdout - no files left behind.
 docker-compose.yml      Runs the pipeline as a long-lived HTTP job API
                         (api + ollama services) for the Android app.
 android/                The Android app: Kotlin + Jetpack Compose, Room
@@ -134,6 +138,19 @@ receipt at a time:
                                            # file in that directory
 ```
 
+**Re-check one receipt** (ground truth optional - compared if a `*.ods`
+is found next to the photo):
+
+```bash
+./test_single.sh -in testdata/<some-set>/some-receipt.jpg
+```
+
+**Just see what Tesseract reads**, no LLM call, no files left behind:
+
+```bash
+./test_ocr.sh -in testdata/<some-set>/some-receipt.jpg
+```
+
 **Serve it for the app**:
 
 ```bash

test_batch.sh

@@ -1,17 +1,20 @@
 #!/usr/bin/env bash
-# Test-only tool: runs the full container pipeline (1_preprocess_image.sh,
+# Test-only tool: runs the full container pipeline
-# 2_ocr_image.sh, 3_extract_data.sh) over every *.jpg in a test-data
+# (12_adaptive_preprocess_ocr.sh, 3_extract_data.sh) over every *.jpg in
-# directory, then compares the results against that directory's
+# a test-data directory, then compares the results against that
-# ground-truth spreadsheet (benzin_kosten.ods). Everything -
+# directory's ground-truth spreadsheet (benzin_kosten.ods). Everything -
 # preprocessing, OCR, structuring, and the comparison itself - runs
 # inside the tooling container; nothing here depends on a host Python
 # environment.
 #
 # For an input directory <path>/<name>, two sibling directories are
 # created (mirroring <path>/<name>):
-#   <path>/<name>-scratch/<receipt>/  - intermediate step output
+#   <path>/<name>-scratch/<receipt>/  - intermediate step output: one
-#                                       (01_gray.png, 02_bilateral.png,
+#                                       variant<N>/ subdir per
-#                                       03_final.png, ocr.txt) for
+#                                       preprocessing variant tried
+#                                       (01_gray.png, ..., 0X_final.png,
+#                                       ocr.txt each) plus ocr.txt for
+#                                       the winning variant, for
 #                                       inspecting each step on its own
 #   <path>/<name>-result/<receipt>/   - final extracted.json per receipt,
 #                                       plus summary.txt with the overall
@@ -56,8 +59,7 @@ for img in "${IMAGES[@]}"; do
     receipt_result="$RESULT_DIR/$name"
     mkdir -p "$receipt_scratch" "$receipt_result"
     echo "--- $name ---"
-    ./1_preprocess_image.sh -in "$img" -out "$receipt_scratch" >/dev/null
+    ./12_adaptive_preprocess_ocr.sh -in "$img" -out "$receipt_scratch" >/dev/null
-    ./2_ocr_image.sh -in "$receipt_scratch/03_final.png" -out "$receipt_scratch" >/dev/null
     ./3_extract_data.sh -in "$receipt_scratch/ocr.txt" -out "$receipt_result" >/dev/null
 done
 

test_ocr.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Test-only tool: runs preprocessing + OCR (1_preprocess_image.sh,
+# 2_ocr_image.sh - variant 0/baseline only, deliberately not
+# 12_adaptive_preprocess_ocr.sh) on a single receipt photo and prints the
+# raw OCR text to stdout - no LLM extraction step, no ground-truth
+# comparison, and no files left behind (unlike test_single.sh/
+# test_batch.sh, which persist scratch/result directories and go through
+# the real adaptive pipeline). Useful for quickly checking what Tesseract
+# actually reads off one specific preprocessing variant in isolation -
+# e.g. while iterating on tooling-container/src/preprocess.py - without
+# the adaptive step's variant selection in the way, waiting on the LLM
+# call, or needing ground truth.
+#
+# Usage: ./test_ocr.sh -in <photo.jpg>
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+fail() { echo "FAIL: $*" >&2; exit 1; }
+
+IN_PATH=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -in) IN_PATH="${2:-}"; shift 2 ;;
+        *) fail "unknown argument: $1" ;;
+    esac
+done
+[ -n "$IN_PATH" ] || fail "usage: $0 -in <photo.jpg>"
+[ -f "$IN_PATH" ] || fail "input file not found: $IN_PATH"
+
+SCRATCH_DIR="$(mktemp -d)"
+trap 'rm -rf "$SCRATCH_DIR"' EXIT
+
+./1_preprocess_image.sh -in "$IN_PATH" -out "$SCRATCH_DIR" >/dev/null
+./2_ocr_image.sh -in "$SCRATCH_DIR/03_final.png" -out "$SCRATCH_DIR" >/dev/null
+
+cat "$SCRATCH_DIR/ocr.txt"

test_single.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# Test-only tool: runs the full container pipeline
+# (12_adaptive_preprocess_ocr.sh, 3_extract_data.sh) on a single receipt
+# photo - the test_batch.sh equivalent for re-checking one receipt (e.g.
+# a known failure, after a preprocessing/prompt tweak) without re-running
+# a whole directory. If a ground-truth spreadsheet (*.ods) is found next
+# to the photo, compares the result against it the same way
+# test_batch.sh does; otherwise just prints the extracted JSON.
+# Everything - preprocessing, OCR, structuring, and the comparison itself
+# - runs inside the tooling container; nothing here depends on a host
+# Python environment.
+#
+# For an input photo <path>/<name>.jpg, two sibling directories are
+# created (mirroring test_batch.sh's layout, just for one receipt):
+#   <path>/<name>-scratch/<name>/  - intermediate step output: one
+#                                    variant<N>/ subdir per preprocessing
+#                                    variant tried, plus ocr.txt for the
+#                                    winning variant
+#   <path>/<name>-result/<name>/   - extracted.json, plus summary.txt
+#                                    if a ground-truth .ods was found
+#
+# Usage: ./test_single.sh -in <photo.jpg>
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+ROOT="$(pwd)"
+
+fail() { echo "FAIL: $*" >&2; exit 1; }
+
+IN_PATH=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -in) IN_PATH="${2:-}"; shift 2 ;;
+        *) fail "unknown argument: $1" ;;
+    esac
+done
+[ -n "$IN_PATH" ] || fail "usage: $0 -in <photo.jpg>"
+[ -f "$IN_PATH" ] || fail "input file not found: $IN_PATH"
+
+IN_ABS="$(realpath "$IN_PATH")"
+IN_DIR="$(dirname "$IN_ABS")"
+NAME="$(basename "${IN_ABS%.jpg}")"
+
+SCRATCH_DIR="$IN_DIR/$NAME-scratch"
+RESULT_DIR="$IN_DIR/$NAME-result"
+receipt_scratch="$SCRATCH_DIR/$NAME"
+receipt_result="$RESULT_DIR/$NAME"
+mkdir -p "$receipt_scratch" "$receipt_result"
+
+echo "== Running pipeline on $NAME =="
+./12_adaptive_preprocess_ocr.sh -in "$IN_ABS" -out "$receipt_scratch" >/dev/null
+./3_extract_data.sh -in "$receipt_scratch/ocr.txt" -out "$receipt_result" >/dev/null
+
+echo
+echo "== Extracted =="
+cat "$receipt_result/extracted.json"
+echo
+
+ODS_PATH="$(find "$IN_DIR" -maxdepth 1 -name '*.ods' | head -n1)"
+if [ -n "$ODS_PATH" ]; then
+    echo
+    echo "== Comparing against ground truth =="
+
+    IMAGE="${REGISTRY_IMAGE:-fuelish/tools}"
+    if [ -f "$ROOT/registry.env" ]; then
+        # shellcheck disable=SC1091
+        source "$ROOT/registry.env"
+        IMAGE="${REGISTRY_IMAGE:-$IMAGE}"
+    fi
+    VERSION="$(<"$ROOT/tooling-container/VERSION")"
+    IMAGE="$IMAGE:$VERSION"
+
+    ODS_DIR="$(dirname "$ODS_PATH")"
+    ODS_FILE="$(basename "$ODS_PATH")"
+    GROUP_ADD_ARGS=(--group-add "$(stat -c %g "$ODS_DIR")")
+    RESULT_GID="$(stat -c %g "$RESULT_DIR")"
+    [ "$RESULT_GID" = "$(stat -c %g "$ODS_DIR")" ] || GROUP_ADD_ARGS+=(--group-add "$RESULT_GID")
+
+    docker run --rm \
+        --user "$(id -u):$(id -g)" \
+        "${GROUP_ADD_ARGS[@]}" \
+        -v "$ODS_DIR:/data/gt:ro" \
+        -v "$RESULT_DIR:/data/results:ro" \
+        "$IMAGE" \
+        python3 /opt/app/src/compare_ground_truth.py -gt "/data/gt/$ODS_FILE" -results /data/results \
+        | tee "$RESULT_DIR/summary.txt"
+else
+    echo "(no ground-truth .ods found next to $IN_PATH - skipping comparison)"
+fi
+
+echo
+echo "== Done =="
+echo "Intermediate step output: $receipt_scratch/"
+echo "Final result:             $receipt_result/extracted.json"

testing/2026_10_04_gimp_highpass_nlfilter_experiment.md

@@ -0,0 +1,182 @@
+# GIMP High Pass + NL Filter preprocessing experiment
+
+Tested whether a GIMP sharpening/denoising combo, applied to the photo
+*before* our existing `preprocess.py` (grayscale → bilateral filter →
+percentile autocontrast), recovers the compact "litres / EUR-per-litre"
+price line that's the pipeline's worst-performing OCR target (see
+`AGENT.md`'s "Known remaining failure modes"). Three stress-test receipts
+with known-bad baseline OCR on that line (`testdata/2026_stress_test/`,
+not checked in):
+
+- **A** `IMG_20260525_182252232` — ground truth 29,31 l / 1,949 EUR/l
+- **B** `IMG_20260525_182144541` — ground truth 17,88 l / 1,989 EUR/l
+- **C** `IMG_20260929_182218993` — ground truth 18,86 l / 2,129 EUR/l
+
+The recipe (discovered manually in GIMP's GUI, applied to the full
+uncropped color photo):
+
+1. **Gaussian Blur** (Size X=Y=4.15, Filter=Auto, Abyss=Clamp) with
+   **Blending Options: Mode=Grain Extract, Opacity=100** set directly in
+   the filter dialog — GIMP 2.10's GEGL-based "High Pass" filter has no
+   Script-Fu/PDB wrapper at all, so this is a Script-Fu-reachable
+   reproduction of it via the classic manual technique (blur, then
+   grain-extract-blend against the original).
+2. **Levels**: input low=58, gamma=0.96, high=135; output 0–255.
+3. **NL Filter**, filter-type=2 (edge-enhancement), alpha=0.69,
+   radius=0.52.
+
+## Result
+
+Every variant beat the pipeline's baseline (no GIMP step) on at least one
+field, for all three images — confirming the general direction helps.
+But no single step generalized on its own; only the full 2-step recipe
+was consistently good-to-exact across all three:
+
+| Image (ground truth) | Baseline | 1. Blur+GrainExtract | 2. +Levels | 3. NL Filter alone | 4. Full recipe (2+3) |
+|---|---|---|---|---|---|
+| **A** (29,31l / 1,949) | both unreadable | price≈missing, litres✗ | litres 29,91✗(1 digit) / price **1,949✓** | litres 28,31✗(1 digit) / price 1,983≈ | litres **29,31✓** / price 1,993≈ |
+| **B** (17,88l / 1,989) | litres+price both wrong | litres **17,88✓** / price 1,909≈ | litres **17,88✓** / price **1,989✓** | litres✗missing / price≈1989 (garbled) | litres 17,B8≈(1 char) / price **1,989✓** |
+| **C** (18,86l / 2,129) | litres wrong, price✗missing | litres **18,86✓** / price✗missing | litres **0.86✗** (broke it!) / price≈2,129 | litres **18,86✓** / price✗missing | litres **18,86✓** / price≈2,129 |
+
+Note step 2 (Levels) actively *breaks* C's litres reading that was fine
+after step 1 alone — a clear example of a step helping one receipt while
+hurting another, which is why no single step is a safe generalization on
+its own even though each one looks promising in isolation.
+
+## Root causes / methodology notes
+
+1. **GIMP's "High Pass" filter has no PDB/Script-Fu wrapper** — confirmed
+   by querying the full procedure database (`gimp-procedural-db-query`)
+   for anything matching `high.?pass`; only `plug-in-edge`,
+   `plug-in-unsharp-mask`, `plug-in-sharpen`, and `gimp-selection-sharpen`
+   exist. It's GEGL-only in this GIMP version (2.10.30). The manual
+   Gaussian-Blur-with-Grain-Extract-blend technique is the only
+   automatable equivalent.
+2. **GIMP's own "default" vs "legacy" grain-extract blend-mode variants
+   turned out not to matter** — tested both (`LAYER-MODE-GRAIN-EXTRACT`
+   = 46 and `LAYER-MODE-GRAIN-EXTRACT-LEGACY` = 20 against the same
+   input): numerically near-identical output either way.
+3. **A real, unresolved gap remains between GIMP's GUI-applied
+   Gaussian-Blur-with-in-dialog-blend and a classic two-layer Script-Fu
+   reproduction** (duplicate layer → blur → `gimp-layer-set-mode` →
+   flatten) of the *same nominal parameters*. The GUI version's grain-
+   extract output has roughly double the standard deviation (spread) of
+   the scripted version's, confirmed via direct pixel-value percentile
+   comparison (user's real `_5_1.jpg`: mean 118.5, std 40.4; scripted
+   equivalent: mean 128.0, std 8.7–23.2 across a wide blur-radius sweep
+   from 8 to 180 — never converges, ruling out a simple radius/scale
+   miscalibration). `plug-in-gauss`'s legacy-compat wrapper applies an
+   internal ~0.32× multiplier converting its "radius" input to GEGL's
+   `std-dev`, and the Grain Extract formula itself was confirmed correct
+   from GEGL source (`CLAMP(lower - upper + 0.5)` in normalized space).
+   The actual cause (most likely GEGL's abyss-policy/edge-handling or an
+   image-precision difference) was not pinned down — **GIMP batch
+   scripting cannot currently reproduce this step's real-world output
+   closely enough to trust for pipeline automation.**
+4. Separately (fixed along the way, not the cause of the above): GIMP
+   batch-mode has two sharp edges worth remembering if scripted again —
+   (a) `gimp-console -b <script> -b '(gimp-quit 0)'` as two separate
+   `-b` flags does not reliably terminate the process on error or even
+   on success; always pipe one script via stdin (`-b -`) with
+   `(gimp-quit 0)` as the last form *inside* that same stream.
+   (b) `gimp-image-insert-layer`'s position argument needs an explicit
+   `0` (top of stack) — `-1` does not reliably mean "top" and silently
+   produces a no-op-looking flatten (new layer ends up under the base
+   layer, which then opaquely covers it).
+
+## Native Python reimplementation and full-batch result — rejected
+
+Reimplemented the two steps natively (no GIMP dependency) in
+`tooling-container/src/preprocess.py`, inserted before the existing
+bilateral+autocontrast stage:
+
+- `highpass_substitute()`: `cv2.GaussianBlur` + the confirmed
+  `clip(gray - blurred + 128)` grain-extract formula + a linear
+  levels-stretch (low=58, high=135, gamma=0.96 — the same values as the
+  GIMP recipe).
+- `edge_enhance()`: a calibrated unsharp-mask-style boost
+  (`gray + 0.69 * (gray - GaussianBlur(gray, sigma=0.52))`) as the
+  OpenCV substitute for GIMP's NL Filter edge-enhancement mode (no direct
+  OpenCV equivalent exists).
+- Both steps' blur radii scale with image width (`/ 768`, the width A/B
+  were shot at) — necessary because C is 4096×3072, roughly 4-5x A/B's
+  linear resolution, and a fixed pixel radius calibrated at one scale
+  clearly under/over-shoots at the other (confirmed: before scaling, C's
+  litres read `0.86` instead of `18,86`; after scaling, `18,86✓`).
+
+On the A/B/C set, this **outperformed the GIMP-based matrix** — litres
+exact on all three images, price exact-or-close on all three (no GIMP
+dependency, no reproduction-gap uncertainty).
+
+**But it failed full-batch validation** against the real 52-receipt
+ground-truth set (`testdata/2026_benzin_kosten/`), the required next step
+per this project's evidence-based philosophy before touching the real
+pipeline:
+
+| Metric | Baseline (before) | With new preprocessing | Change |
+|---|---|---|---|
+| Date-matched | 37/52 | 30/52 | **-7** |
+| price_per_litre | 27/37 (73%) | 18/30 (60%) | ↓ |
+| litres (the targeted field) | 19/37 (51%) | 12/30 (40%) | ↓ worse |
+| discount | 29/37 (78%) | 22/30 (73%) | ↓ |
+| total_price | 23/37 (62%) | 18/30 (60%) | ↓ |
+
+Regression across every metric, including the exact field the change
+targeted — the same failure pattern as the earlier-rejected unsharp-
+masking attempt (see `AGENT.md`): it visibly helps the specific hard
+receipts it was designed for, but amplifies the dotted security-
+background pattern into OCR noise on receipts that were already reading
+fine, making the batch worse overall. **Reverted** from
+`preprocess.py`/the numbered scripts/docs; not going into the pipeline.
+
+## Adaptive per-receipt escalation — accepted
+
+This is now the second distinct sharpening-family approach (after naive
+unsharp masking - see `AGENT.md`) confirmed to fail *when applied
+globally*: strong locally on hard receipts, net-negative on the full
+batch, because it amplifies the security background pattern into noise
+on receipts the cheap baseline already reads fine. Both failures point
+at the same fix: apply it *selectively*, only to receipts the cheap pass
+already reads poorly, rather than to every photo.
+
+Implemented as `tooling-container/src/preprocess.py`'s four variants
+(0=baseline, 1=+`highpass_substitute`, 2=+`edge_enhance`, 3=+both) tried
+cheapest-first by `tooling-container/src/adaptive.py`, escalating only
+when the current variant's raw OCR text fails a cheap arithmetic-
+consistency check (`tooling-container/src/plausibility.py`:
+`price_per_litre * litres - discount ≈ total_price` within 0.05 EUR —
+the same identity and tolerance `android/.../ReceiptPlausibility.kt` and
+`compare_ground_truth.py` already use, applied here via regex-extracted
+candidate numbers straight from OCR text so the check needs no LLM call
+and runs once per variant instead of once per receipt). This keeps the
+expensive LLM structuring call to exactly one per receipt regardless of
+how many variants are tried, since tier selection happens entirely
+before that call.
+
+Full-batch result against `testdata/2026_benzin_kosten/` (same baseline
+as above):
+
+| Metric | Baseline | Adaptive | Change |
+|---|---|---|---|
+| Date-matched | 37/52 | 40/52 | **+3** |
+| price_per_litre | 27/37 (73%) | 34/40 (85%) | **↑** |
+| litres | 19/37 (51%) | 24/40 (60%) | **↑** |
+| discount | 29/37 (78%) | 34/40 (85%) | **↑** |
+| total_price | 23/37 (62%) | 24/40 (60%) | ≈ flat (+1 in count) |
+
+Improves every metric except total_price, which is essentially flat.
+**Accepted** — wired into `server.py`, `test_batch.sh`, `test_single.sh`,
+and the new `12_adaptive_preprocess_ocr.sh` (replacing the separate
+`1_preprocess_image.sh` + `2_ocr_image.sh` calls those two use;
+`1_preprocess_image.sh`/`2_ocr_image.sh` themselves are unchanged and
+still useful for inspecting one variant in isolation).
+
+## Conclusion
+
+The litres/price-per-litre line was the pipeline's hardest OCR target
+specifically because every fix tried so far has been a global tradeoff:
+good for hard receipts, bad for easy ones, on the same underlying cause
+(amplifying the security background pattern). Making the fix conditional
+on the cheap pass already looking wrong - judged cheaply, without an
+LLM call - was what actually let the hard-receipt win through without
+the easy-receipt cost.

tooling-container/src/adaptive.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""Adaptive preprocessing: tries preprocess.py's variants (0=cheapest,
+up to VARIANT_COUNT-1=most elaborate) in order, OCRing each and scoring
+the result with plausibility.best_residual() - no LLM call - stopping at
+the first variant whose OCR text already looks receipt-consistent.
+
+Exists because always applying the heavier variants regressed the full
+test batch even though they clearly help the specific hard receipts they
+were designed for (see
+testing/2026_10_04_gimp_highpass_nlfilter_experiment.md): escalating only
+when the cheap variant already looks implausible avoids that cost - and
+that risk - on the majority of receipts the baseline already reads
+correctly, while keeping the expensive LLM structuring call to exactly
+one per receipt no matter how many variants are tried.
+"""
+import argparse
+import os
+
+from ocr import ocr as run_ocr
+from plausibility import best_residual, is_plausible
+from preprocess import VARIANT_COUNT, preprocess
+
+
+def preprocess_and_ocr_adaptive(image_path: str, out_dir: str) -> dict:
+    """Returns {"ocr_text", "final_image_path", "variant", "residual",
+    "variants_tried"} - "variant" is whichever one was kept (the first
+    plausible one, or else the one with the lowest residual), not
+    necessarily the last one tried."""
+    best = None  # (residual, ocr_text, final_path, variant)
+    variants_tried = 0
+
+    for variant in range(VARIANT_COUNT):
+        variant_dir = os.path.join(out_dir, f"variant{variant}")
+        final_path = preprocess(image_path, variant_dir, variant=variant)
+        ocr_txt_path = run_ocr(final_path, variant_dir)
+        with open(ocr_txt_path, "r", encoding="utf-8") as f:
+            ocr_text = f.read()
+        variants_tried += 1
+
+        residual = best_residual(ocr_text)
+        if best is None or residual < best[0]:
+            best = (residual, ocr_text, final_path, variant)
+        if is_plausible(residual):
+            break
+
+    residual, ocr_text, final_path, variant = best
+    return {
+        "ocr_text": ocr_text,
+        "final_image_path": final_path,
+        "variant": variant,
+        "residual": residual,
+        "variants_tried": variants_tried,
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-in", dest="in_path", required=True)
+    parser.add_argument("-out", dest="out_dir", required=True)
+    args = parser.parse_args()
+
+    result = preprocess_and_ocr_adaptive(args.in_path, args.out_dir)
+
+    ocr_path = os.path.join(args.out_dir, "ocr.txt")
+    with open(ocr_path, "w", encoding="utf-8") as f:
+        f.write(result["ocr_text"])
+
+    print(
+        f"variant={result['variant']} "
+        f"variants_tried={result['variants_tried']} "
+        f"residual={result['residual']:.3f} "
+        f"final_image={result['final_image_path']}"
+    )
+    print(ocr_path)
+
+
+if __name__ == "__main__":
+    main()

tooling-container/src/plausibility.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""Cheap, LLM-free plausibility scoring of raw OCR text from a German fuel
+receipt. Used by adaptive.py to rank several preprocessing variants' OCR
+output against each other - and decide whether to bother trying a more
+elaborate variant at all - without paying for an LLM call per variant.
+
+Mirrors the same arithmetic identity and EUR tolerance
+android/.../ReceiptPlausibility.kt and compare_ground_truth.py use
+(total_price == price_per_litre * litres - discount, within 0.05 EUR),
+applied here to candidate numbers found directly in raw OCR text rather
+than already-structured fields, since this runs before (and to help pick
+input for) the single structuring LLM call.
+"""
+import re
+
+TOLERANCE_EUR = 0.05
+
+_NUMBER_RE = re.compile(r"(\d{1,3}),(\d{2,3})")
+
+# Plausible-value ranges for German fuel receipts, used to keep the
+# combination search small and to reject obviously-wrong candidates
+# (transaction IDs, card digits, etc. pulled in by the same regex).
+_PRICE_RANGE = (0.5, 3.5)
+_LITRES_RANGE = (1.0, 150.0)
+_TOTAL_RANGE = (1.0, 500.0)
+_DISCOUNT_RANGE = (0.0, 50.0)
+
+
+def _candidates(ocr_text: str) -> list[float]:
+    out = []
+    for m in _NUMBER_RE.finditer(ocr_text):
+        try:
+            out.append(float(f"{m.group(1)}.{m.group(2)}"))
+        except ValueError:
+            continue
+    return out
+
+
+def best_residual(ocr_text: str) -> float:
+    """Smallest |price_per_litre * litres - discount - total_price| over
+    plausible-range combinations of numbers found in ocr_text. Lower is
+    more plausible; see is_plausible() for the pass/fail threshold. inf
+    if the text doesn't contain enough candidate numbers in the right
+    ranges to try any combination at all."""
+    candidates = _candidates(ocr_text)
+    prices = [c for c in candidates if _PRICE_RANGE[0] <= c <= _PRICE_RANGE[1]]
+    litres = [c for c in candidates if _LITRES_RANGE[0] <= c <= _LITRES_RANGE[1]]
+    totals = [c for c in candidates if _TOTAL_RANGE[0] <= c <= _TOTAL_RANGE[1]]
+    discounts = [0.0] + [
+        c for c in candidates if _DISCOUNT_RANGE[0] < c <= _DISCOUNT_RANGE[1]
+    ]
+
+    best = float("inf")
+    for p in prices:
+        for litre in litres:
+            if litre == p:
+                continue
+            price_times_litres = p * litre
+            for t in totals:
+                if t == p or t == litre:
+                    continue
+                for d in discounts:
+                    residual = abs(price_times_litres - d - t)
+                    if residual < best:
+                        best = residual
+    return best
+
+
+def is_plausible(residual: float) -> bool:
+    return residual <= TOLERANCE_EUR

tooling-container/src/preprocess.py

@@ -8,12 +8,22 @@ the output directory (01_gray, 02_bilateral, 03_final) so the effect of
 each step can be inspected on its own before trusting the final image to
 OCR.
 
-Heavier processing (strong denoising, local/CLAHE contrast, unsharp
+Heavier processing (strong denoising, local/CLAHE contrast, naive unsharp
 masking) was tried and tested worse: each one amplifies the security
 background pattern into OCR noise rather than suppressing it. This
-lighter two-step version was chosen because, across a set of receipts
+lighter two-step version (variant 0 below, and the default) was chosen
-that the plain-grayscale baseline OCR'd incorrectly, it recovered data
+because, across a set of receipts that the plain-grayscale baseline
-the baseline missed and never made a previously-correct read worse.
+OCR'd incorrectly, it recovered data the baseline missed and never made
+a previously-correct read worse.
+
+Variants 1-3 add a high-pass-style detail boost and/or edge enhancement
+before this same bilateral+autocontrast stage. Applied unconditionally
+to every receipt, this combination also tested worse on the full batch
+for the same reason (see
+testing/2026_10_04_gimp_highpass_nlfilter_experiment.md) - but it
+noticeably helps the specific hard receipts it was designed for, so
+adaptive.py tries these variants only when the cheap variant 0 already
+looks implausible, rather than applying them globally.
 """
 import argparse
 import os
@@ -22,6 +32,13 @@ import cv2
 import numpy as np
 from PIL import Image, ImageOps
 
+# Width the variant 1-3 parameters below were calibrated at; their blur
+# radii scale relative to this so they hold up across this project's test
+# photos, which range from ~768px to ~4096px wide.
+REFERENCE_WIDTH = 768
+
+VARIANT_COUNT = 4  # 0=baseline, 1=+highpass, 2=+edge-enhance, 3=+both
+
 
 def load_grayscale(path: str) -> np.ndarray:
     img = Image.open(path)
@@ -29,23 +46,66 @@ def load_grayscale(path: str) -> np.ndarray:
     return np.array(img)
 
 
+def highpass_substitute(
+    gray: np.ndarray, blur_sigma=4.15, low_in=58, high_in=135, gamma=0.96
+) -> np.ndarray:
+    scale = gray.shape[1] / REFERENCE_WIDTH
+    blurred = cv2.GaussianBlur(gray, (0, 0), sigmaX=blur_sigma * scale)
+    grain = np.clip(gray.astype(np.float32) - blurred.astype(np.float32) + 128.0, 0, 255)
+    normalized = np.clip((grain - low_in) / (high_in - low_in), 0, 1)
+    gamma_applied = np.power(normalized, 1.0 / gamma)
+    return np.clip(gamma_applied * 255.0, 0, 255).astype(np.uint8)
+
+
+def edge_enhance(gray: np.ndarray, amount=0.69, sigma=0.52) -> np.ndarray:
+    scale = gray.shape[1] / REFERENCE_WIDTH
+    blurred = cv2.GaussianBlur(gray, (0, 0), sigmaX=sigma * scale)
+    sharpened = gray.astype(np.float32) + amount * (
+        gray.astype(np.float32) - blurred.astype(np.float32)
+    )
+    return np.clip(sharpened, 0, 255).astype(np.uint8)
+
+
 def autocontrast(gray: np.ndarray, low_pct=2, high_pct=98) -> np.ndarray:
     lo, hi = np.percentile(gray, (low_pct, high_pct))
     stretched = (gray.astype(np.float32) - lo) * 255.0 / (hi - lo)
     return np.clip(stretched, 0, 255).astype(np.uint8)
 
 
-def preprocess(path: str, out_dir: str) -> str:
+def preprocess(path: str, out_dir: str, variant: int = 0) -> str:
+    """variant: 0=baseline (bilateral+autocontrast only, unchanged default
+    behavior), 1=+highpass_substitute, 2=+edge_enhance, 3=+both. See
+    adaptive.py for why this is a parameter rather than always-on."""
+    if not 0 <= variant < VARIANT_COUNT:
+        raise ValueError(f"variant must be 0-{VARIANT_COUNT - 1}, got {variant}")
+
     os.makedirs(out_dir, exist_ok=True)
 
-    gray = load_grayscale(path)
+    stage = load_grayscale(path)
-    cv2.imwrite(os.path.join(out_dir, "01_gray.png"), gray)
+    cv2.imwrite(os.path.join(out_dir, "01_gray.png"), stage)
 
-    bilateral = cv2.bilateralFilter(gray, d=5, sigmaColor=50, sigmaSpace=50)
+    # Stage filenames only extend beyond the original 01/02/03 numbering
-    cv2.imwrite(os.path.join(out_dir, "02_bilateral.png"), bilateral)
+    # for variants 1-3, which adaptive.py always runs in their own
+    # per-variant subdirectory - so this never changes what
+    # 1_preprocess_image.sh's default (variant 0) manual single-stage
+    # inspection produces, or any doc/script that hardcodes 03_final.png.
+    next_n = 2
+    if variant in (1, 3):
+        stage = highpass_substitute(stage)
+        cv2.imwrite(os.path.join(out_dir, f"0{next_n}_highpass.png"), stage)
+        next_n += 1
+
+    if variant in (2, 3):
+        stage = edge_enhance(stage)
+        cv2.imwrite(os.path.join(out_dir, f"0{next_n}_edgeenhance.png"), stage)
+        next_n += 1
+
+    bilateral = cv2.bilateralFilter(stage, d=5, sigmaColor=50, sigmaSpace=50)
+    cv2.imwrite(os.path.join(out_dir, f"0{next_n}_bilateral.png"), bilateral)
+    next_n += 1
 
     final = autocontrast(bilateral)
-    final_path = os.path.join(out_dir, "03_final.png")
+    final_path = os.path.join(out_dir, f"0{next_n}_final.png")
     cv2.imwrite(final_path, final)
 
     return final_path
@@ -55,9 +115,10 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("-in", dest="in_path", required=True)
     parser.add_argument("-out", dest="out_dir", required=True)
+    parser.add_argument("-variant", dest="variant", type=int, default=0)
     args = parser.parse_args()
 
-    final_path = preprocess(args.in_path, args.out_dir)
+    final_path = preprocess(args.in_path, args.out_dir, args.variant)
     print(final_path)
 
 

tooling-container/src/server.py

@@ -24,9 +24,8 @@ import requests
 from fastapi import FastAPI, HTTPException, UploadFile
 from fastapi.responses import JSONResponse
 
+from adaptive import preprocess_and_ocr_adaptive
 from extract import structure
-from ocr import ocr as run_ocr
-from preprocess import preprocess
 
 OLLAMA_BASE_URL = os.environ.get("OLLAMA_BASE_URL", "http://127.0.0.1:11434")
 OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "qwen2.5:3b")
@@ -85,15 +84,10 @@ def health() -> dict:
 def _process_job(job_id: str, image_path: str, scratch_dir: str) -> None:
     try:
         _set_job(job_id, status="preprocessing")
-        final_path = preprocess(image_path, scratch_dir)
+        adaptive_result = preprocess_and_ocr_adaptive(image_path, scratch_dir)
-
-        _set_job(job_id, status="ocr")
-        ocr_txt_path = run_ocr(final_path, scratch_dir)
-        with open(ocr_txt_path, "r", encoding="utf-8") as f:
-            ocr_text = f.read()
 
         _set_job(job_id, status="extracting")
-        result = structure(ocr_text)
+        result = structure(adaptive_result["ocr_text"])
 
         _set_job(job_id, status="done", result=result)
     except Exception as exc:  # noqa: BLE001 - reported to the client, not swallowed