fuelish/AGENT.md

fuelish

Extracts structured data from photos of German fuel/petrol receipts: petrol station, transaction number, date, time, price per litre, litres, discount, and total price.

The extraction pipeline is self-hosted (no cloud AI APIs) and is built to eventually run on a Raspberry Pi 4 cluster under k3s. Everything runs in containers; there is no host Python environment to set up.

Pipeline

Three verifiable, independently-inspectable steps, each a thin host script that runs the actual work inside the fuelish/tools container (server.py runs the same functions in-process for the live app - see 12_adaptive_preprocess_ocr.sh below for why steps 1+2 are combined there):

1. 1_preprocess_image.sh -in <photo> -out <dir> - grayscale conversion, a mild bilateral filter, and a percentile contrast stretch (variant 0 of tooling-container/src/preprocess.py's 4 variants - see 12_adaptive_preprocess_ocr.sh). Writes every intermediate stage (01_gray.png, 02_bilateral.png, 03_final.png) so each step can be checked on its own.
2. 2_ocr_image.sh -in <dir>/03_final.png -out <dir> - Tesseract OCR (German), writes ocr.txt.
3. 3_extract_data.sh -in <dir>/ocr.txt -out <dir> - sends the raw OCR text to a small local LLM (qwen2.5:3b, served by Ollama) with a JSON schema, writes extracted.json.

Preprocessing choice is evidence-based, not assumed: heavier processing (strong denoising, CLAHE, naive unsharp masking, and a GIMP-derived high-pass/edge-enhance combo - see testing/2026_10_04_gimp_highpass_nlfilter_experiment.md) consistently measured worse on the full batch when applied unconditionally to every receipt - each one amplifies the receipt's dotted security-background pattern into OCR noise, even though each visibly helps the specific hard receipts it was designed for. That experiment's accepted fix - trying heavier variants only on receipts the cheap pass already reads implausibly - is 12_adaptive_preprocess_ocr.sh below.

Supporting scripts

• 12_adaptive_preprocess_ocr.sh -in <photo> -out <dir> - combines steps 1+2 adaptively: tries preprocess.py's 4 variants (0=baseline/same as 1_preprocess_image.sh's default, up to 3=highpass+edge-enhance combined) cheapest-first, scoring each variant's OCR text with tooling-container/src/plausibility.py's cheap regex-based arithmetic check (no LLM call) and stopping at the first variant that looks receipt-consistent. This is what test_batch.sh, test_single.sh, and server.py actually use (not steps 1+2 directly) - see testing/2026_10_04_gimp_highpass_nlfilter_experiment.md for why. Writes <dir>/variant<N>/ per variant tried plus <dir>/ocr.txt for the winning one.
• 0_run_ollama.sh [-model <name>] - starts Ollama as its own long-lived Docker container (fuelish-ollama, model weights in the named volume fuelish-ollama-data), pulling the model if not already present. Deliberately separate from the tooling container: Ollama needs to stay resident in memory with the model loaded, while the pipeline steps are short-lived and stateless (one docker run per receipt). 3_extract_data.sh reaches it via --network host at http://127.0.0.1:11434 by default (override with OLLAMA_URL/ OLLAMA_MODEL; OLLAMA_NUM_CTX (default 3072) and OLLAMA_NUM_THREAD (default: Ollama's) set the request options).
• build-image.sh / upload-image.sh - build/push the tooling-container/ image, tagged from tooling-container/VERSION. Image name defaults to fuelish/tools, overridable via registry.env (copy from registry.env.example; gitignored).
• test_batch.sh -in <test-data-dir> - runs the full pipeline over every *.jpg in a directory and compares results against that directory's ground-truth spreadsheet (*.ods). For an input dir <path>/<name> it creates sibling <path>/<name>-scratch/ (intermediate step output per receipt) and <path>/<name>-result/ (final extracted.json per receipt, plus summary.txt). Both are gitignored.
• test_single.sh -in <photo.jpg> - the same pipeline run and ground-truth comparison as test_batch.sh, for one receipt instead of a whole directory - useful for re-checking a single known failure without re-running a full batch. Ground truth is optional: if a *.ods is found next to the photo it compares against it, otherwise it just prints the extracted JSON. Same sibling -scratch/-result layout as test_batch.sh, scoped to the one receipt.
• test_ocr.sh -in <photo.jpg> - preprocessing + OCR only (no LLM extraction, no ground truth), printing the raw OCR text to stdout. Leaves no files behind (runs in a temp dir, cleaned up on exit) - for quickly checking what Tesseract actually reads off a receipt, e.g. while iterating on tooling-container/src/preprocess.py.

Current state

Last full run against the 52-receipt ground-truth test set (testdata/2026_benzin_kosten/, not checked in - see .gitignore), with the adaptive preprocessing escalation described above:

• 40/52 receipts date-matched to ground truth (up from 37/52 with the single-variant/baseline-only preprocessing)
• price_per_litre: 34/40 correct, litres: 24/40, discount: 34/40, total_price: 24/40

Known remaining failure modes:

• OCR garbling on the compact "litres EUR/l price" line on some receipts (the raw text sometimes lacks recoverable digits) even after trying all 4 preprocessing variants - the adaptive escalation above meaningfully improved this but didn't eliminate it
• Occasional year-digit misreads in the date causing a false date-match failure
• Low-resolution source photos (well below the ~4000x3000 the good results come from) OCR poorly regardless of preprocessing

The structuring prompt in tooling-container/src/extract.py uses deliberately non-colliding example values. An earlier version used real receipt values as worked examples, which caused the small model to copy/anchor on those exact numbers instead of reading the actual OCR text - a subtle failure mode worth remembering if the prompt is revised.

Not yet built

The Android app (android/), the HTTP job API (docker-compose.yml/ server.py), and cross-checking total_price against price_per_litre * litres - discount (android/.../ReceiptPlausibility.kt on the app side, tooling-container/src/plausibility.py on the pipeline side) all now exist - see README.md for how they fit together. Not yet built: a web UI to query/filter/export outside the app, and auth/TLS on the server (deliberately out of scope for now - see server.py's docstring).