# fuelish Extracts structured data from photos of German fuel/petrol receipts: petrol station, transaction number, date, time, price per litre, litres, discount, and total price. The extraction pipeline is self-hosted (no cloud AI APIs) and is built to eventually run on a Raspberry Pi 4 cluster under k3s. Everything runs in containers; there is no host Python environment to set up. ## Pipeline Three verifiable, independently-inspectable steps, each a thin host script that runs the actual work inside the `fuelish/tools` container (`server.py` runs the same functions in-process for the live app - see `12_adaptive_preprocess_ocr.sh` below for why steps 1+2 are combined there): 1. `1_preprocess_image.sh -in -out ` - grayscale conversion, a mild bilateral filter, and a percentile contrast stretch (variant 0 of `tooling-container/src/preprocess.py`'s 4 variants - see `12_adaptive_preprocess_ocr.sh`). Writes every intermediate stage (`01_gray.png`, `02_bilateral.png`, `03_final.png`) so each step can be checked on its own. 2. `2_ocr_image.sh -in /03_final.png -out ` - Tesseract OCR (German), writes `ocr.txt`. 3. `3_extract_data.sh -in /ocr.txt -out ` - sends the raw OCR text to a small local LLM (`qwen2.5:3b`, served by Ollama) with a JSON schema, writes `extracted.json`. Preprocessing choice is evidence-based, not assumed: heavier processing (strong denoising, CLAHE, naive unsharp masking, and a GIMP-derived high-pass/edge-enhance combo - see `testing/2026_10_04_gimp_highpass_nlfilter_experiment.md`) consistently measured worse on the full batch *when applied unconditionally to every receipt* - each one amplifies the receipt's dotted security-background pattern into OCR noise, even though each visibly helps the specific hard receipts it was designed for. That experiment's accepted fix - trying heavier variants only on receipts the cheap pass already reads implausibly - is `12_adaptive_preprocess_ocr.sh` below. ## Supporting scripts - `12_adaptive_preprocess_ocr.sh -in -out ` - combines steps 1+2 adaptively: tries `preprocess.py`'s 4 variants (0=baseline/same as `1_preprocess_image.sh`'s default, up to 3=highpass+edge-enhance combined) cheapest-first, scoring each variant's OCR text with `tooling-container/src/plausibility.py`'s cheap regex-based arithmetic check (no LLM call) and stopping at the first variant that looks receipt-consistent. This is what `test_batch.sh`, `test_single.sh`, and `server.py` actually use (not steps 1+2 directly) - see `testing/2026_10_04_gimp_highpass_nlfilter_experiment.md` for why. Writes `/variant/` per variant tried plus `/ocr.txt` for the winning one. - `0_run_ollama.sh [-model ]` - starts Ollama as its own long-lived Docker container (`fuelish-ollama`, model weights in the named volume `fuelish-ollama-data`), pulling the model if not already present. Deliberately separate from the tooling container: Ollama needs to stay resident in memory with the model loaded, while the pipeline steps are short-lived and stateless (one `docker run` per receipt). `3_extract_data.sh` reaches it via `--network host` at `http://127.0.0.1:11434` by default (override with `OLLAMA_URL`/ `OLLAMA_MODEL`). - `build-image.sh` / `upload-image.sh` - build/push the `tooling-container/` image, tagged from `tooling-container/VERSION`. Image name defaults to `fuelish/tools`, overridable via `registry.env` (copy from `registry.env.example`; gitignored). - `test_batch.sh -in ` - runs the full pipeline over every `*.jpg` in a directory and compares results against that directory's ground-truth spreadsheet (`*.ods`). For an input dir `/` it creates sibling `/-scratch/` (intermediate step output per receipt) and `/-result/` (final `extracted.json` per receipt, plus `summary.txt`). Both are gitignored. - `test_single.sh -in ` - the same pipeline run and ground-truth comparison as `test_batch.sh`, for one receipt instead of a whole directory - useful for re-checking a single known failure without re-running a full batch. Ground truth is optional: if a `*.ods` is found next to the photo it compares against it, otherwise it just prints the extracted JSON. Same sibling `-scratch`/`-result` layout as `test_batch.sh`, scoped to the one receipt. - `test_ocr.sh -in ` - preprocessing + OCR only (no LLM extraction, no ground truth), printing the raw OCR text to stdout. Leaves no files behind (runs in a temp dir, cleaned up on exit) - for quickly checking what Tesseract actually reads off a receipt, e.g. while iterating on `tooling-container/src/preprocess.py`. ## Current state Last full run against the 52-receipt ground-truth test set (`testdata/2026_benzin_kosten/`, not checked in - see `.gitignore`), with the adaptive preprocessing escalation described above: - 40/52 receipts date-matched to ground truth (up from 37/52 with the single-variant/baseline-only preprocessing) - price_per_litre: 34/40 correct, litres: 24/40, discount: 34/40, total_price: 24/40 Known remaining failure modes: - OCR garbling on the compact "litres EUR/l price" line on some receipts (the raw text sometimes lacks recoverable digits) even after trying all 4 preprocessing variants - the adaptive escalation above meaningfully improved this but didn't eliminate it - Occasional year-digit misreads in the date causing a false date-match failure - Low-resolution source photos (well below the ~4000x3000 the good results come from) OCR poorly regardless of preprocessing The structuring prompt in `tooling-container/src/extract.py` uses deliberately non-colliding example values. An earlier version used real receipt values as worked examples, which caused the small model to copy/anchor on those exact numbers instead of reading the actual OCR text - a subtle failure mode worth remembering if the prompt is revised. ## Not yet built The Android app (`android/`), the HTTP job API (`docker-compose.yml`/ `server.py`), and cross-checking total_price against price_per_litre * litres - discount (`android/.../ReceiptPlausibility.kt` on the app side, `tooling-container/src/plausibility.py` on the pipeline side) all now exist - see `README.md` for how they fit together. Not yet built: a web UI to query/filter/export outside the app, and auth/TLS on the server (deliberately out of scope for now - see `server.py`'s docstring).