fuelish/AGENT.md

fuelish

Extracts structured data from photos of German fuel/petrol receipts: petrol station, transaction number, date, time, price per litre, litres, discount, and total price.

The extraction pipeline is self-hosted (no cloud AI APIs) and is built to eventually run on a Raspberry Pi 4 cluster under k3s. Everything runs in containers; there is no host Python environment to set up.

Pipeline

Three verifiable, independently-inspectable steps, each a thin host script that runs the actual work inside the fuelish/tools container:

1. 1_preprocess_image.sh -in <photo> -out <dir> - grayscale conversion, a mild bilateral filter, and a percentile contrast stretch. Writes every intermediate stage (01_gray.png, 02_bilateral.png, 03_final.png) so each step can be checked on its own.
2. 2_ocr_image.sh -in <dir>/03_final.png -out <dir> - Tesseract OCR (German), writes ocr.txt.
3. 3_extract_data.sh -in <dir>/ocr.txt -out <dir> - sends the raw OCR text to a small local LLM (qwen2.5:3b, served by Ollama) with a JSON schema, writes extracted.json.

Preprocessing choice is evidence-based, not assumed: heavier processing (strong denoising, CLAHE, unsharp masking) was tried and measured worse - it amplifies the receipt's dotted security-background pattern into OCR noise. The lighter bilateral+autocontrast approach in tooling-container/src/preprocess.py is what survived testing.

Supporting scripts

• 0_run_ollama.sh [-model <name>] - starts Ollama as its own long-lived Docker container (fuelish-ollama, model weights in the named volume fuelish-ollama-data), pulling the model if not already present. Deliberately separate from the tooling container: Ollama needs to stay resident in memory with the model loaded, while the pipeline steps are short-lived and stateless (one docker run per receipt). 3_extract_data.sh reaches it via --network host at http://127.0.0.1:11434 by default (override with OLLAMA_URL/ OLLAMA_MODEL).
• build-image.sh / upload-image.sh - build/push the tooling-container/ image, tagged from tooling-container/VERSION. Image name defaults to fuelish/tools, overridable via registry.env (copy from registry.env.example; gitignored).
• test_batch.sh -in <test-data-dir> - runs the full pipeline over every *.jpg in a directory and compares results against that directory's ground-truth spreadsheet (*.ods). For an input dir <path>/<name> it creates sibling <path>/<name>-scratch/ (intermediate step output per receipt) and <path>/<name>-result/ (final extracted.json per receipt, plus summary.txt). Both are gitignored.

Current state

Last full run against the 17-receipt ground-truth test set (testdata/2026_new/, not checked in - see .gitignore):

• 15/17 receipts date-matched to ground truth (up from 11/17 with the original plain-grayscale preprocessing)
• price_per_litre: 11/15 correct, litres: 9/15, discount: 11/15, total_price: 12/15

Known remaining failure modes, not expected to be fixable by further preprocessing tuning:

• OCR garbling on the compact "litres EUR/l price" line on some receipts (the raw text sometimes lacks recoverable digits)
• Occasional year-digit misreads in the date causing a false date-match failure
• Low-resolution source photos (well below the ~4000x3000 the good results come from) OCR poorly regardless of preprocessing

The structuring prompt in tooling-container/src/extract.py uses deliberately non-colliding example values. An earlier version used real receipt values as worked examples, which caused the small model to copy/anchor on those exact numbers instead of reading the actual OCR text - a subtle failure mode worth remembering if the prompt is revised.

Not yet built

Per the original idea (see git history / project notes): an Android app to capture and confirm receipts, a backend server to store raw + confirmed submissions, a simple authenticated web UI to query/filter/export (CSV/JSON), and cross-checking total_price against price_per_litre * litres - discount. None of that exists yet - this repo currently covers only the extraction pipeline.