Add fuelish API; protect the host against memory exhaustion

- fuelish role: receipt OCR API at fuelish.ladkau.de, API-key auth, shared Ollama model and request options - base: 4 GB swap file; act_runner: job containers capped at 4 GB without swap - docs and vault/service checks updated
2026-10-04 13:54:09 +02:00
parent 974486bc53
commit 82d944a0a8
15 changed files with 246 additions and 25 deletions
@@ -43,6 +43,10 @@ Local dependencies: `pip install ansible passlib` and `openssl`.
 1. `scripts/run-bootstrap.sh` — copies and runs `bootstrap-deploy-user.sh` on the server as root; creates the `deploy` user with the SSH public key from `keys/notroot_cloud_ladkau_de.pub`
 2. `ansible-playbook ansible/site.yml` — runs all roles in order against the `deploy` user (passwordless sudo)

+### Host memory
+
+The VPS has 8 GB RAM. The `base` role creates a swap file (`/swapfile`, `base_swap_size`, `vm.swappiness` = `base_swappiness`) so memory pressure slows the host instead of freezing it. The largest consumers are the Ollama model (about 4 GB while loaded; `ollama_keep_alive` controls how long it stays) and CI job containers (capped by `act_runner_job_memory`).
+
 ### Role → service → data directory

 Each role deploys one Docker Compose stack. The pattern is:
@@ -98,17 +102,27 @@ n8n (`agent.ladkau.de`) runs the chat page and the scheduled workflows. The `n8n
 - Replies are converted from Markdown to Telegram HTML by Code nodes and sent with `parse_mode` HTML
 - Chat memory (Window Buffer Memory) lives in n8n's process memory, per workflow, and is lost on every n8n restart (including every deploy that changes workflows)

-### SearXNG
+#### n8n backend: SearXNG

 The `searxng` role runs a metasearch engine (`searxng/searxng`) on the internal network `searxng_internal` (`http://searxng:8080`), used only by the n8n `web_search` tool. `settings.yml` is rendered from a template with `searxng_secret_key` from the vault; the JSON output format is enabled and the bot limiter is off because there is no public route. Some upstream engines (DuckDuckGo, Brave) answer with captchas or rate limits; SearXNG skips them and uses the others.

-### Browserless
+#### n8n backend: Browserless

 The `browserless` role runs headless Chromium (`ghcr.io/browserless/chromium`) on the internal network `browserless_internal` (`http://browserless:3000`), one session at a time with a memory limit (`browserless_mem_limit`).

+### Fuelish (receipt OCR API)
+
+The `fuelish` role runs the job API of the fuelish project (`/home/ml/krotik/fuelish`, image `cr.ladkau.de/fuelish/tools`) at `fuelish.ladkau.de`. Key details:
+
+- The image is built and pushed from the fuelish project with `./build-image.sh` and `./upload-image.sh`; `fuelish_version` in `vars.yml` selects the tag. The role logs in to the registry with `fuelish_registry_user` / `fuelish_registry_password` (default: the first `registry_users` entry) and pulls on every run
+- The version must be built from fuelish commit `7a91a37` or later: older images ignore `FUELISH_API_KEY` and accept every request
+- `fuelish_api_key` (vault) is passed as `FUELISH_API_KEY`; the Android app sends it as the `X-API-Key` header. `/health` stays open
+- The API uses the shared `ollama` container (`ollama_internal`) with `fuelish_model` (default `ollama_model`, the model n8n uses, so Ollama keeps a single model loaded; a different value is pulled by the API on first start). The role passes `OLLAMA_NUM_CTX` (`fuelish_num_ctx`) and `OLLAMA_NUM_THREAD` (`fuelish_num_thread`), equal to the n8n Ollama nodes (16384 / 4), because Ollama restarts its runner (about 7 s) when a request's context size or thread count differs from the loaded one. Images built before fuelish `tooling-container/VERSION` 0.1.1 ignore both variables
+- Traefik routes `fuelish.ladkau.de` with TLS and the `rate-limit-lax` middleware; there is no basic auth because the app authenticates with the API key
+
 ### Ollama (local LLM)

-The `ollama` role runs `ollama/ollama` (CPU only) on the internal Docker network `ollama_internal` — no Traefik route, no public port. n8n joins that network (as `external`) and reaches it at `http://ollama:11434`.
+The `ollama` role runs `ollama/ollama` (CPU only) on the internal Docker network `ollama_internal` — no Traefik route, no public port. n8n and fuelish join that network (as `external`) and reach it at `http://ollama:11434`.

 - Model, context length and memory limit come from `ollama_*` in `vars.yml` (default `qwen3:4b-instruct-2507-q4_K_M`, 16k context, 4.5 GB limit — the VPS has 8 GB shared with ~24 containers)
 - Models persist in `/opt/ollama/models`; the role pulls `ollama_model` if it is missing
@@ -124,6 +138,7 @@ The `act_runner` role deploys three `gitea/act_runner` containers (Docker execut
 - Each runner gets its own data dir (`/opt/act_runner/runner-N/`) for its `.runner` registration file
 - All runners share a single `config.yml` at `/opt/act_runner/config.yml`
 - `/var/run/docker.sock` is mounted — job containers are spawned directly on the host
+- Job containers are limited to `act_runner_job_memory` of RAM with swap disabled (`container.options` in `config.yml`): a build that needs more is killed and the host stays up. Android builds use a Gradle heap of 2 GB (`org.gradle.jvmargs`) and need about 3–4 GB in total
 - `gitea_runner_registration_token` is obtained from Gitea admin UI **after** Gitea is running; add it to the vault and then deploy with `--tags act_runner`

 ### SSO
@@ -27,6 +27,7 @@ DNS Names:
 	vault.ladkau.de
 	dl.ladkau.de
 	agent.ladkau.de
+	fuelish.ladkau.de

 Purpose:
 --
@@ -75,6 +76,7 @@ Repo layout:
 			browserless/     # headless Chromium (internal network only)
 			searxng/         # metasearch for the chat web_search tool (internal network only)
 			n8n/             # AI chat and scheduled digests (agent.ladkau.de)
+			fuelish/         # receipt OCR job API for the Android app (fuelish.ladkau.de)
 	docs/
 		runbook-provisioning.md  # step-by-step reinstall instructions
 		runbook-configuration.md # first-run service configuration
@@ -93,6 +93,13 @@ roundcube_smtp_port: "587"
 act_runner_version: "latest"
 act_runner_data_dir: /opt/act_runner
 act_runner_count: 3
+# Swap file created by the base role (the VPS has no swap partition)
+base_swap_size: "4G"
+base_swappiness: 10   # use swap only under real memory pressure
+
+# Memory cap per CI job container (swap disabled inside the job): a build that needs more is killed
+# instead of exhausting the host
+act_runner_job_memory: "4g"
 # Secrets — store values in ansible/group_vars/all/vault.yml (Ansible Vault)
 # gitea_runner_registration_token: ""  # Gitea admin → Site Administration → Runners → Create runner token

@@ -133,17 +140,23 @@ ollama_context_length: 16384
 ollama_mem_limit: 4608m
 ollama_keep_alive: "1h"   # unload the model after this idle time to free RAM

-# Browserless — headless Chromium for rendering JavaScript-heavy pages (internal only)
-browserless_version: "v2.57.0"
-browserless_data_dir: /opt/browserless
-browserless_mem_limit: 1g
-
-# SearXNG — metasearch backend for the n8n web_search tool (internal only)
-searxng_version: "2026.10.2-19ffbcd30"
-searxng_data_dir: /opt/searxng
-searxng_mem_limit: 384m
+# Fuelish — receipt OCR job API for the Android app (fuelish.ladkau.de)
+# The image is pushed to the private registry with upload-image.sh in the fuelish project;
+# after pushing a new version, set fuelish_version and deploy with --tags fuelish.
+domain_fuelish: "fuelish.{{ domain_base }}"
+fuelish_image: "{{ domain_registry }}/fuelish/tools"
+fuelish_version: "0.1.1"   # needs X-API-Key support (fuelish commit 7a91a37 or later) and OLLAMA_NUM_CTX / OLLAMA_NUM_THREAD (version 0.1.1 or later)
+fuelish_data_dir: /opt/fuelish
+fuelish_model: "{{ ollama_model }}"   # same model as n8n, so the shared Ollama keeps a single model loaded
+fuelish_mem_limit: 768m
+# Request options of the API; they equal the numCtx / numThread of the n8n Ollama nodes (16384 / 4)
+fuelish_num_ctx: "{{ ollama_context_length }}"
+fuelish_num_thread: 4
+# Registry account used to pull the image: the first registry_users entry unless overridden in the vault
+fuelish_registry_user: "{{ registry_users[0].username }}"
+fuelish_registry_password: "{{ registry_users[0].password }}"
 # Secrets — store values in ansible/group_vars/all/vault.yml (Ansible Vault)
-# searxng_secret_key: ""   # generate: openssl rand -hex 32
+# fuelish_api_key: ""   # generate: openssl rand -hex 32 — the same value goes into the Android app settings

 # n8n — workflow automation, AI chat and scheduled digests (agent.ladkau.de)
 domain_n8n: "agent.{{ domain_base }}"
@@ -170,6 +183,7 @@ smtp_port: 465
 smtp_secure: true   # true = implicit TLS (port 465); false = STARTTLS (port 587)
 # Secrets — store values in ansible/group_vars/all/vault.yml (Ansible Vault)
 # n8n_encryption_key: ""   # generate: openssl rand -hex 32
+# searxng_secret_key: ""   # generate: openssl rand -hex 32 (SearXNG, the internal search backend)
 # gemini_api_key: ""      # Google AI Studio API key (default model of the Telegram chat)
 # telegram_bot_token: ""   # from @BotFather
 # telegram_user_id: ""     # your numeric Telegram user id (the only user the bot answers)
@@ -178,3 +192,13 @@ smtp_secure: true   # true = implicit TLS (port 465); false = STARTTLS (port 587
 # n8n_users:
 #   - username: admin
 #     password: "your-password"
+
+# n8n backend: Browserless — headless Chromium for the fetch_page tool (internal only)
+browserless_version: "v2.57.0"
+browserless_data_dir: /opt/browserless
+browserless_mem_limit: 1g
+
+# n8n backend: SearXNG — metasearch for the web_search tool (internal only)
+searxng_version: "2026.10.2-19ffbcd30"
+searxng_data_dir: /opt/searxng
+searxng_mem_limit: 384m
@@ -15,6 +15,7 @@ cache:
 container:
  network: bridge      # job containers get default bridge network with internet access
  privileged: false
+  options: "--memory={{ act_runner_job_memory }} --memory-swap={{ act_runner_job_memory }}"
  valid_volumes:
    - /tmp
  force_pull: true     # otherwise a job container image update (e.g. a build-image
@@ -152,3 +152,53 @@
    state: started
    enabled: true
  tags: base
+
+# Swap file: without swap, memory pressure ends in a frozen host instead of a slow one
+- name: Create swap file
+  ansible.builtin.command: fallocate -l {{ base_swap_size }} /swapfile
+  args:
+    creates: /swapfile
+  tags: base
+
+- name: Set swap file permissions
+  ansible.builtin.file:
+    path: /swapfile
+    owner: root
+    group: root
+    mode: "0600"
+  tags: base
+
+- name: Format swap file
+  ansible.builtin.shell: blkid -p /swapfile | grep -q 'TYPE="swap"' || mkswap /swapfile
+  register: base_mkswap
+  changed_when: "'Setting up swapspace' in base_mkswap.stdout"
+  tags: base
+
+- name: List active swap
+  ansible.builtin.command: swapon --noheadings --show=NAME
+  register: base_active_swap
+  changed_when: false
+  tags: base
+
+- name: Enable swap file
+  ansible.builtin.command: swapon /swapfile
+  when: "'/swapfile' not in base_active_swap.stdout"
+  tags: base
+
+- name: Mount swap file at boot
+  ansible.posix.mount:
+    src: /swapfile
+    path: none
+    fstype: swap
+    opts: sw
+    state: present
+  tags: base
+
+- name: Set swappiness
+  ansible.posix.sysctl:
+    name: vm.swappiness
+    value: "{{ base_swappiness }}"
+    sysctl_file: /etc/sysctl.d/60-swap.conf
+    state: present
+    reload: true
+  tags: base
@@ -25,6 +25,7 @@ SERVICES = [
    ("k8s",          "https://{{ domain_k8s }}",                200, "https://{{ domain_k8s }}"),
    ("Vaultwarden",  "https://{{ domain_vault }}",             200, "https://{{ domain_vault }}"),
    ("Downloads",    "https://{{ domain_dl }}",               200, "https://{{ domain_dl }}"),
+    ("Fuelish",     "https://{{ domain_fuelish }}/health",   200, "https://{{ domain_fuelish }}/health"),
    ("AI Chat",      "https://{{ domain_n8n }}/chat",         401, "https://{{ domain_n8n }}/chat"),
    ("n8n",          "https://{{ domain_n8n }}",              401, "https://{{ domain_n8n }}"),
 ]
@@ -0,0 +1,6 @@
+---
+- name: Restart fuelish
+  community.docker.docker_compose_v2:
+    project_src: "{{ fuelish_data_dir }}"
+    state: present
+    recreate: always
@@ -0,0 +1,35 @@
+---
+- name: Create fuelish directory
+  ansible.builtin.file:
+    path: "{{ fuelish_data_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  tags: fuelish
+
+# The tools image is pushed to the private registry with upload-image.sh
+- name: Log in to the container registry
+  community.docker.docker_login:
+    registry_url: "{{ domain_registry }}"
+    username: "{{ fuelish_registry_user }}"
+    password: "{{ fuelish_registry_password }}"
+  no_log: true
+  tags: fuelish
+
+- name: Deploy Docker Compose file
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ fuelish_data_dir }}/docker-compose.yml"
+    owner: root
+    group: root
+    mode: "0600"
+  notify: Restart fuelish
+  tags: fuelish
+
+- name: Start fuelish
+  community.docker.docker_compose_v2:
+    project_src: "{{ fuelish_data_dir }}"
+    state: present
+    pull: always
+  tags: fuelish
@@ -0,0 +1,33 @@
+# Managed by Ansible — do not edit manually
+services:
+  fuelish:
+    image: {{ fuelish_image }}:{{ fuelish_version }}
+    container_name: fuelish
+    restart: unless-stopped
+    command: ["uvicorn", "server:app", "--app-dir", "/opt/app/src", "--host", "0.0.0.0", "--port", "8000"]
+    environment:
+      OLLAMA_URL: "http://ollama:11434/api/chat"
+      OLLAMA_BASE_URL: "http://ollama:11434"
+      OLLAMA_MODEL: "{{ fuelish_model }}"
+      # Same request options as the n8n Ollama nodes, so the shared Ollama keeps its runner loaded
+      OLLAMA_NUM_CTX: "{{ fuelish_num_ctx }}"
+      OLLAMA_NUM_THREAD: "{{ fuelish_num_thread }}"
+      FUELISH_API_KEY: "{{ fuelish_api_key }}"
+    mem_limit: {{ fuelish_mem_limit }}
+    networks:
+      - traefik_public
+      - ollama_internal
+    labels:
+      - "traefik.enable=true"
+      - "traefik.http.routers.fuelish.rule=Host(`{{ domain_fuelish }}`)"
+      - "traefik.http.routers.fuelish.entrypoints=websecure"
+      - "traefik.http.routers.fuelish.tls.certresolver=letsencrypt"
+      - "traefik.http.services.fuelish.loadbalancer.server.port=8000"
+      # The app polls GET /jobs/{id} while a receipt is processed
+      - "traefik.http.routers.fuelish.middlewares=rate-limit-lax@docker"
+
+networks:
+  traefik_public:
+    external: true
+  ollama_internal:
+    external: true
@@ -16,6 +16,7 @@
    - vaultwarden
    - dl
    - ollama
+    - fuelish
    - browserless
    - searxng
    - n8n
@@ -550,9 +550,9 @@ A summary of a long page takes several minutes on the CPU-only model.

 ### 9.5 Ollama — local LLM server

-The `ollama` role runs Ollama (CPU only). It is reachable only on the internal
-Docker network `ollama_internal` at `http://ollama:11434`; there is no public
-route.
+The `ollama` role runs Ollama (CPU only). It is shared by n8n and fuelish and
+reachable only on the internal Docker network `ollama_internal` at
+`http://ollama:11434`; there is no public route.

 Settings in `ansible/group_vars/all/vars.yml`:

@@ -736,3 +736,48 @@ curl -s "https://api.telegram.org/bot<TOKEN>/getWebhookInfo"
 ```

 `getWebhookInfo` shows the registered URL and the last delivery error.
+
+## 10. Fuelish — receipt OCR API
+
+`https://fuelish.ladkau.de` serves the job API of the fuelish Android app
+(`POST /jobs`, `GET /jobs/{id}`, `GET /health`). Every `/jobs` request needs
+the header `X-API-Key: <fuelish_api_key>`.
+
+### 10.1 Publish an image version
+
+In the fuelish project (`registry.env` with the registry credentials):
+
+```bash
+echo 0.1.1 > tooling-container/VERSION    # new version number
+./build-image.sh && ./upload-image.sh      # pushes cr.ladkau.de/fuelish/tools:<version> and :latest
+```
+
+Set `fuelish_version` in `ansible/group_vars/all/vars.yml` to the same number
+and deploy:
+
+```bash
+ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags fuelish --ask-vault-pass
+```
+
+The image must be built from fuelish commit `7a91a37` or later; older images do
+not check the API key.
+
+### 10.2 Check the deployment
+
+```bash
+curl -s https://fuelish.ladkau.de/health                     # {"status":"ok"}
+curl -s -o /dev/null -w "%{http_code}\n" -X POST https://fuelish.ladkau.de/jobs   # without key: 401 or 403
+ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 "sudo docker exec ollama ollama list"   # fuelish_model is present (the API pulls it if missing)
+```
+
+### 10.3 Android app
+
+In the app's Settings tab enter `https://fuelish.ladkau.de` as server URL and
+`fuelish_api_key` as API key.
+
+`fuelish_model` defaults to `ollama_model`, the model used by n8n, so both share
+one loaded model. The API sends the request options from `fuelish_num_ctx` and
+`fuelish_num_thread` (16384 and 4, the same as the n8n Ollama nodes), so
+alternating requests from both apps do not restart the Ollama runner. Keep these
+values equal to `ollama_context_length` and the `numThread` of the n8n Ollama
+nodes. Images before fuelish version 0.1.1 ignore the two settings.
@@ -82,6 +82,7 @@ Let's Encrypt certificates on first start and DNS must resolve at that point.
 | vault.ladkau.de     | A → server IP |
 | dl.ladkau.de        | A → server IP |
 | agent.ladkau.de     | A → server IP |
+| fuelish.ladkau.de   | A → server IP |

 ### 4. Create the vault and populate secrets

@@ -122,15 +123,18 @@ registry_users:
  - username: alice
    password: "your-password"

-# n8n — AI chat and scheduled digests at agent.ladkau.de
+# Fuelish — receipt OCR API at fuelish.ladkau.de
+fuelish_api_key: ""                 # generate: openssl rand -hex 32 — same value in the Android app settings
+
+# n8n — AI chat, Telegram bot and scheduled digests at agent.ladkau.de
 n8n_users:                          # basic auth in front of n8n
  - username: admin
    password: "your-password"
 n8n_encryption_key: ""              # generate: openssl rand -hex 32 — never change after first deploy
-searxng_secret_key: ""              # generate: openssl rand -hex 32
+searxng_secret_key: ""              # generate: openssl rand -hex 32 — SearXNG, the internal search backend of the n8n chat
 smtp_user: ""                       # full Strato mailbox address
 smtp_password: ""
-gemini_api_key: ""                 # Google AI Studio API key, see runbook-configuration.md section 9.8
+gemini_api_key: ""                  # Google AI Studio API key, see runbook-configuration.md section 9.8
 telegram_bot_token: ""              # from @BotFather, see runbook-configuration.md section 9.8
 telegram_user_id: ""                # your numeric Telegram user id

@@ -222,11 +226,11 @@ This runs all roles in order:

 | #  | Role          | What it does |
 |----|---------------|--------------|
-| 1  | `base`        | OS hardening, deploy user, SSH config, ufw firewall, fail2ban |
+| 1  | `base`        | OS hardening, deploy user, SSH config, ufw firewall, fail2ban, swap file |
 | 2  | `docker`      | Docker Engine + Compose plugin, shared Traefik network |
 | 3  | `traefik`     | Reverse proxy, automatic TLS via Let's Encrypt |
 | 4  | `gitea`       | Self-hosted Git with PostgreSQL |
-| 5  | `act_runner`  | Three Gitea Actions runners with Docker executor |
+| 5  | `act_runner`  | Three Gitea Actions runners with Docker executor; job containers capped at `act_runner_job_memory` |
 | 6  | `nextcloud`   | File storage with PostgreSQL, Redis, cron sidecar |
 | 7  | `sso`         | Keycloak single sign-on with PostgreSQL |
 | 8  | `mail`        | Roundcube webmail client with PostgreSQL |
@@ -235,10 +239,11 @@ This runs all roles in order:
 | 11 | `vaultwarden` | Vaultwarden password vault at vault.ladkau.de |
 | 12 | `dl`          | Public download server at dl.ladkau.de — nginx HTTPS + SFTP upload |
 | 13 | `ollama`      | Local LLM server (Ollama, CPU) on the internal `ollama_internal` network — pulls `ollama_model` on first run |
-| 14 | `browserless` | Headless Chromium for rendering JavaScript-heavy pages — internal network only |
-| 15 | `searxng`     | Metasearch engine for the chat web_search tool — internal network only |
-| 16 | `n8n`         | n8n at agent.ladkau.de — AI chat (`/chat`), scheduled digests, basic auth; imports the workflows in the repo |
-| 17 | `dashboard`   | Public status dashboard at cloud.ladkau.de — service health and server stats |
+| 14 | `fuelish`     | Receipt OCR job API for the Android app at fuelish.ladkau.de — image from the private registry, API-key auth |
+| 15 | `browserless` | Headless Chromium for rendering JavaScript-heavy pages — internal network only |
+| 16 | `searxng`     | Metasearch engine for the chat web_search tool — internal network only |
+| 17 | `n8n`         | n8n at agent.ladkau.de — AI chat (`/chat`), scheduled digests, basic auth; imports the workflows in the repo |
+| 18 | `dashboard`   | Public status dashboard at cloud.ladkau.de — service health and server stats |

 > **Note:** The `act_runner` role requires `gitea_runner_registration_token` in the
 > vault, which can only be obtained after Gitea is running and an admin account has
@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7fwictXrRTJElAOHpWj99qPbik1iAvFtR/ejNdG90z gitea-actions
@@ -18,6 +18,7 @@ CHECKS=(
  "https://k8s.ladkau.de 200 10"
  "https://vault.ladkau.de 200 10"
  "https://dl.ladkau.de 200 10"
+  "https://fuelish.ladkau.de/health 200 10"
  "https://agent.ladkau.de 401 10"
  "https://agent.ladkau.de/chat 401 10"
 )
@@ -45,6 +45,7 @@ required_scalars = [
    "roundcube_des_key",
    "vaultwarden_admin_token",
    "n8n_encryption_key",
+    "fuelish_api_key",
    "searxng_secret_key",
    "smtp_user",
    "smtp_password",
				`@@ -0,0 +1 @@`
				`ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7fwictXrRTJElAOHpWj99qPbik1iAvFtR/ejNdG90z gitea-actions`