server_cloud_ladkau_de/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Purpose

Infrastructure-as-code for a Strato VPS at cloud.ladkau.de. Every config change must be committed — the git history is the change log. The goal is a fully reproducible reinstall from this repo alone.

## Key commands

```bash
# Install Ansible collections (one-time, after cloning)
ansible-galaxy collection install -r ansible/requirements.yml

# Run the full playbook (most common operation)
ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass

# Run a single role
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags mail --ask-vault-pass

# Bootstrap a fresh server (before Ansible can run — creates the deploy user)
bash scripts/run-bootstrap.sh

# Verify all vault secrets are present and valid
bash scripts/check-vault.sh

# Verify all services are reachable
bash scripts/check-services.sh

# Edit encrypted secrets
ansible-vault edit ansible/group_vars/all/vault.yml

# Connect to server
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148
```

Local dependencies: `pip install ansible passlib` and `openssl`.

## Architecture

### Provisioning flow

1. `scripts/run-bootstrap.sh` — copies and runs `bootstrap-deploy-user.sh` on the server as root; creates the `deploy` user with the SSH public key from `keys/notroot_cloud_ladkau_de.pub`
2. `ansible-playbook ansible/site.yml` — runs all roles in order against the `deploy` user (passwordless sudo)

### Host memory

The VPS has 8 GB RAM. The `base` role creates a swap file (`/swapfile`, `base_swap_size`, `vm.swappiness` = `base_swappiness`) so memory pressure slows the host instead of freezing it. The largest consumers are the Ollama model (about 4 GB while loaded; `ollama_keep_alive` controls how long it stays) and CI job containers (capped by `act_runner_job_memory`).

### Role → service → data directory

Each role deploys one Docker Compose stack. The pattern is:

- **Template** lives in `roles/<name>/templates/docker-compose.yml.j2`
- Ansible renders it and writes it to `/opt/<name>/docker-compose.yml` on the server
- Data/config for each service lives under `/opt/<name>/` on the server
- The role also writes any config files (e.g. `dovecot.conf`, `passwd`, `fetchmailrc`) to `/opt/<name>/` before starting the stack

### Networking

Two Docker networks:
- `traefik_public` (external, created once by the `traefik` role) — all services that need HTTPS exposure join this network alongside Traefik
- Per-service internal networks (e.g. `mail_internal`) — isolate backend containers (databases, Dovecot) from the internet

Traefik routes HTTP/HTTPS/IMAPS via Docker labels on each service container. TLS certificates are issued via Let's Encrypt ACME. IMAPS (993) is handled as a TCP passthrough — Traefik terminates TLS and forwards plain IMAP to Dovecot port 143.

### Secrets

All secrets live in `ansible/group_vars/all/vault.yml` (Ansible Vault, AES-256). Plain variables are in `ansible/group_vars/all/vars.yml` — commented-out keys there show what each role expects from the vault. `scripts/check-vault.sh` validates all required keys are present.

Passwords that need bcrypt hashes (Traefik dashboard, container registry) are stored as plaintext in the vault and hashed in the Jinja2 templates at deploy time using `password_hash('bcrypt', ...)`. The `passlib` Python package is required locally for this.

### Mail stack specifics

The mail role deploys four containers on `mail_internal`: `mail-db` (PostgreSQL for Roundcube), `dovecot` (IMAP server), `roundcube` (webmail at mail.ladkau.de), and conditionally `fetchmail` (polls external POP3 accounts).

- Dovecot uses passwd-file auth: `roles/mail/templates/passwd.j2` generates `/opt/mail/passwd` with 8-field format (`user:hash::::::`) — the trailing `::::::` is required for userdb lookup
- Fetchmail delivers via LMTP to Dovecot port 24; the `auth_username_format = %Ln` in Dovecot's LMTP config strips the `@dovecot` domain that fetchmail appends
- Fetchmail 6.6.x rcfile syntax: `ssl` is a user-level option (inline after `password`), port is `smtphost dovecot/24` (slash-separated), `smtpport` keyword does not exist
- `mail.ladkau.de` is the Roundcube webmail client, not an MTA — no outgoing SMTP server is configured

### n8n (AI chat and scheduled digests)

n8n (`agent.ladkau.de`) runs the chat page and the scheduled workflows. The `n8n` role templates its compose file to `/opt/n8n` and imports the repo's workflows into it. Key details:

- Traefik basic auth (`n8n-auth` middleware) uses `n8n_users` from the vault; n8n has its own owner login behind it
- `/chat` is redirected by a Traefik `redirectregex` middleware to the hosted chat webhook `/webhook/<n8n_chat_webhook_id>/chat`; `n8n_chat_webhook_id` must match `webhookId` in `roles/n8n/files/workflows/chat.json`
- Workflows are `roles/n8n/files/workflows/*.json`; credentials are imported from `roles/n8n/files/credentials/ollama.json` and a template rendered from the vault (`smtp_user`, `smtp_password`). Import (credentials, workflows, publish `"active": true` ones, restart) runs as a handler only when those files change
- Imports replace workflows by id: edits made in the n8n UI are lost unless exported back into the repo
- Data and the SQLite database are in `/opt/n8n/data`; `n8n_encryption_key` protects stored credentials and must never change after the first deploy
- n8n joins `ollama_internal` and `browserless_internal`; neither backend has a public route
- The `fetch_page` tool and the digest workflow call browserless `POST /function`, which runs a small Puppeteer script and returns the page text
- The `web_search` tool (`web-search.json`) queries SearXNG's JSON API and returns the top 6 results
- n8n also joins `searxng_internal`
- Telegram: the `Telegram chat` workflow answers one Telegram user (`telegram_user_id`); other users and non-text messages are ignored
- Model routing (`Route` step): messages go to the Gemini chain (`gemini_api_key` from the vault); messages starting with `/qwen` go to the local Qwen agent. The Gemini chain is `gemini-3.8-flash`, then `gemini-3.5-flash` if the answer is empty, then Qwen. Free-tier Gemini limits are per model; the reply ends with a note when a fallback answered
- Model names in the workflow JSON are plain strings. `__OLLAMA_MODEL__` is replaced by `ollama_model` and `__TELEGRAM_USER_ID__` by `telegram_user_id` when Ansible copies the files; keep the placeholders when exporting workflows from the UI
- All agents have the `web_search` and `fetch_page` tools, 4 iterations at most, and the instruction to make one `fetch_page` call per web address or one `web_search` per question. The Gemini agents share one conversation memory; the Qwen agent has its own small memory
- Scheduled workflows (`digest-telegram.json`) post through the `Post message` entry point of `Telegram chat`, so the text lands in the chat and in its memory
- The `Telegram chat` workflow has a schedule (`Stock price schedule`, three cron expressions, weekdays) and a manual trigger. Both send `/stocks` through `Route`, which expands it to `STOCK_QUESTION` (names the two Yahoo quote pages); `/stocks` typed in Telegram does the same and `/qwen /stocks` sends it to Qwen
- Telegram's servers call n8n at `/webhook/<n8n_telegram_webhook_id>/webhook`. That route has no basic auth; Traefik only admits `telegram_webhook_source_ranges` and n8n checks Telegram's secret token header. `n8n_telegram_webhook_id` must match `webhookId` in `telegram-chat.json`
- Replies are converted from Markdown to Telegram HTML by Code nodes and sent with `parse_mode` HTML
- Chat memory (Window Buffer Memory) lives in n8n's process memory, per workflow, and is lost on every n8n restart (including every deploy that changes workflows)

#### n8n backend: SearXNG

The `searxng` role runs a metasearch engine (`searxng/searxng`) on the internal network `searxng_internal` (`http://searxng:8080`), used only by the n8n `web_search` tool. `settings.yml` is rendered from a template with `searxng_secret_key` from the vault; the JSON output format is enabled and the bot limiter is off because there is no public route. Some upstream engines (DuckDuckGo, Brave) answer with captchas or rate limits; SearXNG skips them and uses the others.

#### n8n backend: Browserless

The `browserless` role runs headless Chromium (`ghcr.io/browserless/chromium`) on the internal network `browserless_internal` (`http://browserless:3000`), one session at a time with a memory limit (`browserless_mem_limit`).

### Fuelish (receipt OCR API)

The `fuelish` role runs the job API of the fuelish project (`/home/ml/krotik/fuelish`, image `cr.ladkau.de/fuelish/tools`) at `fuelish.ladkau.de`. Key details:

- The image is built and pushed from the fuelish project with `./build-image.sh` and `./upload-image.sh`; `fuelish_version` in `vars.yml` selects the tag. The role logs in to the registry with `fuelish_registry_user` / `fuelish_registry_password` (default: the first `registry_users` entry) and pulls on every run
- The version must be built from fuelish commit `7a91a37` or later: older images ignore `FUELISH_API_KEY` and accept every request
- `fuelish_api_key` (vault) is passed as `FUELISH_API_KEY`; the Android app sends it as the `X-API-Key` header. `/health` stays open
- The API uses the shared `ollama` container (`ollama_internal`) with `fuelish_model` (default `ollama_model`, the model n8n uses, so Ollama keeps a single model loaded; a different value is pulled by the API on first start). The role passes `OLLAMA_NUM_CTX` (`fuelish_num_ctx`) and `OLLAMA_NUM_THREAD` (`fuelish_num_thread`), equal to the n8n Ollama nodes (16384 / 4), because Ollama restarts its runner (about 7 s) when a request's context size or thread count differs from the loaded one. Images built before fuelish `tooling-container/VERSION` 0.1.1 ignore both variables
- Traefik routes `fuelish.ladkau.de` with TLS and the `rate-limit-lax` middleware; there is no basic auth because the app authenticates with the API key

### Ollama (local LLM)

The `ollama` role runs `ollama/ollama` (CPU only) on the internal Docker network `ollama_internal` — no Traefik route, no public port. n8n and fuelish join that network (as `external`) and reach it at `http://ollama:11434`.

- Model, context length and memory limit come from `ollama_*` in `vars.yml` (default `qwen3:4b-instruct-2507-q4_K_M`, 16k context, 4.5 GB limit — the VPS has 8 GB shared with ~24 containers)
- Models persist in `/opt/ollama/models`; the role pulls `ollama_model` if it is missing
- `ollama_keep_alive` unloads the model from RAM when idle
- Ollama starts 2 threads on this 2-core/4-thread host; the Ollama chat model nodes in the workflows set `numThread` to 4
- Changing `ollama_model` and re-running `--tags ollama` pulls the new model; old models are removed manually with `docker exec ollama ollama rm <model>`

### Gitea Actions runners

The `act_runner` role deploys three `gitea/act_runner` containers (Docker executor). Each runner handles one concurrent job. Key details:

- Runners connect to Gitea via the public HTTPS URL — no internal Docker network needed
- Each runner gets its own data dir (`/opt/act_runner/runner-N/`) for its `.runner` registration file
- All runners share a single `config.yml` at `/opt/act_runner/config.yml`
- `/var/run/docker.sock` is mounted — job containers are spawned directly on the host
- Job containers are limited to `act_runner_job_memory` of RAM with swap disabled (`container.options` in `config.yml`): a build that needs more is killed and the host stays up. Android builds use a Gradle heap of 2 GB (`org.gradle.jvmargs`) and need about 3–4 GB in total
- `gitea_runner_registration_token` is obtained from Gitea admin UI **after** Gitea is running; add it to the vault and then deploy with `--tags act_runner`

### SSO

Keycloak at `sso.ladkau.de` is the identity provider. Gitea and Nextcloud are configured post-provisioning to use it (see `docs/runbook-configuration.md`).

## Runbooks

- `docs/runbook-provisioning.md` — step-by-step reinstall from scratch
- `docs/runbook-configuration.md` — first-run service configuration (Keycloak realm, Gitea admin, Nextcloud OIDC, etc.)
- `docs/runbook-mail-import.md` — migrating mbox-format mail from an old Dovecot server to Maildir++