server_cloud_ladkau_de/docs/runbook-configuration.md

# Configuration Runbook

First-run configuration steps to perform after the Ansible playbook has
provisioned the server. See `runbook-provisioning.md` for the provisioning steps.

## 1. Verify all services are reachable

Before configuring individual services, confirm every HTTPS endpoint is up and
TLS certificates are valid. Run this from your local machine:

```bash
bash scripts/check-services.sh
```

The script checks these endpoints and verifies the expected HTTP status code:

| URL | Expected | Notes |
|-----|----------|-------|
| `https://cloud.ladkau.de/dashboard/` | 401 | Basic-auth prompt — correct without credentials |
| `https://gitea.ladkau.de` | 200 | Gitea sign-in page |
| `https://nextcloud.ladkau.de` | 200 | First visit triggers setup and takes 1–2 min |
| `https://sso.ladkau.de/realms/master` | 200 | Keycloak realm JSON — first start takes ~90 s |
| `https://mail.ladkau.de` | 200 | Roundcube webmail login |
| `https://cr.ladkau.de/v2/` | 401 | Registry API — auth required, correct without credentials |
| `https://k8s.ladkau.de` | 200 | Placeholder page |
| `https://vault.ladkau.de` | 200 | Vaultwarden web vault |
| `https://dl.ladkau.de` | 200 | Public download server |
| `https://agent.ladkau.de` | 401 | n8n — basic-auth prompt, correct without credentials |
| `https://agent.ladkau.de/chat` | 401 | AI chat — basic-auth prompt, correct without credentials |

A `000` result means the connection was refused or timed out — a container that
did not start. Keycloak and Nextcloud may return `502` for up to 90 seconds on
first boot; wait and retry before investigating.

Once the script passes, open `https://cloud.ladkau.de/dashboard/` in a browser
and authenticate with the `traefik_dashboard_users` credentials to verify the
dashboard loads correctly.

## 2. Keycloak

Keycloak is the identity provider for all services. Configure it first so that
Gitea and Nextcloud can be wired up to SSO in the steps that follow.

Keycloak takes ~90 seconds to start on first boot. If the login page at
`https://sso.ladkau.de` is not immediately available, wait and retry.

### 2.1 Log in and change the admin password

1. Go to `https://sso.ladkau.de` and sign in with username `admin` and the
   value of `keycloak_admin_password` from the vault.
2. Click the **admin** dropdown (top right) → **Manage account** →
   **Password** → set a new password.

The bootstrap credentials are only used on first start. After changing the
password, the vault value is no longer authoritative — store the new password
in your password manager.

### 2.2 Create an application realm

1. In the left sidebar, click the realm dropdown (shows **master**) →
   **Create realm**.
2. Set **Realm name** to `ladkau` and click **Create**.
3. Make sure the `ladkau` realm is selected in the sidebar for all steps below.

The `master` realm is for Keycloak administration only. All application users
and clients live in `ladkau`.

### 2.3 Create users

For each user you want to be able to log in to Gitea and Nextcloud:

1. **Users** → **Add user**.
2. Fill in **Username**, **Email**, **First name**, **Last name** → **Create**.
3. Go to the **Credentials** tab → **Set password**.
4. Enter a password, turn off **Temporary**, and click **Save**.

### 2.4 Create the Gitea OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `gitea` → **Next**.
3. Turn **Client authentication** on (confidential client) → **Next**.
4. **Valid redirect URIs**: `https://gitea.ladkau.de/user/oauth2/keycloak/callback`
5. **Web origins**: `https://gitea.ladkau.de` → **Save**.
6. Go to the **Credentials** tab and copy the **Client secret** — you will
   need it in step 3.

### 2.5 Create the Nextcloud OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `nextcloud` → **Next**.
3. Turn **Client authentication** on → **Next**.
4. **Valid redirect URIs**: `https://nextcloud.ladkau.de/apps/user_oidc/code`
5. **Valid post logout redirect URIs**: `https://nextcloud.ladkau.de/*`
6. **Web origins**: `https://nextcloud.ladkau.de` → **Save**.
7. Go to the **Credentials** tab and copy the **Client secret** — you will
   need it in step 4.

### 2.6 Create the Vaultwarden OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `vaultwarden` → **Next**.
3. Turn **Client authentication** on → **Next**.
4. **Valid redirect URIs**: `https://vault.ladkau.de/*`
5. **Valid post logout redirect URIs**: `https://vault.ladkau.de/*`
6. **Web origins**: `https://vault.ladkau.de` → **Save**.
7. Go to the **Credentials** tab and copy the **Client secret**.
8. Add it to the vault as `vaultwarden_sso_client_secret` and redeploy:
   ```bash
   ansible-vault edit ansible/group_vars/all/vault.yml
   ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags vaultwarden --ask-vault-pass
   ```

## 3. Gitea

### 3.1 Create the admin account

1. Go to `https://gitea.ladkau.de` — Gitea shows an initial configuration
   dialog on the first visit.
2. Scroll to the bottom of the form to the **Administrator Account Settings**
   section and fill in the admin username, email, and password.
3. Click **Install Gitea**.

Gitea initialises the database and creates the admin account in one step.
Public registration is closed — only users authenticating via Keycloak can
get an account (controlled by `ALLOW_ONLY_EXTERNAL_REGISTRATION`).

### 3.2 Connect Gitea to Keycloak

1. In Gitea, go to **Site Administration** → **Identity & Access** →
   **Authentication Sources** → **Add Authentication Source**.
2. Set **Authentication type** to **OAuth2**.
3. Set **OAuth2 provider** to **OpenID Connect**.
4. Fill in:
   - **Name**: `keycloak`
   - **Client ID**: `gitea`
   - **Client secret**: the secret copied in step 2.4
   - **OpenID Connect Auto Discovery URL**:
     `https://sso.ladkau.de/realms/ladkau/.well-known/openid-configuration`
5. Click **Add Authentication Source**.

Users can now sign in via **Sign in with keycloak** on the Gitea login page.

### 3.3 Use SSH key authentication for Git

Gitea exposes SSH on port **2222** (host port mapped from the container's port 22).
Clone URLs use the full `ssh://` form because the port is non-standard.

**Generate an SSH key**

If you do not already have a key, generate a dedicated one for Gitea:

```bash
ssh-keygen -t ed25519 -C "gitea.ladkau.de" -f ~/.ssh/id_ed25519_gitea
```

Leave the passphrase empty for unattended use, or set one for additional
security. The command creates two files:

- `~/.ssh/id_ed25519_gitea` — private key (never share this)
- `~/.ssh/id_ed25519_gitea.pub` — public key (upload this to Gitea)

**Add your key to Gitea**

1. Go to `https://gitea.ladkau.de` → click your avatar (top right) →
   **Settings** → **SSH / GPG Keys** → **Add Key**.
2. Paste the contents of `~/.ssh/id_ed25519_gitea.pub`.
3. Click **Add Key**.

**Test the connection**

```bash
ssh -p 2222 -i ~/.ssh/id_ed25519_gitea -T git@gitea.ladkau.de
```

A successful response looks like:

```
Hi <username>! You've successfully authenticated, but Gitea does not provide shell access.
```

**Clone, push, and pull**

Gitea shows the correct SSH URL on every repository page. It takes the form:

```
ssh://git@gitea.ladkau.de:2222/<username>/<repo>.git
```

```bash
# Clone
git clone ssh://git@gitea.ladkau.de:2222/<username>/myrepo.git

# Add remote on an existing repo
git remote add origin ssh://git@gitea.ladkau.de:2222/<username>/myrepo.git
```

**Optional: SSH config alias**

Add this to `~/.ssh/config` to avoid repeating the port:

```
Host gitea.ladkau.de
    User git
    Port 2222
    IdentityFile ~/.ssh/id_ed25519_gitea
```

With the alias in place you can use the shorter SCP-like URL:

```bash
git clone git@gitea.ladkau.de:<username>/myrepo.git
```

### 3.4 Deploy Gitea Actions runners

Three `act_runner` containers (Docker executor) are provisioned by the
`act_runner` Ansible role. Each runner handles one concurrent job; together
they allow up to three parallel workflow jobs.

**This step requires Gitea to be running and the admin account to exist
(step 3.1). Runners cannot register until a token is obtained from Gitea.**

**Step 1 — Get a runner registration token**

1. Go to `https://gitea.ladkau.de` → sign in as admin.
2. Navigate to **Site Administration** (top-right menu) →
   **Actions** → **Runners**.
3. Click **Create runner token** and copy the token.

**Step 2 — Add the token to the vault**

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
```

Add the key:

```yaml
gitea_runner_registration_token: "<token-from-step-1>"
```

**Step 3 — Deploy the runners**

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml \
  --tags act_runner --ask-vault-pass
```

Ansible creates `/opt/act_runner/` with a shared `config.yml` and three
per-runner data directories (`runner-1/`, `runner-2/`, `runner-3/`). On first
start each container auto-registers with Gitea and writes a `.runner` file to
its data directory. Subsequent restarts reuse the saved registration.

**Verify**

Back in Gitea **Site Administration → Actions → Runners**, all three runners
should appear as **Online** within a few seconds.

> **Security note:** Each runner container mounts `/var/run/docker.sock`.
> This gives workflow jobs root-equivalent access to the host Docker daemon.
> Only run workflows from trusted repositories.

## 4. Nextcloud

The admin credentials are set via `nextcloud_admin_user` and
`nextcloud_admin_password` in the vault. The first HTTP request triggers
installation, which takes a minute or two.

### 4.1 Log in

1. Go to `https://nextcloud.ladkau.de` and sign in with the admin credentials
   from the vault.

### 4.2 Connect Nextcloud to Keycloak

1. Go to **Apps** → search for **OpenID Connect user backend** → **Download
   and enable**.
2. Go to **Administration settings** → **OpenID Connect** → **Add provider**.
3. Fill in:
   - **Identifier**: `keycloak`
   - **Client ID**: `nextcloud`
   - **Client secret**: the secret copied in step 2.5
   - **Discovery endpoint**:
     `https://sso.ladkau.de/realms/ladkau/.well-known/openid-configuration`
4. Click **Save**.

Users can now sign in to Nextcloud with their Keycloak credentials via the
**Log in with keycloak** button.

## 5. Mail (Dovecot + Roundcube + Fetchmail)

The mail stack runs a local Dovecot IMAP server. Roundcube connects to it
internally. External IMAP clients connect to `mail.ladkau.de:993` (IMAPS —
TLS terminated by Traefik). Fetchmail polls external POP3 accounts and
delivers to Dovecot.

Dovecot users and fetchmail accounts are configured in the vault before
provisioning — see step 4 of `runbook-provisioning.md`. To add users or
accounts after initial deployment, edit the vault and redeploy:

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags mail --ask-vault-pass
```

### 5.1 Import mail from an old Dovecot server

If the old server uses **Maildir** format, rsync directly:

```bash
# Run on the old server — syncs to the new server's maildir volume
rsync -av -e "ssh -i keys/root_cloud_ladkau_de" --progress ./* \
  root@217.154.207.148:/opt/mail/maildir/alice/Maildir/

# Fix ownership for Dovecot's vmail user (UID 5000)
ssh -i keys/root_cloud_ladkau_de root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice"
```

If the old server uses **mbox** format (e.g. `mail_location = mbox:~/mail:...`),
see `runbook-mail-import.md` for the conversion procedure.

Alternatively, use `imapsync` to copy via IMAP from the old server to the
new one (works without direct server access):

```bash
imapsync \
  --host1 old.server.com --user1 alice --password1 oldpass \
  --host2 mail.ladkau.de  --user2 alice --password2 newpass \
  --port2 993 --ssl2
```

### 5.2 Connect an IMAP client

| Setting   | Value                |
|-----------|----------------------|
| Server    | `mail.ladkau.de`     |
| Port      | `993`                |
| Security  | SSL/TLS              |
| Username  | as set in vault      |

## 6. Vaultwarden

Vaultwarden is a self-hosted Bitwarden-compatible password vault. Any official
Bitwarden client (browser extension, desktop, mobile) can connect to it.

### 6.1 Log in to the admin panel

Go to `https://vault.ladkau.de/admin/` and enter the `vaultwarden_admin_token`
from the vault. From here you can invite users and manage the instance.

### 6.2 Sign in with Keycloak SSO

Users can sign in at `https://vault.ladkau.de` using the **Enterprise Single
Sign-On** button — enter `https://sso.ladkau.de/realms/ladkau` as the
organisation identifier. On first login Vaultwarden creates an account
automatically (public self-registration is otherwise disabled).

### 6.3 Connect a Bitwarden client

In any official Bitwarden client, set the **Server URL** to
`https://vault.ladkau.de` before logging in.

## 7. Download server

Files placed under `dl.ladkau.de` are publicly browsable and downloadable over
HTTPS. Upload is via SFTP on port 2223, key auth only.

### 7.1 Generate the deploy key

Run this once locally and keep both files:

```bash
ssh-keygen -t ed25519 -f dl_deploy_key -N "" -C "gitea-actions"
```

- Add the contents of `dl_deploy_key.pub` to the vault as `dl_sftp_authorized_keys`
  and redeploy (`--tags dl`) to authorise the key.
- Store `dl_deploy_key` (the private key) as a Gitea Actions secret named
  `DL_SSH_KEY` in any repository that needs to publish releases.

### 7.2 Upload from Gitea Actions

Add a step to your workflow after building the binary:

```yaml
- name: Upload release
  run: |
    echo "${{ secrets.DL_SSH_KEY }}" > /tmp/deploy_key
    chmod 600 /tmp/deploy_key
    ssh -i /tmp/deploy_key -p 2223 \
        -o StrictHostKeyChecking=no \
        uploader@dl.ladkau.de \
        "mkdir -p files/releases/${{ gitea.ref_name }}"
    scp -i /tmp/deploy_key -P 2223 \
        -o StrictHostKeyChecking=no \
        dist/myapp-linux-amd64 \
        uploader@dl.ladkau.de:files/releases/${{ gitea.ref_name }}/
    rm /tmp/deploy_key
```

The file is then available at:
`https://dl.ladkau.de/releases/v1.2.3/myapp-linux-amd64`

### 7.3 Connect an SFTP client

| Setting   | Value              |
|-----------|--------------------|
| Host      | `dl.ladkau.de`     |
| Port      | `2223`             |
| User      | `uploader`         |
| Auth      | SSH key            |
| Root path | `files/`           |

## 8. Container registry

### 8.1 Docker CLI usage

```bash
# Login
docker login cr.ladkau.de

# Push
docker tag myimage:latest cr.ladkau.de/myimage:latest
docker push cr.ladkau.de/myimage:latest

# Pull
docker pull cr.ladkau.de/myimage:latest
```

Registry users are managed via `registry_users` in the vault — store plaintext
passwords, Ansible generates bcrypt hashes at deploy time. To add or rotate a
user, edit the vault and redeploy:

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags registry --ask-vault-pass
```

### 8.2 Automated garbage collection

A cleanup script runs every **Sunday at 03:00** via cron (deployed by the
`registry` Ansible role). Each run:

1. Deletes stale `_uploads/` sessions (failed or abandoned pushes)
2. Stops the registry, runs `garbage-collect` to remove unreferenced blobs,
   then restarts it
3. Prunes dangling Docker images from the host

Output is appended to `/var/log/registry-gc.log` and tagged `registry-gc` in
syslog. To check recent runs:

```bash
tail -50 /var/log/registry-gc.log
# or
grep registry-gc /var/log/syslog
```

To run manually at any time:

```bash
sudo /usr/local/bin/registry-gc
```

### 8.3 Registry web UI

A web dashboard is available at `https://cr.ladkau.de/`. Sign in with any
`registry_users` credential from the vault. The UI allows browsing repositories
and tags, inspecting image manifests and digests, and deleting images.

Traefik routes `/v2/` (Docker API) to the registry container and everything else
to the UI container — both on the same domain, so no CORS configuration is
needed. `docker push` and `docker pull` work exactly as before.

## 9. n8n — AI chat and scheduled digests

n8n is served at `https://agent.ladkau.de` behind HTTP basic auth
(`n8n_users` from the vault). It provides:

- an AI chat page at `https://agent.ladkau.de/chat` that can search the web and read web pages
- a Telegram bot that answers with Gemini (default) or the local Qwen model (`/qwen`), and that scheduled workflows can also post to
- scheduled workflows (scrape a page, summarise it with the local model, send the summary by mail)

Components: `n8n` (workflows), `browserless` (headless Chromium for
JavaScript-heavy pages), `searxng` (web search) and `ollama` (local model,
section 9.5). Ollama, browserless and SearXNG are reachable only on internal
Docker networks.

### 9.1 First login

Open `https://agent.ladkau.de`, pass the basic-auth prompt and create the n8n
owner account. The Ollama credential ("Ollama (local)") and the SMTP
credential ("Strato SMTP") are imported by Ansible; nothing needs to be
entered in the UI.

### 9.2 Chat

`https://agent.ladkau.de/chat` redirects to the hosted chat page of the
`Chat` workflow (`/webhook/<n8n_chat_webhook_id>/chat`). The chat is an AI
agent with two tools:

- `web_search` — queries SearXNG and returns the top 6 results (title, URL, snippet)
- `fetch_page` — opens a URL in browserless and returns the readable text

- Every page load starts a new conversation. The last 6 messages are kept as context; history is held in memory and is lost when n8n restarts.
- `fetch_page` returns at most about 3000 characters of page text (title plus visible text without navigation). Tool results are kept in the chat memory.

### 9.3 Workflows in git

Workflows live in `ansible/roles/n8n/files/workflows/*.json` and are imported
by Ansible on every deploy that changes them:

| File | Workflow | Purpose |
|------|----------|---------|
| `chat.json` | Chat | Hosted chat page with the `fetch_page` tool |
| `fetch-page.json` | Fetch page text | Sub-workflow: renders a URL in browserless and returns title and text |
| `web-search.json` | Web search | Sub-workflow: queries SearXNG and returns the top results |
| `digest.json` | Weekly page digest (example) | Schedule → fetch page → summarise → mail; inactive until configured |
| `telegram-chat.json` | Telegram chat | Telegram bot: Gemini agent by default, Qwen agent for `/qwen`, both with the search and page tools; also has a `Post message` entry point used by other workflows |
| `digest-telegram.json` | Weekly page digest to Telegram (example) | Schedule → fetch page → summarise → post to the Telegram chat; inactive until configured |

A workflow with `"active": true` in its JSON is published on import. Sub-workflows
that are used as tools (`fetch-page`, `web-search`) must be published too; n8n
refuses to run an unpublished workflow, even when another workflow calls it. The
import replaces workflows with the same id, so changes made in the editor must
be exported to the repo or they are overwritten by the next import:

1. In the n8n editor open the workflow, then **… → Download** (or copy it with Ctrl+A, Ctrl+C).
2. Save the JSON as `ansible/roles/n8n/files/workflows/<name>.json`. Keep the `id` and, for the chat workflow, the `webhookId`.
3. Set `"active": true` in the file if the workflow should be published.
4. Commit, then deploy with `--tags n8n`.

Workflows call each other by id (`fetchpage0000001`, `websearch000001`, `telegramchat0001`).
They reference credentials by id: `ollama0000000001` ("Ollama (local)")
and `smtp00000000001` ("Strato SMTP").

To force a re-import without changing a file, remove `/opt/n8n/import/workflows`
on the server and run the role again.

### 9.4 Scheduled digest

Copy `digest.json` (mail) or `digest-telegram.json` (Telegram, section 9.8), then in the editor adjust:

- **Every Monday 06:00** — cron expression (`n8n_timezone` applies)
- **Sites** — one item per page: `url`, `prompt` and recipient `to`
- **Send mail** (mail version) — sender is the `smtp_user` address

Export the result to the repo (section 9.3) and set `"active": true`. Run it
once manually with **Execute workflow** before relying on the schedule.
A summary of a long page takes several minutes on the CPU-only model.

### 9.5 Ollama — local LLM server

The `ollama` role runs Ollama (CPU only). It is shared by n8n and fuelish and
reachable only on the internal Docker network `ollama_internal` at
`http://ollama:11434`; there is no public route.

Settings in `ansible/group_vars/all/vars.yml`:

| Variable | Default | Meaning |
|----------|---------|---------|
| `ollama_version` | `0.35.1` | Image tag |
| `ollama_model` | `qwen3:4b-instruct-2507-q4_K_M` | Model pulled on deploy; also written into the workflows (placeholder `__OLLAMA_MODEL__`) |
| `ollama_context_length` | `16384` | Context window in tokens |
| (workflow option) `numThread` | `4` | Threads used by the Ollama chat model nodes; Ollama defaults to 2 on this 2-core/4-thread host, 4 reads prompts about 60% faster |
| `ollama_mem_limit` | `4608m` | Container memory cap (VPS has 8 GB shared with ~25 containers) |
| `ollama_keep_alive` | `1h` | Idle time before the model is unloaded from RAM |

Deploy and verify:

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags ollama --ask-vault-pass
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
  "sudo docker exec ollama ollama list && sudo docker stats --no-stream ollama"
```

Changing `ollama_model` and redeploying pulls the new model; also change the
`model` field of the Ollama Chat Model nodes in the workflow JSON files.
Remove unused models with `docker exec ollama ollama rm <model>`. A CPU-only 4B
model at 16k context uses most of the memory available on this VPS; check
`docker stats` after the first model load.

### 9.6 Browserless

The `browserless` role runs headless Chromium (one session at a time, 1 GB
memory limit) on the internal network `browserless_internal` at
`http://browserless:3000`. Check it with:

```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
  "sudo docker run --rm --network browserless_internal curlimages/curl -s http://browserless:3000/pressure"
```

### 9.7 SearXNG

The `searxng` role runs SearXNG on the internal network `searxng_internal` at
`http://searxng:8080`. Its settings (`/opt/searxng/settings.yml`) are rendered
from `roles/searxng/templates/settings.yml.j2`. The JSON format must stay
enabled; the `web_search` tool depends on it. Check it with:

```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
  "sudo docker run --rm --network searxng_internal curlimages/curl -s 'http://searxng:8080/search?q=test&format=json' | head -c 300"
```

Some upstream engines (DuckDuckGo, Brave) may answer with captchas or rate
limits from the server's IP address; SearXNG then returns results from the
remaining engines. The search may return fewer results at times.

### 9.8 Telegram bot

The `Telegram chat` workflow lets you chat with the assistant from the Telegram
app and lets scheduled workflows post to the same conversation. The bot answers
only the Telegram user id in `telegram_user_id`; messages from anyone else and
non-text messages are ignored.

**Models**

| Message | Model |
|---------|-------|
| `what is the weather in Osnabrück?` | Google Gemini `gemini-3.8-flash`, then `gemini-3.5-flash` if that fails, then local Qwen |
| `/qwen what is the weather in Osnabrück?` | Local Qwen via Ollama (slow, free, private) |

If a Gemini model fails (daily quota, overload, anything else), the same
question goes to the next stage, and the reply ends with a note saying which
model answered. The stages are separate agents in `telegram-chat.json`
(`Gemini 3.8 Agent`, `Gemini 3.5 Agent`, `Qwen Agent`); the checks
`Gemini 3.8 ok?` and `Gemini 3.5 ok?` look for an empty answer.

All agents have the `web_search` and `fetch_page` tools. The Gemini agents share
one conversation memory (last 8 messages, session `telegram-main`), so the Gemini
models can be switched in the middle of a conversation. The Qwen agent has its own
memory (last 2 messages, session `telegram-qwen`).

The Gemini models are set in the `Gemini 3.8` and `Gemini 3.5` nodes of
`telegram-chat.json`. The model name is a plain string such as
`models/gemini-3.8-flash`. List the models a key can use with
`curl "https://generativelanguage.googleapis.com/v1beta/models?key=<KEY>"`; a
listed model can still be unavailable, so check it with a direct
`generateContent` call before using it.

Gemini API limits:

- The free tier limits requests per model, per minute and per day. The limit that was hit is named in the API error message. A question that uses a tool costs at least two requests (tool call, then answer).
- When a model is over its limit or overloaded, the agent retries once after 3 seconds and then the next stage answers.
- The system prompt asks for one `fetch_page` per web address or one `web_search` per question, never the same call twice. Agents stop after 4 iterations (`maxIterations`).

Gemini requests send messages, search results and page contents to Google;
check the terms of the API tier in use. If every stage fails the bot replies
with `Error: ...`.

The Qwen model is the non-thinking `qwen3:4b-instruct-2507` variant. The web chat
at `/chat` and the digests use the local Qwen model only.

When `ollama_model` changes, remove the old model on the server with
`docker exec ollama ollama rm <old model>`.

**Message formatting**

Replies are sent with `parse_mode` HTML (the n8n Telegram node otherwise uses
Telegram's legacy Markdown mode). The `Format reply` and `Format posted message`
steps escape the text and convert bold, italics, inline code, code blocks,
headings, bullets and `[text](https://…)` links to Telegram HTML. Tables are shown
as plain text.

**Get a Gemini API key**

Create a key in Google AI Studio (`aistudio.google.com`, **Get API key**) and
store it in the vault as `gemini_api_key`.

**Create the bot (once)**

1. In Telegram open `@BotFather`, send `/newbot` and follow the prompts. Copy the bot token.
2. Open a chat with your new bot and send it any message (for example `/start`).
3. Find your numeric user id. Before the first deploy (while no webhook is set) run:

   ```bash
   curl -s "https://api.telegram.org/bot<TOKEN>/getUpdates"
   ```

   and read `message.from.id` in the result.
4. Add `gemini_api_key`, `telegram_bot_token` and `telegram_user_id` to the vault
   (`ansible-vault edit ansible/group_vars/all/vault.yml`), check with
   `bash scripts/check-vault.sh`, and deploy:

   ```bash
   ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags n8n --ask-vault-pass
   ```

   On start n8n registers the webhook `https://agent.ladkau.de/webhook/<n8n_telegram_webhook_id>/webhook` with Telegram.

**Use it**

- Write to the bot like in the web chat. Replying to an earlier message (swipe on the message) adds the quoted text as context.
- Gemini answers within seconds; `/qwen` answers take a minute or more on the CPU-only model.
- Conversation memory holds the last 8 messages and is lost whenever n8n restarts; replying to a message restores its context.

**Stock price report**

The `Telegram chat` workflow asks for the stock prices of MTE.DE and PTX.F three
times a day on weekdays at 07:30, 13:00 and 17:40 (time zone
`n8n_timezone`). The answer arrives in your Telegram chat through the normal
model chain, so you can ask follow-up questions. It can also be started:

- by sending `/stocks` to the bot (Gemini chain), or `/qwen /stocks` to test the local model with the same question
- from the n8n editor: open `Telegram chat` and run the `Run stock price now` node with **Execute step**; the answer is sent to Telegram

The question is the constant `STOCK_QUESTION` in the `Route` node. It names the
two Yahoo quote pages and tells the model to call `fetch_page` for each; adding
a ticker means adding its URL to that text. The model chooses the tool calls;
there is no fixed fetch step. The times are the three cron expressions of the
`Stock price schedule` node (a standard 5-field cron expression, where `1-5`
means Monday to Friday). Change either in
the editor, export the workflow to the repo (section 9.3) and deploy. The
schedule only runs while the workflow is published; every deploy that changes
workflows restarts n8n and registers it again.

**Scheduled posts**

`digest-telegram.json` ends with the sub-workflow call `Post to Telegram chat`.
The summary is sent to your chat and added to the chat memory, so a follow-up
question such as "what about the second point?" has the context. Any workflow
can do the same by calling the `Telegram chat` workflow with a `message`
input.

**Security notes**

- The Telegram webhook route has no basic auth because Telegram's servers call it. Traefik admits only the address ranges in `telegram_webhook_source_ranges` (Telegram's webhook servers; set the list to `[]` to turn the filter off if Telegram's ranges change and the bot stops receiving messages). n8n verifies Telegram's secret token header and the workflow filters on `telegram_user_id`.
- Telegram bot messages are not end-to-end encrypted. Summaries and questions pass through Telegram's servers.
- If the bot token leaks, revoke it with `/revoke` in `@BotFather`, update the vault and redeploy.

**Troubleshooting**

```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 "sudo docker logs --tail 50 n8n"
curl -s "https://api.telegram.org/bot<TOKEN>/getWebhookInfo"
```

`getWebhookInfo` shows the registered URL and the last delivery error.

## 10. Fuelish — receipt OCR API

`https://fuelish.ladkau.de` serves the job API of the fuelish Android app
(`POST /jobs`, `GET /jobs/{id}`, `GET /health`). Every `/jobs` request needs
the header `X-API-Key: <fuelish_api_key>`.

### 10.1 Publish an image version

In the fuelish project (`registry.env` with the registry credentials):

```bash
echo 0.1.1 > tooling-container/VERSION    # new version number
./build-image.sh && ./upload-image.sh      # pushes cr.ladkau.de/fuelish/tools:<version> and :latest
```

Set `fuelish_version` in `ansible/group_vars/all/vars.yml` to the same number
and deploy:

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags fuelish --ask-vault-pass
```

The image must be built from fuelish commit `7a91a37` or later; older images do
not check the API key.

### 10.2 Check the deployment

```bash
curl -s https://fuelish.ladkau.de/health                     # {"status":"ok"}
curl -s -o /dev/null -w "%{http_code}\n" -X POST https://fuelish.ladkau.de/jobs   # without key: 401 or 403
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 "sudo docker exec ollama ollama list"   # fuelish_model is present (the API pulls it if missing)
```

### 10.3 Android app

In the app's Settings tab enter `https://fuelish.ladkau.de` as server URL and
`fuelish_api_key` as API key.

`fuelish_model` defaults to `ollama_model`, the model used by n8n, so both share
one loaded model. The API sends the request options from `fuelish_num_ctx` and
`fuelish_num_thread` (16384 and 4, the same as the n8n Ollama nodes), so
alternating requests from both apps do not restart the Ollama runner. Keep these
values equal to `ollama_context_length` and the `numThread` of the n8n Ollama
nodes. Images before fuelish version 0.1.1 ignore the two settings.