matthias/server_cloud_ladkau_de
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace Clay with n8n, Ollama, SearXNG and browserless; add Telegram assistant

Browse Source 
Remove the Clay role (Claude Code web UI) and its Anthropic API key.

New roles:
- ollama: local CPU LLM (qwen3:4b-instruct), internal network only
- browserless: headless Chromium for JavaScript-heavy pages
- searxng: metasearch backend for the chat web_search tool
- n8n at agent.ladkau.de: web chat (/chat), scheduled workflows,
  Telegram bot; workflows and credentials are imported from the repo

Telegram bot: Gemini 3.8 -> Gemini 3.5 -> local Qwen fallback chain,
/qwen to force the local model, /stocks and a weekday schedule
(07:30, 13:00, 17:40) for the MTE.DE / PTX.F stock report.

Also: dashboard and check-services entries for the chat, vault checks
for the new secrets (n8n, SearXNG, Gemini, Telegram, SMTP), updated
CLAUDE.md, runbooks and README.
This commit is contained in:
ml
2026-10-03 17:10:33 +02:00
parent b3cf4e250a
commit 974486bc53
37 changed files with 2805 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbook-configuration.md
+245 -139
View File
@@ -25,7 +25,8 @@ The script checks these endpoints and verifies the expected HTTP status code:
| `https://k8s.ladkau.de` | 200 | Placeholder page |
| `https://vault.ladkau.de` | 200 | Vaultwarden web vault |
| `https://dl.ladkau.de` | 200 | Public download server |
| `https://agent.ladkau.de` | 401 | Clay — basic-auth prompt, correct without credentials |
| `https://agent.ladkau.de` | 401 | n8n — basic-auth prompt, correct without credentials |
| `https://agent.ladkau.de/chat` | 401 | AI chat — basic-auth prompt, correct without credentials |
A `000` result means the connection was refused or timed out — a container that
did not start. Keycloak and Nextcloud may return `502` for up to 90 seconds on
@@ -470,163 +471,268 @@ Traefik routes `/v2/` (Docker API) to the registry container and everything else
to the UI container — both on the same domain, so no CORS configuration is
needed. `docker push` and `docker pull` work exactly as before.
## 9. Clay — Claude Code web interface
## 9. n8n — AI chat and scheduled digests
Clay is a browser-based front end for Claude Code, served at
`https://agent.ladkau.de`. Access is protected by HTTP basic auth using the
`clay_users` credentials from the vault.
n8n is served at `https://agent.ladkau.de` behind HTTP basic auth
(`n8n_users` from the vault). It provides:
### 9.1 Add projects
- an AI chat page at `https://agent.ladkau.de/chat` that can search the web and read web pages
- a Telegram bot that answers with Gemini (default) or the local Qwen model (`/qwen`), and that scheduled workflows can also post to
- scheduled workflows (scrape a page, summarise it with the local model, send the summary by mail)
The container mounts `/opt/clay/workspace` at `/workspace`. Clone Gitea
repositories there to make them available to Claude Code:
Components: `n8n` (workflows), `browserless` (headless Chromium for
JavaScript-heavy pages), `searxng` (web search) and `ollama` (local model,
section 9.5). Ollama, browserless and SearXNG are reachable only on internal
Docker networks.
### 9.1 First login
Open `https://agent.ladkau.de`, pass the basic-auth prompt and create the n8n
owner account. The Ollama credential ("Ollama (local)") and the SMTP
credential ("Strato SMTP") are imported by Ansible; nothing needs to be
entered in the UI.
### 9.2 Chat
`https://agent.ladkau.de/chat` redirects to the hosted chat page of the
`Chat` workflow (`/webhook/<n8n_chat_webhook_id>/chat`). The chat is an AI
agent with two tools:
- `web_search` — queries SearXNG and returns the top 6 results (title, URL, snippet)
- `fetch_page` — opens a URL in browserless and returns the readable text
- Every page load starts a new conversation. The last 6 messages are kept as context; history is held in memory and is lost when n8n restarts.
- `fetch_page` returns at most about 3000 characters of page text (title plus visible text without navigation). Tool results are kept in the chat memory.
### 9.3 Workflows in git
Workflows live in `ansible/roles/n8n/files/workflows/*.json` and are imported
by Ansible on every deploy that changes them:
| File | Workflow | Purpose |
|------|----------|---------|
| `chat.json` | Chat | Hosted chat page with the `fetch_page` tool |
| `fetch-page.json` | Fetch page text | Sub-workflow: renders a URL in browserless and returns title and text |
| `web-search.json` | Web search | Sub-workflow: queries SearXNG and returns the top results |
| `digest.json` | Weekly page digest (example) | Schedule → fetch page → summarise → mail; inactive until configured |
| `telegram-chat.json` | Telegram chat | Telegram bot: Gemini agent by default, Qwen agent for `/qwen`, both with the search and page tools; also has a `Post message` entry point used by other workflows |
| `digest-telegram.json` | Weekly page digest to Telegram (example) | Schedule → fetch page → summarise → post to the Telegram chat; inactive until configured |
A workflow with `"active": true` in its JSON is published on import. Sub-workflows
that are used as tools (`fetch-page`, `web-search`) must be published too; n8n
refuses to run an unpublished workflow, even when another workflow calls it. The
import replaces workflows with the same id, so changes made in the editor must
be exported to the repo or they are overwritten by the next import:
1. In the n8n editor open the workflow, then **… → Download** (or copy it with Ctrl+A, Ctrl+C).
2. Save the JSON as `ansible/roles/n8n/files/workflows/<name>.json`. Keep the `id` and, for the chat workflow, the `webhookId`.
3. Set `"active": true` in the file if the workflow should be published.
4. Commit, then deploy with `--tags n8n`.
Workflows call each other by id (`fetchpage0000001`, `websearch000001`, `telegramchat0001`).
They reference credentials by id: `ollama0000000001` ("Ollama (local)")
and `smtp00000000001` ("Strato SMTP").
To force a re-import without changing a file, remove `/opt/n8n/import/workflows`
on the server and run the role again.
### 9.4 Scheduled digest
Copy `digest.json` (mail) or `digest-telegram.json` (Telegram, section 9.8), then in the editor adjust:
- **Every Monday 06:00** — cron expression (`n8n_timezone` applies)
- **Sites** — one item per page: `url`, `prompt` and recipient `to`
- **Send mail** (mail version) — sender is the `smtp_user` address
Export the result to the repo (section 9.3) and set `"active": true`. Run it
once manually with **Execute workflow** before relying on the schedule.
A summary of a long page takes several minutes on the CPU-only model.
### 9.5 Ollama — local LLM server
The `ollama` role runs Ollama (CPU only). It is reachable only on the internal
Docker network `ollama_internal` at `http://ollama:11434`; there is no public
route.
Settings in `ansible/group_vars/all/vars.yml`:
| Variable | Default | Meaning |
|----------|---------|---------|
| `ollama_version` | `0.35.1` | Image tag |
| `ollama_model` | `qwen3:4b-instruct-2507-q4_K_M` | Model pulled on deploy; also written into the workflows (placeholder `__OLLAMA_MODEL__`) |
| `ollama_context_length` | `16384` | Context window in tokens |
| (workflow option) `numThread` | `4` | Threads used by the Ollama chat model nodes; Ollama defaults to 2 on this 2-core/4-thread host, 4 reads prompts about 60% faster |
| `ollama_mem_limit` | `4608m` | Container memory cap (VPS has 8 GB shared with ~25 containers) |
| `ollama_keep_alive` | `1h` | Idle time before the model is unloaded from RAM |
Deploy and verify:
```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148
cd /opt/clay/workspace
git clone https://gitea.ladkau.de/<username>/<repo>.git
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags ollama --ask-vault-pass
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
"sudo docker exec ollama ollama list && sudo docker stats --no-stream ollama"
```
Then in the clay web UI, use **Add project** and point it at `/workspace/<repo>`.
Changing `ollama_model` and redeploying pulls the new model; also change the
`model` field of the Ollama Chat Model nodes in the workflow JSON files.
Remove unused models with `docker exec ollama ollama rm <model>`. A CPU-only 4B
model at 16k context uses most of the memory available on this VPS; check
`docker stats` after the first model load.
### 9.2 MCP servers and skills
### 9.6 Browserless
Clay configuration (MCP server entries, session history, project registrations)
is stored in `~/.clay`, which is mounted from `/opt/clay/data` on the host and
survives container rebuilds.
MCP server **binaries** installed interactively inside a running container (e.g.
via `npm install -g` in a clay terminal) are part of the container filesystem
and are lost when the image is rebuilt.
To make a package survive rebuilds, add it to
`ansible/roles/clay/files/Dockerfile` and redeploy:
```dockerfile
# MCP servers and skills — add packages here to survive container rebuilds
RUN npm install -g @clay-ai/clay-ralph
RUN npm install -g @modelcontextprotocol/server-filesystem
```
```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags clay --ask-vault-pass
```
Ansible detects the Dockerfile change and triggers a rebuild automatically.
### 9.3 Rebuild after clay updates
The Docker image is built from `ansible/roles/clay/files/Dockerfile` at deploy
time. To pull upstream changes to `clay-server` or `@anthropic-ai/claude-code`,
rebuild the image and restart the container:
```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags clay --ask-vault-pass
```
The Ansible role forces a rebuild whenever the Dockerfile changes; to force a
rebuild without a Dockerfile change, delete the cached image first:
The `browserless` role runs headless Chromium (one session at a time, 1 GB
memory limit) on the internal network `browserless_internal` at
`http://browserless:3000`. Check it with:
```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
"sudo docker image rm clay:local && sudo docker compose -f /opt/clay/docker-compose.yml up -d"
"sudo docker run --rm --network browserless_internal curlimages/curl -s http://browserless:3000/pressure"
```
### 9.4 Interactive sessions vs. scheduled runs
### 9.7 SearXNG
Clay and Gitea Actions serve different purposes and are independent of each
other:
The `searxng` role runs SearXNG on the internal network `searxng_internal` at
`http://searxng:8080`. Its settings (`/opt/searxng/settings.yml`) are rendered
from `roles/searxng/templates/settings.yml.j2`. The JSON format must stay
enabled; the `web_search` tool depends on it. Check it with:
| | Clay | Gitea Actions |
|---|---|---|
| **Trigger** | You, in the browser | Schedule or manual button click |
| **Use case** | Exploratory, iterative work | Recurring automated tasks |
| **Project** | Repos cloned into `/opt/clay/workspace/` | Checked out fresh per run |
| **Output** | Interactive UI | Logs + optional email |
Clay has no scheduling API — there is no way to trigger a Clay session
programmatically. For recurring agent tasks, Gitea Actions runs Claude Code
directly on the act_runners, independent of Clay. If both operate on the same
git repository the context is identical, but the sessions are separate.
### 9.5 Scheduled Claude Code runs via Gitea Actions
The act_runner containers can run Claude Code on a recurring schedule using
Gitea's `on.schedule` trigger. Adding `workflow_dispatch` alongside it allows
the same workflow to be triggered manually from the Gitea UI without waiting
for the next scheduled run.
**Set up secrets**
In the repository go to **Settings → Secrets → Actions** (or
**Site Administration → Actions → Secrets** for org-wide secrets) and add:
| Secret | Value |
|--------|-------|
| `ANTHROPIC_API_KEY` | API key from the vault |
| `SMTP_USERNAME` | Gmail address used as the SMTP sender |
| `SMTP_PASSWORD` | Gmail App Password — myaccount.google.com → Security → App passwords |
**Workflow file**
Add `.gitea/workflows/weekly-agent.yml` to the repository:
```yaml
name: Weekly agent run
on:
schedule:
- cron: '0 6 * * 1' # Every Monday at 06:00 UTC
workflow_dispatch: # also runnable manually from the Gitea Actions UI
jobs:
agent:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Claude Code
run: npm install -g @anthropic-ai/claude-code
- name: Run agent
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
claude --print "Your prompt here" \
--output-format text \
| tee agent-output.txt
- name: Send report email
env:
SMTP_USERNAME: ${{ secrets.SMTP_USERNAME }}
SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}
run: |
python3 - <<'EOF'
import smtplib, os
from email.mime.text import MIMEText
with open("agent-output.txt") as f:
body = f.read()
msg = MIMEText(body)
msg["Subject"] = "Weekly agent report"
msg["From"] = os.environ["SMTP_USERNAME"]
msg["To"] = os.environ["SMTP_USERNAME"]
with smtplib.SMTP("smtp.gmail.com", 587) as s:
s.starttls()
s.login(os.environ["SMTP_USERNAME"], os.environ["SMTP_PASSWORD"])
s.send_message(msg)
EOF
```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
"sudo docker run --rm --network searxng_internal curlimages/curl -s 'http://searxng:8080/search?q=test&format=json' | head -c 300"
```
Adjust the `cron` expression, prompt, and `msg["To"]` address as needed.
Some upstream engines (DuckDuckGo, Brave) may answer with captchas or rate
limits from the server's IP address; SearXNG then returns results from the
remaining engines. The search may return fewer results at times.
**Run manually**
### 9.8 Telegram bot
Go to **Repository → Actions**, select **Weekly agent run** from the workflow
list, and click **Run workflow**. The run appears immediately in the Actions log.
The `Telegram chat` workflow lets you chat with the assistant from the Telegram
app and lets scheduled workflows post to the same conversation. The bot answers
only the Telegram user id in `telegram_user_id`; messages from anyone else and
non-text messages are ignored.
**Verify the schedule is active**
**Models**
After pushing the workflow file, scheduled runs appear automatically in
**Repository → Actions** at the configured time.
| Message | Model |
|---------|-------|
| `what is the weather in Osnabrück?` | Google Gemini `gemini-3.8-flash`, then `gemini-3.5-flash` if that fails, then local Qwen |
| `/qwen what is the weather in Osnabrück?` | Local Qwen via Ollama (slow, free, private) |
If a Gemini model fails (daily quota, overload, anything else), the same
question goes to the next stage, and the reply ends with a note saying which
model answered. The stages are separate agents in `telegram-chat.json`
(`Gemini 3.8 Agent`, `Gemini 3.5 Agent`, `Qwen Agent`); the checks
`Gemini 3.8 ok?` and `Gemini 3.5 ok?` look for an empty answer.
All agents have the `web_search` and `fetch_page` tools. The Gemini agents share
one conversation memory (last 8 messages, session `telegram-main`), so the Gemini
models can be switched in the middle of a conversation. The Qwen agent has its own
memory (last 2 messages, session `telegram-qwen`).
The Gemini models are set in the `Gemini 3.8` and `Gemini 3.5` nodes of
`telegram-chat.json`. The model name is a plain string such as
`models/gemini-3.8-flash`. List the models a key can use with
`curl "https://generativelanguage.googleapis.com/v1beta/models?key=<KEY>"`; a
listed model can still be unavailable, so check it with a direct
`generateContent` call before using it.
Gemini API limits:
- The free tier limits requests per model, per minute and per day. The limit that was hit is named in the API error message. A question that uses a tool costs at least two requests (tool call, then answer).
- When a model is over its limit or overloaded, the agent retries once after 3 seconds and then the next stage answers.
- The system prompt asks for one `fetch_page` per web address or one `web_search` per question, never the same call twice. Agents stop after 4 iterations (`maxIterations`).
Gemini requests send messages, search results and page contents to Google;
check the terms of the API tier in use. If every stage fails the bot replies
with `Error: ...`.
The Qwen model is the non-thinking `qwen3:4b-instruct-2507` variant. The web chat
at `/chat` and the digests use the local Qwen model only.
When `ollama_model` changes, remove the old model on the server with
`docker exec ollama ollama rm <old model>`.
**Message formatting**
Replies are sent with `parse_mode` HTML (the n8n Telegram node otherwise uses
Telegram's legacy Markdown mode). The `Format reply` and `Format posted message`
steps escape the text and convert bold, italics, inline code, code blocks,
headings, bullets and `[text](https://…)` links to Telegram HTML. Tables are shown
as plain text.
**Get a Gemini API key**
Create a key in Google AI Studio (`aistudio.google.com`, **Get API key**) and
store it in the vault as `gemini_api_key`.
**Create the bot (once)**
1. In Telegram open `@BotFather`, send `/newbot` and follow the prompts. Copy the bot token.
2. Open a chat with your new bot and send it any message (for example `/start`).
3. Find your numeric user id. Before the first deploy (while no webhook is set) run:
```bash
curl -s "https://api.telegram.org/bot<TOKEN>/getUpdates"
```
and read `message.from.id` in the result.
4. Add `gemini_api_key`, `telegram_bot_token` and `telegram_user_id` to the vault
(`ansible-vault edit ansible/group_vars/all/vault.yml`), check with
`bash scripts/check-vault.sh`, and deploy:
```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags n8n --ask-vault-pass
```
On start n8n registers the webhook `https://agent.ladkau.de/webhook/<n8n_telegram_webhook_id>/webhook` with Telegram.
**Use it**
- Write to the bot like in the web chat. Replying to an earlier message (swipe on the message) adds the quoted text as context.
- Gemini answers within seconds; `/qwen` answers take a minute or more on the CPU-only model.
- Conversation memory holds the last 8 messages and is lost whenever n8n restarts; replying to a message restores its context.
**Stock price report**
The `Telegram chat` workflow asks for the stock prices of MTE.DE and PTX.F three
times a day on weekdays at 07:30, 13:00 and 17:40 (time zone
`n8n_timezone`). The answer arrives in your Telegram chat through the normal
model chain, so you can ask follow-up questions. It can also be started:
- by sending `/stocks` to the bot (Gemini chain), or `/qwen /stocks` to test the local model with the same question
- from the n8n editor: open `Telegram chat` and run the `Run stock price now` node with **Execute step**; the answer is sent to Telegram
The question is the constant `STOCK_QUESTION` in the `Route` node. It names the
two Yahoo quote pages and tells the model to call `fetch_page` for each; adding
a ticker means adding its URL to that text. The model chooses the tool calls;
there is no fixed fetch step. The times are the three cron expressions of the
`Stock price schedule` node (a standard 5-field cron expression, where `1-5`
means Monday to Friday). Change either in
the editor, export the workflow to the repo (section 9.3) and deploy. The
schedule only runs while the workflow is published; every deploy that changes
workflows restarts n8n and registers it again.
**Scheduled posts**
`digest-telegram.json` ends with the sub-workflow call `Post to Telegram chat`.
The summary is sent to your chat and added to the chat memory, so a follow-up
question such as "what about the second point?" has the context. Any workflow
can do the same by calling the `Telegram chat` workflow with a `message`
input.
**Security notes**
- The Telegram webhook route has no basic auth because Telegram's servers call it. Traefik admits only the address ranges in `telegram_webhook_source_ranges` (Telegram's webhook servers; set the list to `[]` to turn the filter off if Telegram's ranges change and the bot stops receiving messages). n8n verifies Telegram's secret token header and the workflow filters on `telegram_user_id`.
- Telegram bot messages are not end-to-end encrypted. Summaries and questions pass through Telegram's servers.
- If the bot token leaks, revoke it with `/revoke` in `@BotFather`, update the vault and redeploy.
**Troubleshooting**
```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 "sudo docker logs --tail 50 n8n"
curl -s "https://api.telegram.org/bot<TOKEN>/getWebhookInfo"
```
`getWebhookInfo` shows the registered URL and the last delivery error.