server_cloud_ladkau_de/docs/runbook-configuration.md

# Configuration Runbook

First-run configuration steps to perform after the Ansible playbook has
provisioned the server. See `runbook-provisioning.md` for the provisioning steps.

## 1. Verify all services are reachable

Before configuring individual services, confirm every HTTPS endpoint is up and
TLS certificates are valid. Run this from your local machine:

```bash
bash scripts/check-services.sh
```

The script checks these endpoints and verifies the expected HTTP status code:

| URL | Expected | Notes |
|-----|----------|-------|
| `https://cloud.ladkau.de/dashboard/` | 401 | Basic-auth prompt — correct without credentials |
| `https://gitea.ladkau.de` | 200 | Gitea sign-in page |
| `https://nextcloud.ladkau.de` | 200 | First visit triggers setup and takes 1–2 min |
| `https://sso.ladkau.de/realms/master` | 200 | Keycloak realm JSON — first start takes ~90 s |
| `https://mail.ladkau.de` | 200 | Roundcube webmail login |
| `https://cr.ladkau.de/v2/` | 401 | Registry API — auth required, correct without credentials |
| `https://k8s.ladkau.de` | 200 | Placeholder page |
| `https://vault.ladkau.de` | 200 | Vaultwarden web vault |
| `https://dl.ladkau.de` | 200 | Public download server |
| `https://agent.ladkau.de` | 401 | Clay — basic-auth prompt, correct without credentials |

A `000` result means the connection was refused or timed out — a container that
did not start. Keycloak and Nextcloud may return `502` for up to 90 seconds on
first boot; wait and retry before investigating.

Once the script passes, open `https://cloud.ladkau.de/dashboard/` in a browser
and authenticate with the `traefik_dashboard_users` credentials to verify the
dashboard loads correctly.

## 2. Keycloak

Keycloak is the identity provider for all services. Configure it first so that
Gitea and Nextcloud can be wired up to SSO in the steps that follow.

Keycloak takes ~90 seconds to start on first boot. If the login page at
`https://sso.ladkau.de` is not immediately available, wait and retry.

### 2.1 Log in and change the admin password

1. Go to `https://sso.ladkau.de` and sign in with username `admin` and the
   value of `keycloak_admin_password` from the vault.
2. Click the **admin** dropdown (top right) → **Manage account** →
   **Password** → set a new password.

The bootstrap credentials are only used on first start. After changing the
password, the vault value is no longer authoritative — store the new password
in your password manager.

### 2.2 Create an application realm

1. In the left sidebar, click the realm dropdown (shows **master**) →
   **Create realm**.
2. Set **Realm name** to `ladkau` and click **Create**.
3. Make sure the `ladkau` realm is selected in the sidebar for all steps below.

The `master` realm is for Keycloak administration only. All application users
and clients live in `ladkau`.

### 2.3 Create users

For each user you want to be able to log in to Gitea and Nextcloud:

1. **Users** → **Add user**.
2. Fill in **Username**, **Email**, **First name**, **Last name** → **Create**.
3. Go to the **Credentials** tab → **Set password**.
4. Enter a password, turn off **Temporary**, and click **Save**.

### 2.4 Create the Gitea OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `gitea` → **Next**.
3. Turn **Client authentication** on (confidential client) → **Next**.
4. **Valid redirect URIs**: `https://gitea.ladkau.de/user/oauth2/keycloak/callback`
5. **Web origins**: `https://gitea.ladkau.de` → **Save**.
6. Go to the **Credentials** tab and copy the **Client secret** — you will
   need it in step 3.

### 2.5 Create the Nextcloud OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `nextcloud` → **Next**.
3. Turn **Client authentication** on → **Next**.
4. **Valid redirect URIs**: `https://nextcloud.ladkau.de/apps/user_oidc/code`
5. **Valid post logout redirect URIs**: `https://nextcloud.ladkau.de/*`
6. **Web origins**: `https://nextcloud.ladkau.de` → **Save**.
7. Go to the **Credentials** tab and copy the **Client secret** — you will
   need it in step 4.

### 2.6 Create the Vaultwarden OIDC client

1. **Clients** → **Create client**.
2. **Client type**: OpenID Connect — **Client ID**: `vaultwarden` → **Next**.
3. Turn **Client authentication** on → **Next**.
4. **Valid redirect URIs**: `https://vault.ladkau.de/*`
5. **Valid post logout redirect URIs**: `https://vault.ladkau.de/*`
6. **Web origins**: `https://vault.ladkau.de` → **Save**.
7. Go to the **Credentials** tab and copy the **Client secret**.
8. Add it to the vault as `vaultwarden_sso_client_secret` and redeploy:
   ```bash
   ansible-vault edit ansible/group_vars/all/vault.yml
   ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags vaultwarden --ask-vault-pass
   ```

## 3. Gitea

### 3.1 Create the admin account

1. Go to `https://gitea.ladkau.de` — Gitea shows an initial configuration
   dialog on the first visit.
2. Scroll to the bottom of the form to the **Administrator Account Settings**
   section and fill in the admin username, email, and password.
3. Click **Install Gitea**.

Gitea initialises the database and creates the admin account in one step.
Public registration is closed — only users authenticating via Keycloak can
get an account (controlled by `ALLOW_ONLY_EXTERNAL_REGISTRATION`).

### 3.2 Connect Gitea to Keycloak

1. In Gitea, go to **Site Administration** → **Identity & Access** →
   **Authentication Sources** → **Add Authentication Source**.
2. Set **Authentication type** to **OAuth2**.
3. Set **OAuth2 provider** to **OpenID Connect**.
4. Fill in:
   - **Name**: `keycloak`
   - **Client ID**: `gitea`
   - **Client secret**: the secret copied in step 2.4
   - **OpenID Connect Auto Discovery URL**:
     `https://sso.ladkau.de/realms/ladkau/.well-known/openid-configuration`
5. Click **Add Authentication Source**.

Users can now sign in via **Sign in with keycloak** on the Gitea login page.

### 3.3 Use SSH key authentication for Git

Gitea exposes SSH on port **2222** (host port mapped from the container's port 22).
Clone URLs use the full `ssh://` form because the port is non-standard.

**Generate an SSH key**

If you do not already have a key, generate a dedicated one for Gitea:

```bash
ssh-keygen -t ed25519 -C "gitea.ladkau.de" -f ~/.ssh/id_ed25519_gitea
```

Leave the passphrase empty for unattended use, or set one for additional
security. The command creates two files:

- `~/.ssh/id_ed25519_gitea` — private key (never share this)
- `~/.ssh/id_ed25519_gitea.pub` — public key (upload this to Gitea)

**Add your key to Gitea**

1. Go to `https://gitea.ladkau.de` → click your avatar (top right) →
   **Settings** → **SSH / GPG Keys** → **Add Key**.
2. Paste the contents of `~/.ssh/id_ed25519_gitea.pub`.
3. Click **Add Key**.

**Test the connection**

```bash
ssh -p 2222 -i ~/.ssh/id_ed25519_gitea -T git@gitea.ladkau.de
```

A successful response looks like:

```
Hi <username>! You've successfully authenticated, but Gitea does not provide shell access.
```

**Clone, push, and pull**

Gitea shows the correct SSH URL on every repository page. It takes the form:

```
ssh://git@gitea.ladkau.de:2222/<username>/<repo>.git
```

```bash
# Clone
git clone ssh://git@gitea.ladkau.de:2222/<username>/myrepo.git

# Add remote on an existing repo
git remote add origin ssh://git@gitea.ladkau.de:2222/<username>/myrepo.git
```

**Optional: SSH config alias**

Add this to `~/.ssh/config` to avoid repeating the port:

```
Host gitea.ladkau.de
    User git
    Port 2222
    IdentityFile ~/.ssh/id_ed25519_gitea
```

With the alias in place you can use the shorter SCP-like URL:

```bash
git clone git@gitea.ladkau.de:<username>/myrepo.git
```

### 3.4 Deploy Gitea Actions runners

Three `act_runner` containers (Docker executor) are provisioned by the
`act_runner` Ansible role. Each runner handles one concurrent job; together
they allow up to three parallel workflow jobs.

**This step requires Gitea to be running and the admin account to exist
(step 3.1). Runners cannot register until a token is obtained from Gitea.**

**Step 1 — Get a runner registration token**

1. Go to `https://gitea.ladkau.de` → sign in as admin.
2. Navigate to **Site Administration** (top-right menu) →
   **Actions** → **Runners**.
3. Click **Create runner token** and copy the token.

**Step 2 — Add the token to the vault**

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
```

Add the key:

```yaml
gitea_runner_registration_token: "<token-from-step-1>"
```

**Step 3 — Deploy the runners**

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml \
  --tags act_runner --ask-vault-pass
```

Ansible creates `/opt/act_runner/` with a shared `config.yml` and three
per-runner data directories (`runner-1/`, `runner-2/`, `runner-3/`). On first
start each container auto-registers with Gitea and writes a `.runner` file to
its data directory. Subsequent restarts reuse the saved registration.

**Verify**

Back in Gitea **Site Administration → Actions → Runners**, all three runners
should appear as **Online** within a few seconds.

> **Security note:** Each runner container mounts `/var/run/docker.sock`.
> This gives workflow jobs root-equivalent access to the host Docker daemon.
> Only run workflows from trusted repositories.

## 4. Nextcloud

The admin credentials are set via `nextcloud_admin_user` and
`nextcloud_admin_password` in the vault. The first HTTP request triggers
installation, which takes a minute or two.

### 4.1 Log in

1. Go to `https://nextcloud.ladkau.de` and sign in with the admin credentials
   from the vault.

### 4.2 Connect Nextcloud to Keycloak

1. Go to **Apps** → search for **OpenID Connect user backend** → **Download
   and enable**.
2. Go to **Administration settings** → **OpenID Connect** → **Add provider**.
3. Fill in:
   - **Identifier**: `keycloak`
   - **Client ID**: `nextcloud`
   - **Client secret**: the secret copied in step 2.5
   - **Discovery endpoint**:
     `https://sso.ladkau.de/realms/ladkau/.well-known/openid-configuration`
4. Click **Save**.

Users can now sign in to Nextcloud with their Keycloak credentials via the
**Log in with keycloak** button.

## 5. Mail (Dovecot + Roundcube + Fetchmail)

The mail stack runs a local Dovecot IMAP server. Roundcube connects to it
internally. External IMAP clients connect to `mail.ladkau.de:993` (IMAPS —
TLS terminated by Traefik). Fetchmail polls external POP3 accounts and
delivers to Dovecot.

Dovecot users and fetchmail accounts are configured in the vault before
provisioning — see step 4 of `runbook-provisioning.md`. To add users or
accounts after initial deployment, edit the vault and redeploy:

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags mail --ask-vault-pass
```

### 5.1 Import mail from an old Dovecot server

If the old server uses **Maildir** format, rsync directly:

```bash
# Run on the old server — syncs to the new server's maildir volume
rsync -av -e "ssh -i keys/root_cloud_ladkau_de" --progress ./* \
  root@217.154.207.148:/opt/mail/maildir/alice/Maildir/

# Fix ownership for Dovecot's vmail user (UID 5000)
ssh -i keys/root_cloud_ladkau_de root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice"
```

If the old server uses **mbox** format (e.g. `mail_location = mbox:~/mail:...`),
see `runbook-mail-import.md` for the conversion procedure.

Alternatively, use `imapsync` to copy via IMAP from the old server to the
new one (works without direct server access):

```bash
imapsync \
  --host1 old.server.com --user1 alice --password1 oldpass \
  --host2 mail.ladkau.de  --user2 alice --password2 newpass \
  --port2 993 --ssl2
```

### 5.2 Connect an IMAP client

| Setting   | Value                |
|-----------|----------------------|
| Server    | `mail.ladkau.de`     |
| Port      | `993`                |
| Security  | SSL/TLS              |
| Username  | as set in vault      |

## 6. Vaultwarden

Vaultwarden is a self-hosted Bitwarden-compatible password vault. Any official
Bitwarden client (browser extension, desktop, mobile) can connect to it.

### 6.1 Log in to the admin panel

Go to `https://vault.ladkau.de/admin/` and enter the `vaultwarden_admin_token`
from the vault. From here you can invite users and manage the instance.

### 6.2 Sign in with Keycloak SSO

Users can sign in at `https://vault.ladkau.de` using the **Enterprise Single
Sign-On** button — enter `https://sso.ladkau.de/realms/ladkau` as the
organisation identifier. On first login Vaultwarden creates an account
automatically (public self-registration is otherwise disabled).

### 6.3 Connect a Bitwarden client

In any official Bitwarden client, set the **Server URL** to
`https://vault.ladkau.de` before logging in.

## 7. Download server

Files placed under `dl.ladkau.de` are publicly browsable and downloadable over
HTTPS. Upload is via SFTP on port 2223, key auth only.

### 7.1 Generate the deploy key

Run this once locally and keep both files:

```bash
ssh-keygen -t ed25519 -f dl_deploy_key -N "" -C "gitea-actions"
```

- Add the contents of `dl_deploy_key.pub` to the vault as `dl_sftp_authorized_keys`
  and redeploy (`--tags dl`) to authorise the key.
- Store `dl_deploy_key` (the private key) as a Gitea Actions secret named
  `DL_SSH_KEY` in any repository that needs to publish releases.

### 7.2 Upload from Gitea Actions

Add a step to your workflow after building the binary:

```yaml
- name: Upload release
  run: |
    echo "${{ secrets.DL_SSH_KEY }}" > /tmp/deploy_key
    chmod 600 /tmp/deploy_key
    ssh -i /tmp/deploy_key -p 2223 \
        -o StrictHostKeyChecking=no \
        uploader@dl.ladkau.de \
        "mkdir -p files/releases/${{ gitea.ref_name }}"
    scp -i /tmp/deploy_key -P 2223 \
        -o StrictHostKeyChecking=no \
        dist/myapp-linux-amd64 \
        uploader@dl.ladkau.de:files/releases/${{ gitea.ref_name }}/
    rm /tmp/deploy_key
```

The file is then available at:
`https://dl.ladkau.de/releases/v1.2.3/myapp-linux-amd64`

### 7.3 Connect an SFTP client

| Setting   | Value              |
|-----------|--------------------|
| Host      | `dl.ladkau.de`     |
| Port      | `2223`             |
| User      | `uploader`         |
| Auth      | SSH key            |
| Root path | `files/`           |

## 8. Container registry

### 8.1 Docker CLI usage

```bash
# Login
docker login cr.ladkau.de

# Push
docker tag myimage:latest cr.ladkau.de/myimage:latest
docker push cr.ladkau.de/myimage:latest

# Pull
docker pull cr.ladkau.de/myimage:latest
```

Registry users are managed via `registry_users` in the vault — store plaintext
passwords, Ansible generates bcrypt hashes at deploy time. To add or rotate a
user, edit the vault and redeploy:

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags registry --ask-vault-pass
```

### 8.2 Automated garbage collection

A cleanup script runs every **Sunday at 03:00** via cron (deployed by the
`registry` Ansible role). Each run:

1. Deletes stale `_uploads/` sessions (failed or abandoned pushes)
2. Stops the registry, runs `garbage-collect` to remove unreferenced blobs,
   then restarts it
3. Prunes dangling Docker images from the host

Output is appended to `/var/log/registry-gc.log` and tagged `registry-gc` in
syslog. To check recent runs:

```bash
tail -50 /var/log/registry-gc.log
# or
grep registry-gc /var/log/syslog
```

To run manually at any time:

```bash
sudo /usr/local/bin/registry-gc
```

### 8.3 Registry web UI

A web dashboard is available at `https://cr.ladkau.de/`. Sign in with any
`registry_users` credential from the vault. The UI allows browsing repositories
and tags, inspecting image manifests and digests, and deleting images.

Traefik routes `/v2/` (Docker API) to the registry container and everything else
to the UI container — both on the same domain, so no CORS configuration is
needed. `docker push` and `docker pull` work exactly as before.

## 9. Clay — Claude Code web interface

Clay is a browser-based front end for Claude Code, served at
`https://agent.ladkau.de`. Access is protected by HTTP basic auth using the
`clay_users` credentials from the vault.

### 9.1 Add projects

The container mounts `/opt/clay/workspace` at `/workspace`. Clone Gitea
repositories there to make them available to Claude Code:

```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148
cd /opt/clay/workspace
git clone https://gitea.ladkau.de/<username>/<repo>.git
```

Then in the clay web UI, use **Add project** and point it at `/workspace/<repo>`.

### 9.2 MCP servers and skills

Clay configuration (MCP server entries, session history, project registrations)
is stored in `~/.clay`, which is mounted from `/opt/clay/data` on the host and
survives container rebuilds.

MCP server **binaries** installed interactively inside a running container (e.g.
via `npm install -g` in a clay terminal) are part of the container filesystem
and are lost when the image is rebuilt.

To make a package survive rebuilds, add it to
`ansible/roles/clay/files/Dockerfile` and redeploy:

```dockerfile
# MCP servers and skills — add packages here to survive container rebuilds
RUN npm install -g @clay-ai/clay-ralph
RUN npm install -g @modelcontextprotocol/server-filesystem
```

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags clay --ask-vault-pass
```

Ansible detects the Dockerfile change and triggers a rebuild automatically.

### 9.3 Rebuild after clay updates

The Docker image is built from `ansible/roles/clay/files/Dockerfile` at deploy
time. To pull upstream changes to `clay-server` or `@anthropic-ai/claude-code`,
rebuild the image and restart the container:

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags clay --ask-vault-pass
```

The Ansible role forces a rebuild whenever the Dockerfile changes; to force a
rebuild without a Dockerfile change, delete the cached image first:

```bash
ssh -i keys/notroot_cloud_ladkau_de deploy@217.154.207.148 \
  "sudo docker image rm clay:local && sudo docker compose -f /opt/clay/docker-compose.yml up -d"
```

### 9.4 Interactive sessions vs. scheduled runs

Clay and Gitea Actions serve different purposes and are independent of each
other:

| | Clay | Gitea Actions |
|---|---|---|
| **Trigger** | You, in the browser | Schedule or manual button click |
| **Use case** | Exploratory, iterative work | Recurring automated tasks |
| **Project** | Repos cloned into `/opt/clay/workspace/` | Checked out fresh per run |
| **Output** | Interactive UI | Logs + optional email |

Clay has no scheduling API — there is no way to trigger a Clay session
programmatically. For recurring agent tasks, Gitea Actions runs Claude Code
directly on the act_runners, independent of Clay. If both operate on the same
git repository the context is identical, but the sessions are separate.

### 9.5 Scheduled Claude Code runs via Gitea Actions

The act_runner containers can run Claude Code on a recurring schedule using
Gitea's `on.schedule` trigger. Adding `workflow_dispatch` alongside it allows
the same workflow to be triggered manually from the Gitea UI without waiting
for the next scheduled run.

**Set up secrets**

In the repository go to **Settings → Secrets → Actions** (or
**Site Administration → Actions → Secrets** for org-wide secrets) and add:

| Secret | Value |
|--------|-------|
| `ANTHROPIC_API_KEY` | API key from the vault |
| `SMTP_USERNAME` | Gmail address used as the SMTP sender |
| `SMTP_PASSWORD` | Gmail App Password — myaccount.google.com → Security → App passwords |

**Workflow file**

Add `.gitea/workflows/weekly-agent.yml` to the repository:

```yaml
name: Weekly agent run
on:
  schedule:
    - cron: '0 6 * * 1'   # Every Monday at 06:00 UTC
  workflow_dispatch:       # also runnable manually from the Gitea Actions UI

jobs:
  agent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Claude Code
        run: npm install -g @anthropic-ai/claude-code

      - name: Run agent
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          claude --print "Your prompt here" \
                 --output-format text \
          | tee agent-output.txt

      - name: Send report email
        env:
          SMTP_USERNAME: ${{ secrets.SMTP_USERNAME }}
          SMTP_PASSWORD: ${{ secrets.SMTP_PASSWORD }}
        run: |
          python3 - <<'EOF'
          import smtplib, os
          from email.mime.text import MIMEText

          with open("agent-output.txt") as f:
              body = f.read()

          msg = MIMEText(body)
          msg["Subject"] = "Weekly agent report"
          msg["From"]    = os.environ["SMTP_USERNAME"]
          msg["To"]      = os.environ["SMTP_USERNAME"]

          with smtplib.SMTP("smtp.gmail.com", 587) as s:
              s.starttls()
              s.login(os.environ["SMTP_USERNAME"], os.environ["SMTP_PASSWORD"])
              s.send_message(msg)
          EOF
```

Adjust the `cron` expression, prompt, and `msg["To"]` address as needed.

**Run manually**

Go to **Repository → Actions**, select **Weekly agent run** from the workflow
list, and click **Run workflow**. The run appears immediately in the Actions log.

**Verify the schedule is active**

After pushing the workflow file, scheduled runs appear automatically in
**Repository → Actions** at the configured time.