server_cloud_ladkau_de/docs/runbook-provisioning.md

# Provisioning Runbook

Follow these steps to provision the server from scratch — whether setting it up
for the first time or reinstalling after a wipe.

After provisioning completes, follow `runbook-configuration.md` for first-run
setup of individual services.

## Prerequisites

### Local tools

- Ansible installed (`pip install ansible`)
- `htpasswd` available (`apt install apache2-utils` or `brew install httpd`)
- `docker` available (for generating the registry htpasswd entry)
- `openssl` available (for generating secrets)

### SSH keys

Two SSH key pairs are required. Place them in the `keys/` directory (private keys
are gitignored; public keys are committed).

| File | Purpose |
|------|---------|
| `keys/root_cloud_ladkau_de` | Root access — used only during initial bootstrap |
| `keys/root_cloud_ladkau_de.pub` | Public counterpart |
| `keys/notroot_cloud_ladkau_de` | Deploy user — used by Ansible for all subsequent runs |
| `keys/notroot_cloud_ladkau_de.pub` | Public counterpart — installed on server by bootstrap |

Generate fresh key pairs if they do not already exist:

```bash
ssh-keygen -t ed25519 -f keys/root_cloud_ladkau_de -C "root@cloud.ladkau.de" -N ""
ssh-keygen -t ed25519 -f keys/notroot_cloud_ladkau_de -C "deploy@cloud.ladkau.de" -N ""
```

Upload the root public key to the server via the Strato control panel (or paste
it during the initial OS install) so that root SSH access is available before
running anything. The server IP is defined in `ansible/inventory.ini`
(`ansible_host`).

## Steps

### 1. Set the server IP

Open `ansible/inventory.ini` and set `ansible_host` to the server's public IP:

```ini
cloud.ladkau.de ansible_host=<server-ip> ansible_user=deploy ansible_ssh_private_key_file=keys/notroot_cloud_ladkau_de
```

This is the only place the IP needs to be set — the bootstrap script and all
Ansible roles read it from here.

### 2. Install Ansible collections

From the repo root:

```bash
ansible-galaxy collection install -r ansible/requirements.yml
```

Required collections:
- `community.general` — ufw, timezone modules
- `ansible.posix` — authorized_key module
- `community.docker` — docker_network, docker_compose_v2 modules

### 3. Configure DNS

Ensure the following DNS A records all point to the server IP (`ansible_host`
in `ansible/inventory.ini`) before running the playbook. Traefik requests
Let's Encrypt certificates on first start and DNS must resolve at that point.

| Domain              | Record |
|---------------------|--------|
| cloud.ladkau.de     | A → server IP |
| gitea.ladkau.de     | A → server IP |
| nextcloud.ladkau.de | A → server IP |
| sso.ladkau.de       | A → server IP |
| mail.ladkau.de      | A → server IP |
| cr.ladkau.de        | A → server IP |
| k8s.ladkau.de       | A → server IP |

### 4. Create the vault and populate secrets

Create `ansible/group_vars/all/vault.yml` (gitignored) and encrypt it with Ansible Vault:

```bash
ansible-vault create ansible/group_vars/all/vault.yml
```

Populate all required secrets:

```yaml
# Traefik dashboard basic auth
# Generate: echo $(htpasswd -nB admin) | sed -e 's/\$/\$\$/g'
traefik_dashboard_users: "admin:$$2y$$05$$..."

# Gitea
# Generate secrets: openssl rand -hex 32
gitea_db_password: ""
gitea_secret_key: ""
gitea_internal_token: ""

# Keycloak
keycloak_db_password: ""
keycloak_admin_password: ""

# Nextcloud
nextcloud_db_password: ""
nextcloud_admin_password: ""

# Roundcube
# des_key must be exactly 24 characters: openssl rand -hex 12
roundcube_db_password: ""
roundcube_des_key: ""

# Container registry
# Generate: docker run --entrypoint htpasswd httpd:2 -Bbn <user> <password>
registry_htpasswd: "user:$2y$05$..."
```

Verify all secrets are present and non-empty:

```bash
bash scripts/check-vault.sh
```

To edit the vault later:

```bash
ansible-vault edit ansible/group_vars/all/vault.yml
```

### 5. Bootstrap the deploy user

```bash
bash scripts/run-bootstrap.sh
```

The script runs the following steps in order:

**Preflight checks (local)**
1. Verifies all four key files exist under `keys/` (both root and deploy key pairs)
2. Verifies `scripts/bootstrap-deploy-user.sh` exists
3. Sets `chmod 600` on the private key files (SSH refuses keys with open permissions)
4. Runs `scripts/check-vault.sh` — decrypts the vault and confirms all 11 required
   secrets are present and non-empty

**Remote actions**
5. Opens a test SSH connection as `root` to confirm the root key works
6. Copies `bootstrap-deploy-user.sh` to `/root/` on the server via `scp`
7. Executes it as root — creates the `deploy` user, grants passwordless sudo,
   and installs `keys/notroot_cloud_ladkau_de.pub` as the only authorized key
8. Opens a test SSH connection as `deploy` to confirm the new user can log in

If any step fails the script exits immediately with a descriptive error message.

### 6. Run the Ansible master playbook

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass
```

This runs all roles in order:

| # | Role       | What it does |
|---|------------|--------------|
| 1 | `base`     | OS hardening, deploy user, SSH config, ufw firewall, fail2ban |
| 2 | `docker`   | Docker Engine + Compose plugin, shared Traefik network |
| 3 | `traefik`  | Reverse proxy, automatic TLS via Let's Encrypt |
| 4 | `gitea`    | Self-hosted Git with PostgreSQL |
| 5 | `nextcloud`| File storage with PostgreSQL, Redis, cron sidecar |
| 6 | `sso`      | Keycloak single sign-on with PostgreSQL |
| 7 | `mail`     | Roundcube webmail client with PostgreSQL |
| 8 | `registry` | Docker Registry v2 with htpasswd auth |
| 9 | `k8s`      | Placeholder page at k8s.ladkau.de |

To apply a single role:

```bash
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags <role> --ask-vault-pass
```

## Scripts reference

| Script | Purpose |
|--------|---------|
| `scripts/run-bootstrap.sh` | Copy and run the deploy user bootstrap on a fresh server |
| `scripts/bootstrap-deploy-user.sh` | Runs on the server as root — creates the deploy user |
| `scripts/check-vault.sh` | Verify vault.yml exists and all required secrets are non-empty |
| `scripts/check-services.sh` | Verify all service endpoints are reachable (run after provisioning) |