Files
server_cloud_ladkau_de/docs/runbook-provisioning.md
T
ml e197ba7370 Fix all issues found during first live provisioning run
Traefik:
- Upgrade v3.3 → v3.6 to fix Docker API version negotiation failure
  with Docker Engine 29 (which dropped support for API < 1.40)

Directory permissions:
- Change all service parent directories from 0750 to 0755 so container
  processes can traverse them after dropping from root to a lower UID
- Create db directories owned by postgres UID 999 (mode 0700) so
  PostgreSQL can access its data files across Ansible runs
- Create /opt/gitea/data owned by git UID 1000 (Gitea writes there
  after dropping privileges)

Healthchecks:
- Fix Redis healthcheck: use CMD form instead of CMD-SHELL pipe
  (pipe was unreliable in Alpine)
- Fix Keycloak healthcheck: use bash /dev/tcp on management port 9000
  (curl not available in UBI image; was incorrectly targeting port 8080)

Image tags:
- Fix Roundcube: 1.6 and 1.6-apache do not exist; correct tag is 1.6.x-apache

Bootstrap scripts:
- bootstrap-deploy-user.sh: add sudoers.d entry for passwordless sudo
  (deploy user has no password so sudo group alone was not enough)
- run-bootstrap.sh: read server IP from inventory.ini, chmod 600 keys
  automatically, show actual SSH error on failure

Inventory / config:
- Add ansible_host to inventory.ini — server IP now defined in one place
- Restructure group_vars/ into all/ directory so vault.yml is
  auto-loaded (previously it did not match any group name)
- Move ansible.cfg to repo root (Ansible looks in cwd, not playbook dir)

Docs:
- Split runbook.md into runbook-provisioning.md and
  runbook-configuration.md
- Add step 1 (set server IP) to provisioning runbook
- Expand prerequisites with SSH key generation and upload instructions
- Expand bootstrap step with preflight check details
2026-06-28 07:05:43 +02:00

6.0 KiB

Provisioning Runbook

Follow these steps to provision the server from scratch — whether setting it up for the first time or reinstalling after a wipe.

After provisioning completes, follow runbook-configuration.md for first-run setup of individual services.

Prerequisites

Local tools

  • Ansible installed (pip install ansible)
  • htpasswd available (apt install apache2-utils or brew install httpd)
  • docker available (for generating the registry htpasswd entry)
  • openssl available (for generating secrets)

SSH keys

Two SSH key pairs are required. Place them in the keys/ directory (private keys are gitignored; public keys are committed).

File Purpose
keys/root_cloud_ladkau_de Root access — used only during initial bootstrap
keys/root_cloud_ladkau_de.pub Public counterpart
keys/notroot_cloud_ladkau_de Deploy user — used by Ansible for all subsequent runs
keys/notroot_cloud_ladkau_de.pub Public counterpart — installed on server by bootstrap

Generate fresh key pairs if they do not already exist:

ssh-keygen -t ed25519 -f keys/root_cloud_ladkau_de -C "root@cloud.ladkau.de" -N ""
ssh-keygen -t ed25519 -f keys/notroot_cloud_ladkau_de -C "deploy@cloud.ladkau.de" -N ""

Upload the root public key to the server via the Strato control panel (or paste it during the initial OS install) so that root SSH access is available before running anything. The server IP is defined in ansible/inventory.ini (ansible_host).

Steps

1. Set the server IP

Open ansible/inventory.ini and set ansible_host to the server's public IP:

cloud.ladkau.de ansible_host=<server-ip> ansible_user=deploy ansible_ssh_private_key_file=keys/notroot_cloud_ladkau_de

This is the only place the IP needs to be set — the bootstrap script and all Ansible roles read it from here.

2. Install Ansible collections

From the repo root:

ansible-galaxy collection install -r ansible/requirements.yml

Required collections:

  • community.general — ufw, timezone modules
  • ansible.posix — authorized_key module
  • community.docker — docker_network, docker_compose_v2 modules

3. Configure DNS

Ensure the following DNS A records all point to the server IP (ansible_host in ansible/inventory.ini) before running the playbook. Traefik requests Let's Encrypt certificates on first start and DNS must resolve at that point.

Domain Record
cloud.ladkau.de A → server IP
gitea.ladkau.de A → server IP
nextcloud.ladkau.de A → server IP
sso.ladkau.de A → server IP
mail.ladkau.de A → server IP
cr.ladkau.de A → server IP
k8s.ladkau.de A → server IP

4. Create the vault and populate secrets

Create ansible/group_vars/all/vault.yml (gitignored) and encrypt it with Ansible Vault:

ansible-vault create ansible/group_vars/all/vault.yml

Populate all required secrets:

# Traefik dashboard basic auth
# Generate: echo $(htpasswd -nB admin) | sed -e 's/\$/\$\$/g'
traefik_dashboard_users: "admin:$$2y$$05$$..."

# Gitea
# Generate secrets: openssl rand -hex 32
gitea_db_password: ""
gitea_secret_key: ""
gitea_internal_token: ""

# Keycloak
keycloak_db_password: ""
keycloak_admin_password: ""

# Nextcloud
nextcloud_db_password: ""
nextcloud_admin_password: ""

# Roundcube
# des_key must be exactly 24 characters: openssl rand -hex 12
roundcube_db_password: ""
roundcube_des_key: ""

# Container registry
# Generate: docker run --entrypoint htpasswd httpd:2 -Bbn <user> <password>
registry_htpasswd: "user:$2y$05$..."

Verify all secrets are present and non-empty:

bash scripts/check-vault.sh

To edit the vault later:

ansible-vault edit ansible/group_vars/all/vault.yml

5. Bootstrap the deploy user

bash scripts/run-bootstrap.sh

The script runs the following steps in order:

Preflight checks (local)

  1. Verifies all four key files exist under keys/ (both root and deploy key pairs)
  2. Verifies scripts/bootstrap-deploy-user.sh exists
  3. Sets chmod 600 on the private key files (SSH refuses keys with open permissions)
  4. Runs scripts/check-vault.sh — decrypts the vault and confirms all 11 required secrets are present and non-empty

Remote actions 5. Opens a test SSH connection as root to confirm the root key works 6. Copies bootstrap-deploy-user.sh to /root/ on the server via scp 7. Executes it as root — creates the deploy user, grants passwordless sudo, and installs keys/notroot_cloud_ladkau_de.pub as the only authorized key 8. Opens a test SSH connection as deploy to confirm the new user can log in

If any step fails the script exits immediately with a descriptive error message.

6. Run the Ansible master playbook

ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass

This runs all roles in order:

# Role What it does
1 base OS hardening, deploy user, SSH config, ufw firewall, fail2ban
2 docker Docker Engine + Compose plugin, shared Traefik network
3 traefik Reverse proxy, automatic TLS via Let's Encrypt
4 gitea Self-hosted Git with PostgreSQL
5 nextcloud File storage with PostgreSQL, Redis, cron sidecar
6 sso Keycloak single sign-on with PostgreSQL
7 mail Roundcube webmail client with PostgreSQL
8 registry Docker Registry v2 with htpasswd auth
9 k8s Placeholder page at k8s.ladkau.de

To apply a single role:

ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags <role> --ask-vault-pass

Scripts reference

Script Purpose
scripts/run-bootstrap.sh Copy and run the deploy user bootstrap on a fresh server
scripts/bootstrap-deploy-user.sh Runs on the server as root — creates the deploy user
scripts/check-vault.sh Verify vault.yml exists and all required secrets are non-empty