From 03d2f6d57aed4a998cb92ac6bb514c349c6f73b4 Mon Sep 17 00:00:00 2001 From: ml Date: Sat, 27 Jun 2026 18:44:05 +0200 Subject: [PATCH] Add bootstrap scripts and restructure runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/run-bootstrap.sh: copies and runs the deploy user bootstrap on a fresh server, verifies SSH login, checks vault as a preflight step - scripts/bootstrap-deploy-user.sh: runs on the server as root, accepts the SSH public key as an argument (read from keys/ by run-bootstrap.sh) - scripts/check-vault.sh: decrypts vault.yml and verifies all 11 required secrets are present and non-empty - docs/runbook.md: restructured into correct order (collections → DNS → vault → bootstrap → playbook), added scripts reference table, moved first-run notes for Gitea/Keycloak/registry into their own section - README.md: added scripts/ to repo layout --- README.md | 4 + docs/runbook.md | 216 ++++++++++++++++++------------- scripts/bootstrap-deploy-user.sh | 39 ++++++ scripts/check-vault.sh | 59 +++++++++ scripts/run-bootstrap.sh | 76 +++++++++++ 5 files changed, 306 insertions(+), 88 deletions(-) create mode 100755 scripts/bootstrap-deploy-user.sh create mode 100755 scripts/check-vault.sh create mode 100755 scripts/run-bootstrap.sh diff --git a/README.md b/README.md index f8025e5..46ae141 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,10 @@ Repo layout: k8s/ # k3s Kubernetes node docs/ runbook.md # step-by-step reinstall instructions + scripts/ + run-bootstrap.sh # bootstrap deploy user on a fresh server + bootstrap-deploy-user.sh # runs on server as root — creates deploy user + check-vault.sh # verify vault.yml has all required secrets keys/ *.pub # public SSH keys (private keys are gitignored) diff --git a/docs/runbook.md b/docs/runbook.md index 6ffc824..51f4ba4 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -1,12 +1,39 @@ -# Reinstall Runbook +# Provisioning Runbook -Follow these steps to provision a fresh server from scratch. +Follow these steps to provision the server from scratch — whether setting it up +for the first time or reinstalling after a wipe. ## Prerequisites -On your local machine: +### Local tools + - Ansible installed (`pip install ansible`) -- SSH access to the server as root using `keys/root_cloud_ladkau_de` +- `htpasswd` available (`apt install apache2-utils` or `brew install httpd`) +- `docker` available (for generating the registry htpasswd entry) +- `openssl` available (for generating secrets) + +### SSH keys + +Two SSH key pairs are required. Place them in the `keys/` directory (private keys +are gitignored; public keys are committed). + +| File | Purpose | +|------|---------| +| `keys/root_cloud_ladkau_de` | Root access — used only during initial bootstrap | +| `keys/root_cloud_ladkau_de.pub` | Public counterpart | +| `keys/notroot_cloud_ladkau_de` | Deploy user — used by Ansible for all subsequent runs | +| `keys/notroot_cloud_ladkau_de.pub` | Public counterpart — installed on server by bootstrap | + +Generate fresh key pairs if they do not already exist: + +```bash +ssh-keygen -t ed25519 -f keys/root_cloud_ladkau_de -C "root@cloud.ladkau.de" -N "" +ssh-keygen -t ed25519 -f keys/notroot_cloud_ladkau_de -C "deploy@cloud.ladkau.de" -N "" +``` + +Upload the root public key to the server via the Strato control panel (or paste +it during the initial OS install) so that `root@217.154.207.148` is accessible +before running anything. ## Steps @@ -21,48 +48,41 @@ ansible-galaxy collection install -r ansible/requirements.yml Required collections: - `community.general` — ufw, timezone modules - `ansible.posix` — authorized_key module +- `community.docker` — docker_network, docker_compose_v2 modules -### 2. Initial root login +### 2. Configure DNS -Connect as root and verify the server is reachable: +Ensure the following DNS A records point to `217.154.207.148` before running +the playbook. Traefik requests Let's Encrypt certificates on first start and +DNS must resolve at that point. -``` -ssh -i keys/root_cloud_ladkau_de root@217.154.207.148 -``` +| Domain | Record | +|---------------------|--------| +| cloud.ladkau.de | A → 217.154.207.148 | +| gitea.ladkau.de | A → 217.154.207.148 | +| nextcloud.ladkau.de | A → 217.154.207.148 | +| sso.ladkau.de | A → 217.154.207.148 | +| mail.ladkau.de | A → 217.154.207.148 | +| cr.ladkau.de | A → 217.154.207.148 | +| k8s.ladkau.de | A → 217.154.207.148 | -### 3. Create the deploy user (one-time, manual) +### 3. Create the vault and populate secrets -On the server as root: - -```bash -adduser deploy -usermod -aG sudo deploy -mkdir -p /home/deploy/.ssh -cat >> /home/deploy/.ssh/authorized_keys <<'EOF' - -EOF -chown -R deploy:deploy /home/deploy/.ssh -chmod 700 /home/deploy/.ssh -chmod 600 /home/deploy/.ssh/authorized_keys -``` - -### 4. Set secrets (Ansible Vault) - -Before running the playbook, populate secrets that must not be committed in plain text. Create `ansible/group_vars/vault.yml` (gitignored) and encrypt it with Ansible Vault: ```bash ansible-vault create ansible/group_vars/vault.yml ``` -Required secrets: +Populate all required secrets: ```yaml -# Traefik dashboard basic auth — generate with: -# echo $(htpasswd -nB admin) | sed -e 's/\$/\$\$/g' +# Traefik dashboard basic auth +# Generate: echo $(htpasswd -nB admin) | sed -e 's/\$/\$\$/g' traefik_dashboard_users: "admin:$$2y$$05$$..." -# Gitea — generate secrets with: openssl rand -hex 32 +# Gitea +# Generate secrets: openssl rand -hex 32 gitea_db_password: "" gitea_secret_key: "" gitea_internal_token: "" @@ -75,83 +95,103 @@ keycloak_admin_password: "" nextcloud_db_password: "" nextcloud_admin_password: "" -# Roundcube — des_key must be exactly 24 characters: openssl rand -hex 12 +# Roundcube +# des_key must be exactly 24 characters: openssl rand -hex 12 roundcube_db_password: "" roundcube_des_key: "" -# Container registry — generate with: -# docker run --entrypoint htpasswd httpd:2 -Bbn +# Container registry +# Generate: docker run --entrypoint htpasswd httpd:2 -Bbn registry_htpasswd: "user:$2y$05$..." ``` -**Registry usage after deploy:** +Verify all secrets are present and non-empty: ```bash -# Login -docker login cr.ladkau.de - -# Push an image -docker tag myimage:latest cr.ladkau.de/myimage:latest -docker push cr.ladkau.de/myimage:latest - -# Pull an image -docker pull cr.ladkau.de/myimage:latest +bash scripts/check-vault.sh ``` -**Gitea first-run note:** `gitea_disable_registration` defaults to `true` in -`group_vars/all.yml`. For the very first deploy, override it to `false` in -`vault.yml` so the Gitea setup wizard can create the admin account. After -the admin account exists, remove the override and redeploy. +To edit the vault later: -**Keycloak first-run note:** The `KEYCLOAK_ADMIN` / `KEYCLOAK_ADMIN_PASSWORD` -environment variables bootstrap the initial admin account on first start only. -After logging in at `https://sso.ladkau.de`, change the admin password via the -Keycloak UI and remove the `keycloak_admin_password` reference from vault (or -rotate it). Keycloak ignores these env vars once the admin account already exists. +```bash +ansible-vault edit ansible/group_vars/vault.yml +``` -To run the playbook with vault: +### 4. Bootstrap the deploy user + +Copies the bootstrap script to the server, runs it as root, and verifies the +deploy user can log in. Also runs `check-vault.sh` as a preflight check. + +```bash +bash scripts/run-bootstrap.sh +``` + +### 5. Run the Ansible master playbook ```bash ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass ``` -### 5. Run the Ansible master playbook - -From the repo root: - -```bash -ansible-playbook -i ansible/inventory.ini ansible/site.yml -``` - This runs all roles in order: -1. `base` — OS hardening, SSH config, ufw firewall -2. `docker` — Docker Engine + Compose plugin -3. `traefik` — reverse proxy, TLS via Let's Encrypt -4. `gitea` — self-hosted Git -5. `nextcloud` — file storage -6. `sso` — Single Sign-On -7. `mail` — mail server -8. `registry` — container registry -9. `k8s` — k3s node -### 6. DNS +| # | Role | What it does | +|---|------------|--------------| +| 1 | `base` | OS hardening, deploy user, SSH config, ufw firewall, fail2ban | +| 2 | `docker` | Docker Engine + Compose plugin, shared Traefik network | +| 3 | `traefik` | Reverse proxy, automatic TLS via Let's Encrypt | +| 4 | `gitea` | Self-hosted Git with PostgreSQL | +| 5 | `nextcloud`| File storage with PostgreSQL, Redis, cron sidecar | +| 6 | `sso` | Keycloak single sign-on with PostgreSQL | +| 7 | `mail` | Roundcube webmail client with PostgreSQL | +| 8 | `registry` | Docker Registry v2 with htpasswd auth | +| 9 | `k8s` | Placeholder page at k8s.ladkau.de | -Ensure the following DNS A records point to `217.154.207.148` before running: - -- cloud.ladkau.de -- gitea.ladkau.de -- nextcloud.ladkau.de -- sso.ladkau.de -- mail.ladkau.de -- cr.ladkau.de -- k8s.ladkau.de - -Traefik will attempt ACME certificate issuance on first start; DNS must resolve first. - -## Re-running after changes - -The playbook is idempotent. To apply a single role only: +To apply a single role: ```bash -ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags +ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags --ask-vault-pass ``` + +## First-run notes + +### Gitea + +`gitea_disable_registration` defaults to `true`. For the first deploy, override +it to `false` in `vault.yml` so the setup wizard can create the admin account: + +```bash +ansible-vault edit ansible/group_vars/vault.yml +# add: gitea_disable_registration: false +ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags gitea --ask-vault-pass +``` + +After the admin account is created at `https://gitea.ladkau.de`, remove the +override and redeploy to close public registration. + +### Keycloak + +`KEYCLOAK_ADMIN` / `KEYCLOAK_ADMIN_PASSWORD` bootstrap the initial admin account +on first start only — Keycloak ignores them once the account exists. After +logging in at `https://sso.ladkau.de`, change the admin password via the UI. + +### Container registry + +```bash +# Login +docker login cr.ladkau.de + +# Push +docker tag myimage:latest cr.ladkau.de/myimage:latest +docker push cr.ladkau.de/myimage:latest + +# Pull +docker pull cr.ladkau.de/myimage:latest +``` + +## Scripts reference + +| Script | Purpose | +|--------|---------| +| `scripts/run-bootstrap.sh` | Copy and run the deploy user bootstrap on a fresh server | +| `scripts/bootstrap-deploy-user.sh` | Runs on the server as root — creates the deploy user | +| `scripts/check-vault.sh` | Verify vault.yml exists and all required secrets are non-empty | diff --git a/scripts/bootstrap-deploy-user.sh b/scripts/bootstrap-deploy-user.sh new file mode 100755 index 0000000..20ff0b2 --- /dev/null +++ b/scripts/bootstrap-deploy-user.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Run once as root on a fresh server to create the deploy user Ansible connects as. +# Usage: bash bootstrap-deploy-user.sh + +set -euo pipefail + +SSH_PUBKEY="${1:?Usage: bootstrap-deploy-user.sh }" +DEPLOY_USER="deploy" + +if [ "$(id -u)" -ne 0 ]; then + echo "ERROR: must be run as root" >&2 + exit 1 +fi + +echo "==> Creating user '${DEPLOY_USER}'" +if id "${DEPLOY_USER}" &>/dev/null; then + echo " user already exists, skipping" +else + adduser --disabled-password --gecos "" "${DEPLOY_USER}" +fi + +echo "==> Adding '${DEPLOY_USER}' to sudo group" +usermod -aG sudo "${DEPLOY_USER}" + +echo "==> Installing SSH authorized key" +SSH_DIR="/home/${DEPLOY_USER}/.ssh" +AUTH_KEYS="${SSH_DIR}/authorized_keys" +mkdir -p "${SSH_DIR}" +if grep -qF "${SSH_PUBKEY}" "${AUTH_KEYS}" 2>/dev/null; then + echo " key already present, skipping" +else + echo "${SSH_PUBKEY}" >> "${AUTH_KEYS}" +fi +chown -R "${DEPLOY_USER}:${DEPLOY_USER}" "${SSH_DIR}" +chmod 700 "${SSH_DIR}" +chmod 600 "${AUTH_KEYS}" + +echo "" +echo "Done." diff --git a/scripts/check-vault.sh b/scripts/check-vault.sh new file mode 100755 index 0000000..411f5e2 --- /dev/null +++ b/scripts/check-vault.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Decrypts ansible/group_vars/vault.yml and checks all required secrets are present +# and non-empty. Exits non-zero if any are missing. +# Usage: bash scripts/check-vault.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" +VAULT_FILE="${REPO_ROOT}/ansible/group_vars/vault.yml" + +REQUIRED_KEYS=( + traefik_dashboard_users + gitea_db_password + gitea_secret_key + gitea_internal_token + keycloak_db_password + keycloak_admin_password + nextcloud_db_password + nextcloud_admin_password + roundcube_db_password + roundcube_des_key + registry_htpasswd +) + +if [ ! -f "${VAULT_FILE}" ]; then + echo "ERROR: vault file not found at ${VAULT_FILE}" >&2 + echo " Create it with: ansible-vault create ansible/group_vars/vault.yml" >&2 + exit 1 +fi + +if ! command -v ansible-vault &>/dev/null; then + echo "ERROR: ansible-vault not found — install with: pip install ansible" >&2 + exit 1 +fi + +echo "==> Decrypting vault (you will be prompted for the vault password)" +VAULT_CONTENT=$(ansible-vault view "${VAULT_FILE}") + +MISSING=() +for key in "${REQUIRED_KEYS[@]}"; do + # Match lines like: key: "value" or key: value — but not key: "" or key: '' or missing + if ! echo "${VAULT_CONTENT}" | grep -qE "^${key}:[[:space:]]+[^'\"[:space:]]|^${key}:[[:space:]]+['\"].+['\"]"; then + MISSING+=("${key}") + fi +done + +if [ ${#MISSING[@]} -gt 0 ]; then + echo "" >&2 + echo "ERROR: the following secrets are missing or empty in vault.yml:" >&2 + for key in "${MISSING[@]}"; do + echo " - ${key}" >&2 + done + echo "" >&2 + echo "Edit the vault with: ansible-vault edit ansible/group_vars/vault.yml" >&2 + exit 1 +fi + +echo " OK — all required secrets are present" diff --git a/scripts/run-bootstrap.sh b/scripts/run-bootstrap.sh new file mode 100755 index 0000000..d556fce --- /dev/null +++ b/scripts/run-bootstrap.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Copies bootstrap-deploy-user.sh to the server and runs it as root. +# Run from the repo root: bash scripts/run-bootstrap.sh +# +# Requires: ssh access to the server as root via keys/root_cloud_ladkau_de + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" + +SERVER="217.154.207.148" +ROOT_KEY="${REPO_ROOT}/keys/root_cloud_ladkau_de" +DEPLOY_KEY="${REPO_ROOT}/keys/notroot_cloud_ladkau_de" +DEPLOY_PUBKEY="${REPO_ROOT}/keys/notroot_cloud_ladkau_de.pub" +BOOTSTRAP_SCRIPT="${SCRIPT_DIR}/bootstrap-deploy-user.sh" + +# --- Preflight checks --- + +if [ ! -f "${ROOT_KEY}" ]; then + echo "ERROR: root SSH key not found at ${ROOT_KEY}" >&2 + exit 1 +fi + +if [ ! -f "${DEPLOY_KEY}" ]; then + echo "ERROR: deploy SSH key not found at ${DEPLOY_KEY}" >&2 + exit 1 +fi + +if [ ! -f "${DEPLOY_PUBKEY}" ]; then + echo "ERROR: deploy public key not found at ${DEPLOY_PUBKEY}" >&2 + exit 1 +fi + +if [ ! -f "${BOOTSTRAP_SCRIPT}" ]; then + echo "ERROR: bootstrap script not found at ${BOOTSTRAP_SCRIPT}" >&2 + exit 1 +fi + +# --- Vault check --- + +bash "${SCRIPT_DIR}/check-vault.sh" + +SSH_OPTS="-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10" + +# --- Verify root access --- + +echo "==> Verifying root SSH access to ${SERVER}" +if ! ssh ${SSH_OPTS} -i "${ROOT_KEY}" "root@${SERVER}" "echo ok" &>/dev/null; then + echo "ERROR: cannot connect as root to ${SERVER}" >&2 + exit 1 +fi + +# --- Copy and run the bootstrap script --- + +echo "==> Copying bootstrap script to server" +scp ${SSH_OPTS} -i "${ROOT_KEY}" "${BOOTSTRAP_SCRIPT}" "root@${SERVER}:/root/bootstrap-deploy-user.sh" + +echo "==> Running bootstrap script on server" +SSH_PUBKEY="$(cat "${DEPLOY_PUBKEY}")" +ssh ${SSH_OPTS} -i "${ROOT_KEY}" "root@${SERVER}" "bash /root/bootstrap-deploy-user.sh '${SSH_PUBKEY}'" + +# --- Verify deploy user access --- + +echo "" +echo "==> Verifying deploy user SSH access" +if ssh ${SSH_OPTS} -i "${DEPLOY_KEY}" "deploy@${SERVER}" "echo ok" &>/dev/null; then + echo " OK — deploy user can log in" +else + echo "ERROR: deploy user login failed — check the bootstrap output above" >&2 + exit 1 +fi + +echo "" +echo "Bootstrap complete. Next step:" +echo " ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass"