Fix all issues found during first live provisioning run
Traefik: - Upgrade v3.3 → v3.6 to fix Docker API version negotiation failure with Docker Engine 29 (which dropped support for API < 1.40) Directory permissions: - Change all service parent directories from 0750 to 0755 so container processes can traverse them after dropping from root to a lower UID - Create db directories owned by postgres UID 999 (mode 0700) so PostgreSQL can access its data files across Ansible runs - Create /opt/gitea/data owned by git UID 1000 (Gitea writes there after dropping privileges) Healthchecks: - Fix Redis healthcheck: use CMD form instead of CMD-SHELL pipe (pipe was unreliable in Alpine) - Fix Keycloak healthcheck: use bash /dev/tcp on management port 9000 (curl not available in UBI image; was incorrectly targeting port 8080) Image tags: - Fix Roundcube: 1.6 and 1.6-apache do not exist; correct tag is 1.6.x-apache Bootstrap scripts: - bootstrap-deploy-user.sh: add sudoers.d entry for passwordless sudo (deploy user has no password so sudo group alone was not enough) - run-bootstrap.sh: read server IP from inventory.ini, chmod 600 keys automatically, show actual SSH error on failure Inventory / config: - Add ansible_host to inventory.ini — server IP now defined in one place - Restructure group_vars/ into all/ directory so vault.yml is auto-loaded (previously it did not match any group name) - Move ansible.cfg to repo root (Ansible looks in cwd, not playbook dir) Docs: - Split runbook.md into runbook-provisioning.md and runbook-configuration.md - Add step 1 (set server IP) to provisioning runbook - Expand prerequisites with SSH key generation and upload instructions - Expand bootstrap step with preflight check details
This commit is contained in:
ml
@@ -0,0 +1,191 @@
|
||||
# Provisioning Runbook
|
||||
|
||||
Follow these steps to provision the server from scratch — whether setting it up
|
||||
for the first time or reinstalling after a wipe.
|
||||
|
||||
After provisioning completes, follow `runbook-configuration.md` for first-run
|
||||
setup of individual services.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Local tools
|
||||
|
||||
- Ansible installed (`pip install ansible`)
|
||||
- `htpasswd` available (`apt install apache2-utils` or `brew install httpd`)
|
||||
- `docker` available (for generating the registry htpasswd entry)
|
||||
- `openssl` available (for generating secrets)
|
||||
|
||||
### SSH keys
|
||||
|
||||
Two SSH key pairs are required. Place them in the `keys/` directory (private keys
|
||||
are gitignored; public keys are committed).
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `keys/root_cloud_ladkau_de` | Root access — used only during initial bootstrap |
|
||||
| `keys/root_cloud_ladkau_de.pub` | Public counterpart |
|
||||
| `keys/notroot_cloud_ladkau_de` | Deploy user — used by Ansible for all subsequent runs |
|
||||
| `keys/notroot_cloud_ladkau_de.pub` | Public counterpart — installed on server by bootstrap |
|
||||
|
||||
Generate fresh key pairs if they do not already exist:
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f keys/root_cloud_ladkau_de -C "root@cloud.ladkau.de" -N ""
|
||||
ssh-keygen -t ed25519 -f keys/notroot_cloud_ladkau_de -C "deploy@cloud.ladkau.de" -N ""
|
||||
```
|
||||
|
||||
Upload the root public key to the server via the Strato control panel (or paste
|
||||
it during the initial OS install) so that root SSH access is available before
|
||||
running anything. The server IP is defined in `ansible/inventory.ini`
|
||||
(`ansible_host`).
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Set the server IP
|
||||
|
||||
Open `ansible/inventory.ini` and set `ansible_host` to the server's public IP:
|
||||
|
||||
```ini
|
||||
cloud.ladkau.de ansible_host=<server-ip> ansible_user=deploy ansible_ssh_private_key_file=keys/notroot_cloud_ladkau_de
|
||||
```
|
||||
|
||||
This is the only place the IP needs to be set — the bootstrap script and all
|
||||
Ansible roles read it from here.
|
||||
|
||||
### 2. Install Ansible collections
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
ansible-galaxy collection install -r ansible/requirements.yml
|
||||
```
|
||||
|
||||
Required collections:
|
||||
- `community.general` — ufw, timezone modules
|
||||
- `ansible.posix` — authorized_key module
|
||||
- `community.docker` — docker_network, docker_compose_v2 modules
|
||||
|
||||
### 3. Configure DNS
|
||||
|
||||
Ensure the following DNS A records all point to the server IP (`ansible_host`
|
||||
in `ansible/inventory.ini`) before running the playbook. Traefik requests
|
||||
Let's Encrypt certificates on first start and DNS must resolve at that point.
|
||||
|
||||
| Domain | Record |
|
||||
|---------------------|--------|
|
||||
| cloud.ladkau.de | A → server IP |
|
||||
| gitea.ladkau.de | A → server IP |
|
||||
| nextcloud.ladkau.de | A → server IP |
|
||||
| sso.ladkau.de | A → server IP |
|
||||
| mail.ladkau.de | A → server IP |
|
||||
| cr.ladkau.de | A → server IP |
|
||||
| k8s.ladkau.de | A → server IP |
|
||||
|
||||
### 4. Create the vault and populate secrets
|
||||
|
||||
Create `ansible/group_vars/all/vault.yml` (gitignored) and encrypt it with Ansible Vault:
|
||||
|
||||
```bash
|
||||
ansible-vault create ansible/group_vars/all/vault.yml
|
||||
```
|
||||
|
||||
Populate all required secrets:
|
||||
|
||||
```yaml
|
||||
# Traefik dashboard basic auth
|
||||
# Generate: echo $(htpasswd -nB admin) | sed -e 's/\$/\$\$/g'
|
||||
traefik_dashboard_users: "admin:$$2y$$05$$..."
|
||||
|
||||
# Gitea
|
||||
# Generate secrets: openssl rand -hex 32
|
||||
gitea_db_password: ""
|
||||
gitea_secret_key: ""
|
||||
gitea_internal_token: ""
|
||||
|
||||
# Keycloak
|
||||
keycloak_db_password: ""
|
||||
keycloak_admin_password: ""
|
||||
|
||||
# Nextcloud
|
||||
nextcloud_db_password: ""
|
||||
nextcloud_admin_password: ""
|
||||
|
||||
# Roundcube
|
||||
# des_key must be exactly 24 characters: openssl rand -hex 12
|
||||
roundcube_db_password: ""
|
||||
roundcube_des_key: ""
|
||||
|
||||
# Container registry
|
||||
# Generate: docker run --entrypoint htpasswd httpd:2 -Bbn <user> <password>
|
||||
registry_htpasswd: "user:$2y$05$..."
|
||||
```
|
||||
|
||||
Verify all secrets are present and non-empty:
|
||||
|
||||
```bash
|
||||
bash scripts/check-vault.sh
|
||||
```
|
||||
|
||||
To edit the vault later:
|
||||
|
||||
```bash
|
||||
ansible-vault edit ansible/group_vars/all/vault.yml
|
||||
```
|
||||
|
||||
### 5. Bootstrap the deploy user
|
||||
|
||||
```bash
|
||||
bash scripts/run-bootstrap.sh
|
||||
```
|
||||
|
||||
The script runs the following steps in order:
|
||||
|
||||
**Preflight checks (local)**
|
||||
1. Verifies all four key files exist under `keys/` (both root and deploy key pairs)
|
||||
2. Verifies `scripts/bootstrap-deploy-user.sh` exists
|
||||
3. Sets `chmod 600` on the private key files (SSH refuses keys with open permissions)
|
||||
4. Runs `scripts/check-vault.sh` — decrypts the vault and confirms all 11 required
|
||||
secrets are present and non-empty
|
||||
|
||||
**Remote actions**
|
||||
5. Opens a test SSH connection as `root` to confirm the root key works
|
||||
6. Copies `bootstrap-deploy-user.sh` to `/root/` on the server via `scp`
|
||||
7. Executes it as root — creates the `deploy` user, grants passwordless sudo,
|
||||
and installs `keys/notroot_cloud_ladkau_de.pub` as the only authorized key
|
||||
8. Opens a test SSH connection as `deploy` to confirm the new user can log in
|
||||
|
||||
If any step fails the script exits immediately with a descriptive error message.
|
||||
|
||||
### 6. Run the Ansible master playbook
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/inventory.ini ansible/site.yml --ask-vault-pass
|
||||
```
|
||||
|
||||
This runs all roles in order:
|
||||
|
||||
| # | Role | What it does |
|
||||
|---|------------|--------------|
|
||||
| 1 | `base` | OS hardening, deploy user, SSH config, ufw firewall, fail2ban |
|
||||
| 2 | `docker` | Docker Engine + Compose plugin, shared Traefik network |
|
||||
| 3 | `traefik` | Reverse proxy, automatic TLS via Let's Encrypt |
|
||||
| 4 | `gitea` | Self-hosted Git with PostgreSQL |
|
||||
| 5 | `nextcloud`| File storage with PostgreSQL, Redis, cron sidecar |
|
||||
| 6 | `sso` | Keycloak single sign-on with PostgreSQL |
|
||||
| 7 | `mail` | Roundcube webmail client with PostgreSQL |
|
||||
| 8 | `registry` | Docker Registry v2 with htpasswd auth |
|
||||
| 9 | `k8s` | Placeholder page at k8s.ladkau.de |
|
||||
|
||||
To apply a single role:
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags <role> --ask-vault-pass
|
||||
```
|
||||
|
||||
## Scripts reference
|
||||
|
||||
| Script | Purpose |
|
||||
|--------|---------|
|
||||
| `scripts/run-bootstrap.sh` | Copy and run the deploy user bootstrap on a fresh server |
|
||||
| `scripts/bootstrap-deploy-user.sh` | Runs on the server as root — creates the deploy user |
|
||||
| `scripts/check-vault.sh` | Verify vault.yml exists and all required secrets are non-empty |
|
||||
Reference in New Issue
Block a user