From 7c690be891b6571b7045d5b674c21956f2a7bed3 Mon Sep 17 00:00:00 2001 From: ml Date: Mon, 29 Jun 2026 17:14:01 +0200 Subject: [PATCH] Fix Dovecot passwd-file format and document mbox mail import - passwd.j2: append '::::::' so entries have full 8-field passwd format. Without this Dovecot's userdb lookup fails ('user is missing userdb info') even though passdb auth succeeds, causing login to return 401. - main.yml: fix passwd file mode from 0640 to 0644 so Dovecot can read it. - Add runbook-mail-import.md: full procedure for migrating mbox-format mail (old Dovecot server with ~/mail/ flat files) to Maildir++ including the conversion script that handles fetchmail >From envelope lines. - runbook-configuration.md: reference new runbook in the mail import section. --- ansible/roles/mail/tasks/main.yml | 2 +- ansible/roles/mail/templates/passwd.j2 | 2 +- docs/runbook-configuration.md | 9 +- docs/runbook-mail-import.md | 286 +++++++++++++++++++++++++ 4 files changed, 294 insertions(+), 5 deletions(-) create mode 100644 docs/runbook-mail-import.md diff --git a/ansible/roles/mail/tasks/main.yml b/ansible/roles/mail/tasks/main.yml index ad8d751..dcde5f7 100644 --- a/ansible/roles/mail/tasks/main.yml +++ b/ansible/roles/mail/tasks/main.yml @@ -44,7 +44,7 @@ dest: "{{ mail_data_dir }}/passwd" owner: root group: root - mode: "0640" + mode: "0644" notify: Restart mail tags: mail diff --git a/ansible/roles/mail/templates/passwd.j2 b/ansible/roles/mail/templates/passwd.j2 index f36d438..3bd5621 100644 --- a/ansible/roles/mail/templates/passwd.j2 +++ b/ansible/roles/mail/templates/passwd.j2 @@ -1,4 +1,4 @@ # Managed by Ansible — do not edit manually {% for user in dovecot_users | default([]) %} -{{ user.username }}:{SHA512-CRYPT}{{ user.password | password_hash('sha512', (user.username | hash('md5'))[:16]) }} +{{ user.username }}:{SHA512-CRYPT}{{ user.password | password_hash('sha512', (user.username | hash('md5'))[:16]) }}:::::: {% endfor %} diff --git a/docs/runbook-configuration.md b/docs/runbook-configuration.md index 8b6feb2..7becc0c 100644 --- a/docs/runbook-configuration.md +++ b/docs/runbook-configuration.md @@ -183,17 +183,20 @@ ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags mail --ask-vau ### 5.1 Import mail from an old Dovecot server -If migrating from an existing Dovecot server, sync the Maildir directly: +If the old server uses **Maildir** format, rsync directly: ```bash # Run on the old server — syncs to the new server's maildir volume -rsync -av --progress /var/mail/alice/Maildir/ \ +rsync -av -e "ssh -i keys/root_cloud_ladkau_de" --progress ./* \ root@217.154.207.148:/opt/mail/maildir/alice/Maildir/ # Fix ownership for Dovecot's vmail user (UID 5000) -ssh root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice" +ssh -i keys/root_cloud_ladkau_de root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice" ``` +If the old server uses **mbox** format (e.g. `mail_location = mbox:~/mail:...`), +see `runbook-mail-import.md` for the conversion procedure. + Alternatively, use `imapsync` to copy via IMAP from the old server to the new one (works without direct server access): diff --git a/docs/runbook-mail-import.md b/docs/runbook-mail-import.md new file mode 100644 index 0000000..51fc820 --- /dev/null +++ b/docs/runbook-mail-import.md @@ -0,0 +1,286 @@ +# Mail Import Runbook + +How to migrate existing mail from a Dovecot mbox-format server into the +Maildir++ format used by the mail role on cloud.ladkau.de. + +## Background + +The old server stores mail in mbox format: + +``` +mail_location = mbox:~/mail:INBOX=/var/mail/%u +``` + +Messages live in flat files under `~/mail/` with a dot-separated naming +convention (e.g. `companies.amazon`, `Archive.uk.friends.zoe`). The INBOX +lives at `/var/mail/`. + +The new server uses Maildir++: + +``` +mail_location = maildir:/var/mail/%u/Maildir +``` + +Folders are stored as dot-prefixed subdirectories of `~//Maildir/` +(e.g. `.companies.amazon/`, `.Archive.uk.friends.zoe/`), each containing +`cur/`, `new/`, and `tmp/` subdirectories. + +## Format quirks to handle + +**`>From ` escaping.** mbox format escapes lines beginning with `From ` in +message bodies by prepending `>`. Python's `mailbox.mbox` does not +automatically undo this when reading. The conversion script must strip the +leading `>` from any `>From ` line in the message body. + +**Fetchmail envelope lines.** When fetchmail delivered mail to the old mbox +store it prepended a `>From fetchmail ` line before the RFC 2822 +headers. This line is not a valid RFC 2822 header, so Python's email parser +fails to parse any headers from the message and treats the entire content +as a body. The result in Maildir is a message that starts with a blank line +(the header/body separator for the empty headers section) followed by +`>From fetchmail...` followed by the actual RFC 2822 headers inside the body. +IMAP clients see such messages as having no Subject, no From, and no Date. +The conversion script must detect and strip this envelope line before writing +the Maildir file. + +**doveadm backup/import cannot be used** because it requires the source and +destination hierarchy separator to match. mbox defaults to `/`; Maildir++ +uses `.`. Attempts to override this cause Dovecot to rename folders with +underscores instead of dots. + +## Step 1 — Copy mbox files to the staging area + +On the old server, the mail files live in `~/mail/`. Copy them to the staging +directory on the new server. The Dovecot container mounts +`/opt/mail/maildir` as `/var/mail`, so the staging directory is accessible +from the host at `/opt/mail/maildir/mbox_/`. + +```bash +# Run from the old server (or adjust rsync source accordingly) +rsync -av -e 'ssh -i /path/to/key' \ + ~/mail/ \ + root@217.154.207.148:/opt/mail/maildir/mbox_ml/ +``` + +The `INBOX` mbox file lives at `/var/mail/` on the old server, not +inside `~/mail/`. Copy it separately if you need it: + +```bash +scp -i /path/to/key /var/mail/ml \ + root@217.154.207.148:/opt/mail/maildir/mbox_ml/INBOX +``` + +After copying, the staging directory should contain flat mbox files with +dot-separated names: + +``` +/opt/mail/maildir/mbox_ml/ + Archive.de.friends.heiko + Archive.uk.ex-family.masha + companies.amazon + Sent + Trash + ... +``` + +Fix ownership so Dovecot (UID/GID 5000) can read the files: + +```bash +sudo chown -R 5000:5000 /opt/mail/maildir/mbox_ml/ +``` + +## Step 2 — Run the conversion script + +Save the script below to the server and run it as root. It reads every mbox +file in the staging directory and writes a Maildir++ folder for each one +under `/opt/mail/maildir//Maildir/`. + +```bash +sudo python3 /opt/mail/convert_mbox.py +``` + +### Conversion script + +```python +#!/usr/bin/env python3 +"""Convert flat dot-separated mbox files to Maildir++ folders. + +Handles two common corruption patterns: +1. From_ envelope line leaked as first line of message. +2. Fetchmail >From envelope line before RFC 2822 headers — Python's email + parser cannot parse headers from such messages and treats all content as + body, producing an empty headers section followed by >From in the body. +""" +import mailbox, os, time, socket + +# Adjust these paths for the target user. +MBOX_BASE = '/opt/mail/maildir/mbox_ml' +MAILDIR_BASE = '/opt/mail/maildir/ml/Maildir' + +# mbox files to skip (not real mail folders). +SKIP = {'INBOX', 'subscriptions', 'maildirfolder'} + + +def ensure_maildir_folder(folder_name): + path = os.path.join(MAILDIR_BASE, folder_name) + for sub in ('cur', 'new', 'tmp'): + os.makedirs(os.path.join(path, sub), exist_ok=True) + return path + + +def strip_envelope_line(raw): + """Strip a From_/>From_ envelope line, plus any leading blank lines.""" + stripped = raw.lstrip(b'\r\n') + if stripped.startswith(b'From ') or stripped.startswith(b'>From '): + nl = stripped.find(b'\n') + return stripped[nl + 1:] if nl >= 0 else stripped + return raw + + +def msg_to_bytes(msg): + # unixfrom=False explicitly excludes the From_ separator line. + raw = msg.as_bytes(unixfrom=False) + + # Case 1: From_ leaked as the very first line. + if raw.startswith(b'From ') or raw.startswith(b'>From '): + raw = strip_envelope_line(raw) + + # Case 2: empty-header message — fetchmail wrote a '>From fetchmail ...' + # envelope line before the RFC 2822 headers. Python could not parse + # headers so as_bytes() produces '\n>From fetchmail...\nReturn-Path:...'. + # Strip the leading blank line and the envelope line. + elif raw.startswith(b'\n') or raw.startswith(b'\r\n'): + stripped = raw.lstrip(b'\r\n') + if stripped.startswith(b'From ') or stripped.startswith(b'>From '): + nl = stripped.find(b'\n') + raw = stripped[nl + 1:] if nl >= 0 else stripped + + # Unescape mbox >From escaping in the message body (>From -> From). + lines = raw.split(b'\n') + lines = [line[1:] if line.startswith(b'>From ') else line for line in lines] + return b'\n'.join(lines) + + +total_msgs = 0 +new_folders = [] +hostname = socket.gethostname() + +for fname in sorted(os.listdir(MBOX_BASE)): + if fname in SKIP or fname.startswith('.'): + continue + fpath = os.path.join(MBOX_BASE, fname) + if not os.path.isfile(fpath): + continue + # Skip non-mbox files (e.g. Dovecot index files accidentally copied over). + try: + with open(fpath, 'rb') as f: + if f.read(5) != b'From ': + continue + except Exception: + continue + + folder_name = '.' + fname # Maildir++ requires a leading dot. + folder_path = ensure_maildir_folder(folder_name) + cur_dir = os.path.join(folder_path, 'cur') + + mbox = mailbox.mbox(fpath) + count = 0 + for i, msg in enumerate(mbox): + ts = int(time.time()) + unique = f"{ts}.P{os.getpid()}Q{i}.{hostname}" + dest = os.path.join(cur_dir, f"{unique}:2,S") + with open(dest, 'wb') as f: + f.write(msg_to_bytes(msg)) + count += 1 + mbox.close() + print(f" {folder_name:55s} {count:5d} msgs") + total_msgs += count + new_folders.append(fname) # IMAP name without leading dot. + +# Write a subscriptions file so IMAP clients see all folders. +subs_path = os.path.join(MAILDIR_BASE, 'subscriptions') +with open(subs_path, 'w') as f: + f.write('V\t2\n') + for s in sorted(new_folders): + f.write(s + '\n') + +print(f"\nDone: {total_msgs} messages in {len(new_folders)} folders.") +``` + +## Step 3 — Fix ownership and restart Dovecot + +```bash +# Ensure Dovecot (UID/GID 5000) owns everything. +sudo chown -R 5000:5000 /opt/mail/maildir/ml + +# Restart Dovecot to clear any cached folder lists. +docker restart dovecot +``` + +## Step 4 — Clear Roundcube caches + +Roundcube caches the IMAP folder list and message indices in PostgreSQL. If +you have already logged in before the migration, stale cache entries will +prevent the new folders from appearing. + +```bash +docker exec mail-db psql -U roundcube -d roundcube \ + -c 'TRUNCATE cache, cache_index, cache_messages, cache_shared, cache_thread, session;' +``` + +## Step 5 — Verify + +Log in to Roundcube at https://mail.ladkau.de. The full folder tree should be +visible. Spot-check a few messages in Trash (the largest folder and the one +most affected by the fetchmail envelope issue) to confirm Subject, From, and +Date are populated correctly. + +To verify programmatically that no messages have the empty-header corruption: + +```bash +sudo python3 - <<'EOF' +import os, random + +maildir = '/opt/mail/maildir/ml/Maildir' +bad = good = 0 +for folder in os.listdir(maildir): + cur = os.path.join(maildir, folder, 'cur') + if not os.path.isdir(cur): + continue + files = os.listdir(cur) + for f in random.sample(files, min(10, len(files))): + with open(os.path.join(cur, f), 'rb') as fh: + start = fh.read(6) + if start.startswith(b'\n') or start.startswith(b'From ') or start.startswith(b'>From'): + bad += 1 + else: + good += 1 + +print(f'Good: {good}, Bad: {bad}') +EOF +``` + +All messages should report as Good (bad count = 0). + +## Step 6 — Clean up staging area + +Once the migration is confirmed working, remove the staging directory: + +```bash +sudo rm -rf /opt/mail/maildir/mbox_ml/ +``` + +## INBOX + +The mbox INBOX (`/var/mail/ml` on the old server) is skipped by the script +because the Dovecot configuration on the new server stores INBOX as the root +Maildir (`/var/mail/ml/Maildir/`), not as a subfolder. To migrate INBOX +messages, either: + +- Copy the INBOX mbox file to the staging area as a file named `INBOX` and + remove `INBOX` from the `SKIP` set — but note the script will create + `.INBOX` as a subfolder, not the real INBOX. Then use `doveadm move INBOX` + to relocate the messages. +- Or use `mb2md` or a similar tool specifically designed for INBOX migration. + +For most use cases, leaving INBOX empty and starting fresh is acceptable.