Fix Dovecot passwd-file format and document mbox mail import

- passwd.j2: append '::::::' so entries have full 8-field passwd format.
  Without this Dovecot's userdb lookup fails ('user is missing userdb
  info') even though passdb auth succeeds, causing login to return 401.
- main.yml: fix passwd file mode from 0640 to 0644 so Dovecot can read it.
- Add runbook-mail-import.md: full procedure for migrating mbox-format mail
  (old Dovecot server with ~/mail/ flat files) to Maildir++ including the
  conversion script that handles fetchmail >From envelope lines.
- runbook-configuration.md: reference new runbook in the mail import section.

ansible/roles/mail/tasks/main.yml

@@ -44,7 +44,7 @@
     dest: "{{ mail_data_dir }}/passwd"
     owner: root
     group: root
-    mode: "0640"
+    mode: "0644"
   notify: Restart mail
   tags: mail
 

ansible/roles/mail/templates/passwd.j2

@@ -1,4 +1,4 @@
 # Managed by Ansible — do not edit manually
 {% for user in dovecot_users | default([]) %}
-{{ user.username }}:{SHA512-CRYPT}{{ user.password | password_hash('sha512', (user.username | hash('md5'))[:16]) }}
+{{ user.username }}:{SHA512-CRYPT}{{ user.password | password_hash('sha512', (user.username | hash('md5'))[:16]) }}::::::
 {% endfor %}

docs/runbook-configuration.md

@@ -183,17 +183,20 @@ ansible-playbook -i ansible/inventory.ini ansible/site.yml --tags mail --ask-vau
 
 ### 5.1 Import mail from an old Dovecot server
 
-If migrating from an existing Dovecot server, sync the Maildir directly:
+If the old server uses **Maildir** format, rsync directly:
 
 ```bash
 # Run on the old server — syncs to the new server's maildir volume
-rsync -av --progress /var/mail/alice/Maildir/ \
+rsync -av -e "ssh -i keys/root_cloud_ladkau_de" --progress ./* \
   root@217.154.207.148:/opt/mail/maildir/alice/Maildir/
 
 # Fix ownership for Dovecot's vmail user (UID 5000)
-ssh root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice"
+ssh -i keys/root_cloud_ladkau_de root@217.154.207.148 "chown -R 5000:5000 /opt/mail/maildir/alice"
 ```
 
+If the old server uses **mbox** format (e.g. `mail_location = mbox:~/mail:...`),
+see `runbook-mail-import.md` for the conversion procedure.
+
 Alternatively, use `imapsync` to copy via IMAP from the old server to the
 new one (works without direct server access):
 

docs/runbook-mail-import.md

@@ -0,0 +1,286 @@
+# Mail Import Runbook
+
+How to migrate existing mail from a Dovecot mbox-format server into the
+Maildir++ format used by the mail role on cloud.ladkau.de.
+
+## Background
+
+The old server stores mail in mbox format:
+
+```
+mail_location = mbox:~/mail:INBOX=/var/mail/%u
+```
+
+Messages live in flat files under `~/mail/` with a dot-separated naming
+convention (e.g. `companies.amazon`, `Archive.uk.friends.zoe`). The INBOX
+lives at `/var/mail/<username>`.
+
+The new server uses Maildir++:
+
+```
+mail_location = maildir:/var/mail/%u/Maildir
+```
+
+Folders are stored as dot-prefixed subdirectories of `~/<user>/Maildir/`
+(e.g. `.companies.amazon/`, `.Archive.uk.friends.zoe/`), each containing
+`cur/`, `new/`, and `tmp/` subdirectories.
+
+## Format quirks to handle
+
+**`>From ` escaping.** mbox format escapes lines beginning with `From ` in
+message bodies by prepending `>`. Python's `mailbox.mbox` does not
+automatically undo this when reading. The conversion script must strip the
+leading `>` from any `>From ` line in the message body.
+
+**Fetchmail envelope lines.** When fetchmail delivered mail to the old mbox
+store it prepended a `>From fetchmail  <date>` line before the RFC 2822
+headers. This line is not a valid RFC 2822 header, so Python's email parser
+fails to parse any headers from the message and treats the entire content
+as a body. The result in Maildir is a message that starts with a blank line
+(the header/body separator for the empty headers section) followed by
+`>From fetchmail...` followed by the actual RFC 2822 headers inside the body.
+IMAP clients see such messages as having no Subject, no From, and no Date.
+The conversion script must detect and strip this envelope line before writing
+the Maildir file.
+
+**doveadm backup/import cannot be used** because it requires the source and
+destination hierarchy separator to match. mbox defaults to `/`; Maildir++
+uses `.`. Attempts to override this cause Dovecot to rename folders with
+underscores instead of dots.
+
+## Step 1 — Copy mbox files to the staging area
+
+On the old server, the mail files live in `~/mail/`. Copy them to the staging
+directory on the new server. The Dovecot container mounts
+`/opt/mail/maildir` as `/var/mail`, so the staging directory is accessible
+from the host at `/opt/mail/maildir/mbox_<user>/`.
+
+```bash
+# Run from the old server (or adjust rsync source accordingly)
+rsync -av -e 'ssh -i /path/to/key' \
+  ~/mail/ \
+  root@217.154.207.148:/opt/mail/maildir/mbox_ml/
+```
+
+The `INBOX` mbox file lives at `/var/mail/<user>` on the old server, not
+inside `~/mail/`. Copy it separately if you need it:
+
+```bash
+scp -i /path/to/key /var/mail/ml \
+  root@217.154.207.148:/opt/mail/maildir/mbox_ml/INBOX
+```
+
+After copying, the staging directory should contain flat mbox files with
+dot-separated names:
+
+```
+/opt/mail/maildir/mbox_ml/
+  Archive.de.friends.heiko
+  Archive.uk.ex-family.masha
+  companies.amazon
+  Sent
+  Trash
+  ...
+```
+
+Fix ownership so Dovecot (UID/GID 5000) can read the files:
+
+```bash
+sudo chown -R 5000:5000 /opt/mail/maildir/mbox_ml/
+```
+
+## Step 2 — Run the conversion script
+
+Save the script below to the server and run it as root. It reads every mbox
+file in the staging directory and writes a Maildir++ folder for each one
+under `/opt/mail/maildir/<user>/Maildir/`.
+
+```bash
+sudo python3 /opt/mail/convert_mbox.py
+```
+
+### Conversion script
+
+```python
+#!/usr/bin/env python3
+"""Convert flat dot-separated mbox files to Maildir++ folders.
+
+Handles two common corruption patterns:
+1. From_ envelope line leaked as first line of message.
+2. Fetchmail >From envelope line before RFC 2822 headers — Python's email
+   parser cannot parse headers from such messages and treats all content as
+   body, producing an empty headers section followed by >From in the body.
+"""
+import mailbox, os, time, socket
+
+# Adjust these paths for the target user.
+MBOX_BASE   = '/opt/mail/maildir/mbox_ml'
+MAILDIR_BASE = '/opt/mail/maildir/ml/Maildir'
+
+# mbox files to skip (not real mail folders).
+SKIP = {'INBOX', 'subscriptions', 'maildirfolder'}
+
+
+def ensure_maildir_folder(folder_name):
+    path = os.path.join(MAILDIR_BASE, folder_name)
+    for sub in ('cur', 'new', 'tmp'):
+        os.makedirs(os.path.join(path, sub), exist_ok=True)
+    return path
+
+
+def strip_envelope_line(raw):
+    """Strip a From_/>From_ envelope line, plus any leading blank lines."""
+    stripped = raw.lstrip(b'\r\n')
+    if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
+        nl = stripped.find(b'\n')
+        return stripped[nl + 1:] if nl >= 0 else stripped
+    return raw
+
+
+def msg_to_bytes(msg):
+    # unixfrom=False explicitly excludes the From_ separator line.
+    raw = msg.as_bytes(unixfrom=False)
+
+    # Case 1: From_ leaked as the very first line.
+    if raw.startswith(b'From ') or raw.startswith(b'>From '):
+        raw = strip_envelope_line(raw)
+
+    # Case 2: empty-header message — fetchmail wrote a '>From fetchmail ...'
+    # envelope line before the RFC 2822 headers. Python could not parse
+    # headers so as_bytes() produces '\n>From fetchmail...\nReturn-Path:...'.
+    # Strip the leading blank line and the envelope line.
+    elif raw.startswith(b'\n') or raw.startswith(b'\r\n'):
+        stripped = raw.lstrip(b'\r\n')
+        if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
+            nl = stripped.find(b'\n')
+            raw = stripped[nl + 1:] if nl >= 0 else stripped
+
+    # Unescape mbox >From escaping in the message body (>From -> From).
+    lines = raw.split(b'\n')
+    lines = [line[1:] if line.startswith(b'>From ') else line for line in lines]
+    return b'\n'.join(lines)
+
+
+total_msgs = 0
+new_folders = []
+hostname = socket.gethostname()
+
+for fname in sorted(os.listdir(MBOX_BASE)):
+    if fname in SKIP or fname.startswith('.'):
+        continue
+    fpath = os.path.join(MBOX_BASE, fname)
+    if not os.path.isfile(fpath):
+        continue
+    # Skip non-mbox files (e.g. Dovecot index files accidentally copied over).
+    try:
+        with open(fpath, 'rb') as f:
+            if f.read(5) != b'From ':
+                continue
+    except Exception:
+        continue
+
+    folder_name = '.' + fname   # Maildir++ requires a leading dot.
+    folder_path = ensure_maildir_folder(folder_name)
+    cur_dir = os.path.join(folder_path, 'cur')
+
+    mbox = mailbox.mbox(fpath)
+    count = 0
+    for i, msg in enumerate(mbox):
+        ts = int(time.time())
+        unique = f"{ts}.P{os.getpid()}Q{i}.{hostname}"
+        dest = os.path.join(cur_dir, f"{unique}:2,S")
+        with open(dest, 'wb') as f:
+            f.write(msg_to_bytes(msg))
+        count += 1
+    mbox.close()
+    print(f"  {folder_name:55s} {count:5d} msgs")
+    total_msgs += count
+    new_folders.append(fname)   # IMAP name without leading dot.
+
+# Write a subscriptions file so IMAP clients see all folders.
+subs_path = os.path.join(MAILDIR_BASE, 'subscriptions')
+with open(subs_path, 'w') as f:
+    f.write('V\t2\n')
+    for s in sorted(new_folders):
+        f.write(s + '\n')
+
+print(f"\nDone: {total_msgs} messages in {len(new_folders)} folders.")
+```
+
+## Step 3 — Fix ownership and restart Dovecot
+
+```bash
+# Ensure Dovecot (UID/GID 5000) owns everything.
+sudo chown -R 5000:5000 /opt/mail/maildir/ml
+
+# Restart Dovecot to clear any cached folder lists.
+docker restart dovecot
+```
+
+## Step 4 — Clear Roundcube caches
+
+Roundcube caches the IMAP folder list and message indices in PostgreSQL. If
+you have already logged in before the migration, stale cache entries will
+prevent the new folders from appearing.
+
+```bash
+docker exec mail-db psql -U roundcube -d roundcube \
+  -c 'TRUNCATE cache, cache_index, cache_messages, cache_shared, cache_thread, session;'
+```
+
+## Step 5 — Verify
+
+Log in to Roundcube at https://mail.ladkau.de. The full folder tree should be
+visible. Spot-check a few messages in Trash (the largest folder and the one
+most affected by the fetchmail envelope issue) to confirm Subject, From, and
+Date are populated correctly.
+
+To verify programmatically that no messages have the empty-header corruption:
+
+```bash
+sudo python3 - <<'EOF'
+import os, random
+
+maildir = '/opt/mail/maildir/ml/Maildir'
+bad = good = 0
+for folder in os.listdir(maildir):
+    cur = os.path.join(maildir, folder, 'cur')
+    if not os.path.isdir(cur):
+        continue
+    files = os.listdir(cur)
+    for f in random.sample(files, min(10, len(files))):
+        with open(os.path.join(cur, f), 'rb') as fh:
+            start = fh.read(6)
+        if start.startswith(b'\n') or start.startswith(b'From ') or start.startswith(b'>From'):
+            bad += 1
+        else:
+            good += 1
+
+print(f'Good: {good}, Bad: {bad}')
+EOF
+```
+
+All messages should report as Good (bad count = 0).
+
+## Step 6 — Clean up staging area
+
+Once the migration is confirmed working, remove the staging directory:
+
+```bash
+sudo rm -rf /opt/mail/maildir/mbox_ml/
+```
+
+## INBOX
+
+The mbox INBOX (`/var/mail/ml` on the old server) is skipped by the script
+because the Dovecot configuration on the new server stores INBOX as the root
+Maildir (`/var/mail/ml/Maildir/`), not as a subfolder. To migrate INBOX
+messages, either:
+
+- Copy the INBOX mbox file to the staging area as a file named `INBOX` and
+  remove `INBOX` from the `SKIP` set — but note the script will create
+  `.INBOX` as a subfolder, not the real INBOX. Then use `doveadm move INBOX`
+  to relocate the messages.
+- Or use `mb2md` or a similar tool specifically designed for INBOX migration.
+
+For most use cases, leaving INBOX empty and starting fresh is acceptable.