# Mail Import Runbook How to migrate existing mail from a Dovecot mbox-format server into the Maildir++ format used by the mail role on cloud.ladkau.de. ## Background The old server stores mail in mbox format: ``` mail_location = mbox:~/mail:INBOX=/var/mail/%u ``` Messages live in flat files under `~/mail/` with a dot-separated naming convention (e.g. `companies.amazon`, `Archive.uk.friends.zoe`). The INBOX lives at `/var/mail/`. The new server uses Maildir++: ``` mail_location = maildir:/var/mail/%u/Maildir ``` Folders are stored as dot-prefixed subdirectories of `~//Maildir/` (e.g. `.companies.amazon/`, `.Archive.uk.friends.zoe/`), each containing `cur/`, `new/`, and `tmp/` subdirectories. ## Format quirks to handle **`>From ` escaping.** mbox format escapes lines beginning with `From ` in message bodies by prepending `>`. Python's `mailbox.mbox` does not automatically undo this when reading. The conversion script must strip the leading `>` from any `>From ` line in the message body. **Fetchmail envelope lines.** When fetchmail delivered mail to the old mbox store it prepended a `>From fetchmail ` line before the RFC 2822 headers. This line is not a valid RFC 2822 header, so Python's email parser fails to parse any headers from the message and treats the entire content as a body. The result in Maildir is a message that starts with a blank line (the header/body separator for the empty headers section) followed by `>From fetchmail...` followed by the actual RFC 2822 headers inside the body. IMAP clients see such messages as having no Subject, no From, and no Date. The conversion script must detect and strip this envelope line before writing the Maildir file. **doveadm backup/import cannot be used** because it requires the source and destination hierarchy separator to match. mbox defaults to `/`; Maildir++ uses `.`. Attempts to override this cause Dovecot to rename folders with underscores instead of dots. ## Step 1 — Copy mbox files to the staging area On the old server, the mail files live in `~/mail/`. Copy them to the staging directory on the new server. The Dovecot container mounts `/opt/mail/maildir` as `/var/mail`, so the staging directory is accessible from the host at `/opt/mail/maildir/mbox_/`. ```bash # Run from the old server (or adjust rsync source accordingly) rsync -av -e 'ssh -i /path/to/key' \ ~/mail/ \ root@217.154.207.148:/opt/mail/maildir/mbox_ml/ ``` The `INBOX` mbox file lives at `/var/mail/` on the old server, not inside `~/mail/`. Copy it separately if you need it: ```bash scp -i /path/to/key /var/mail/ml \ root@217.154.207.148:/opt/mail/maildir/mbox_ml/INBOX ``` After copying, the staging directory should contain flat mbox files with dot-separated names: ``` /opt/mail/maildir/mbox_ml/ Archive.de.friends.heiko Archive.uk.ex-family.masha companies.amazon Sent Trash ... ``` Fix ownership so Dovecot (UID/GID 5000) can read the files: ```bash sudo chown -R 5000:5000 /opt/mail/maildir/mbox_ml/ ``` ## Step 2 — Run the conversion script Save the script below to the server and run it as root. It reads every mbox file in the staging directory and writes a Maildir++ folder for each one under `/opt/mail/maildir//Maildir/`. ```bash sudo python3 /opt/mail/convert_mbox.py ``` ### Conversion script ```python #!/usr/bin/env python3 """Convert flat dot-separated mbox files to Maildir++ folders. Handles two common corruption patterns: 1. From_ envelope line leaked as first line of message. 2. Fetchmail >From envelope line before RFC 2822 headers — Python's email parser cannot parse headers from such messages and treats all content as body, producing an empty headers section followed by >From in the body. """ import mailbox, os, time, socket # Adjust these paths for the target user. MBOX_BASE = '/opt/mail/maildir/mbox_ml' MAILDIR_BASE = '/opt/mail/maildir/ml/Maildir' # mbox files to skip (not real mail folders). SKIP = {'INBOX', 'subscriptions', 'maildirfolder'} def ensure_maildir_folder(folder_name): path = os.path.join(MAILDIR_BASE, folder_name) for sub in ('cur', 'new', 'tmp'): os.makedirs(os.path.join(path, sub), exist_ok=True) return path def strip_envelope_line(raw): """Strip a From_/>From_ envelope line, plus any leading blank lines.""" stripped = raw.lstrip(b'\r\n') if stripped.startswith(b'From ') or stripped.startswith(b'>From '): nl = stripped.find(b'\n') return stripped[nl + 1:] if nl >= 0 else stripped return raw def msg_to_bytes(msg): # unixfrom=False explicitly excludes the From_ separator line. raw = msg.as_bytes(unixfrom=False) # Case 1: From_ leaked as the very first line. if raw.startswith(b'From ') or raw.startswith(b'>From '): raw = strip_envelope_line(raw) # Case 2: empty-header message — fetchmail wrote a '>From fetchmail ...' # envelope line before the RFC 2822 headers. Python could not parse # headers so as_bytes() produces '\n>From fetchmail...\nReturn-Path:...'. # Strip the leading blank line and the envelope line. elif raw.startswith(b'\n') or raw.startswith(b'\r\n'): stripped = raw.lstrip(b'\r\n') if stripped.startswith(b'From ') or stripped.startswith(b'>From '): nl = stripped.find(b'\n') raw = stripped[nl + 1:] if nl >= 0 else stripped # Unescape mbox >From escaping in the message body (>From -> From). lines = raw.split(b'\n') lines = [line[1:] if line.startswith(b'>From ') else line for line in lines] return b'\n'.join(lines) total_msgs = 0 new_folders = [] hostname = socket.gethostname() for fname in sorted(os.listdir(MBOX_BASE)): if fname in SKIP or fname.startswith('.'): continue fpath = os.path.join(MBOX_BASE, fname) if not os.path.isfile(fpath): continue # Skip non-mbox files (e.g. Dovecot index files accidentally copied over). try: with open(fpath, 'rb') as f: if f.read(5) != b'From ': continue except Exception: continue folder_name = '.' + fname # Maildir++ requires a leading dot. folder_path = ensure_maildir_folder(folder_name) cur_dir = os.path.join(folder_path, 'cur') mbox = mailbox.mbox(fpath) count = 0 for i, msg in enumerate(mbox): ts = int(time.time()) unique = f"{ts}.P{os.getpid()}Q{i}.{hostname}" dest = os.path.join(cur_dir, f"{unique}:2,S") with open(dest, 'wb') as f: f.write(msg_to_bytes(msg)) count += 1 mbox.close() print(f" {folder_name:55s} {count:5d} msgs") total_msgs += count new_folders.append(fname) # IMAP name without leading dot. # Write a subscriptions file so IMAP clients see all folders. subs_path = os.path.join(MAILDIR_BASE, 'subscriptions') with open(subs_path, 'w') as f: f.write('V\t2\n') for s in sorted(new_folders): f.write(s + '\n') print(f"\nDone: {total_msgs} messages in {len(new_folders)} folders.") ``` ## Step 3 — Fix ownership and restart Dovecot ```bash # Ensure Dovecot (UID/GID 5000) owns everything. sudo chown -R 5000:5000 /opt/mail/maildir/ml # Restart Dovecot to clear any cached folder lists. docker restart dovecot ``` ## Step 4 — Clear Roundcube caches Roundcube caches the IMAP folder list and message indices in PostgreSQL. If you have already logged in before the migration, stale cache entries will prevent the new folders from appearing. ```bash docker exec mail-db psql -U roundcube -d roundcube \ -c 'TRUNCATE cache, cache_index, cache_messages, cache_shared, cache_thread, session;' ``` ## Step 5 — Verify Log in to Roundcube at https://mail.ladkau.de. The full folder tree should be visible. Spot-check a few messages in Trash (the largest folder and the one most affected by the fetchmail envelope issue) to confirm Subject, From, and Date are populated correctly. To verify programmatically that no messages have the empty-header corruption: ```bash sudo python3 - <<'EOF' import os, random maildir = '/opt/mail/maildir/ml/Maildir' bad = good = 0 for folder in os.listdir(maildir): cur = os.path.join(maildir, folder, 'cur') if not os.path.isdir(cur): continue files = os.listdir(cur) for f in random.sample(files, min(10, len(files))): with open(os.path.join(cur, f), 'rb') as fh: start = fh.read(6) if start.startswith(b'\n') or start.startswith(b'From ') or start.startswith(b'>From'): bad += 1 else: good += 1 print(f'Good: {good}, Bad: {bad}') EOF ``` All messages should report as Good (bad count = 0). ## Step 6 — Clean up staging area Once the migration is confirmed working, remove the staging directory: ```bash sudo rm -rf /opt/mail/maildir/mbox_ml/ ``` ## INBOX The mbox INBOX (`/var/mail/ml` on the old server) is skipped by the script because the Dovecot configuration on the new server stores INBOX as the root Maildir (`/var/mail/ml/Maildir/`), not as a subfolder. To migrate INBOX messages, either: - Copy the INBOX mbox file to the staging area as a file named `INBOX` and remove `INBOX` from the `SKIP` set — but note the script will create `.INBOX` as a subfolder, not the real INBOX. Then use `doveadm move INBOX` to relocate the messages. - Or use `mb2md` or a similar tool specifically designed for INBOX migration. For most use cases, leaving INBOX empty and starting fresh is acceptable.