server_cloud_ladkau_de/docs/runbook-mail-import.md

Mail Import Runbook

How to migrate existing mail from a Dovecot mbox-format server into the Maildir++ format used by the mail role on cloud.ladkau.de.

Background

The old server stores mail in mbox format:

mail_location = mbox:~/mail:INBOX=/var/mail/%u

Messages live in flat files under ~/mail/ with a dot-separated naming convention (e.g. companies.amazon, Archive.uk.friends.zoe). The INBOX lives at /var/mail/<username>.

The new server uses Maildir++:

mail_location = maildir:/var/mail/%u/Maildir

Folders are stored as dot-prefixed subdirectories of ~/<user>/Maildir/ (e.g. .companies.amazon/, .Archive.uk.friends.zoe/), each containing cur/, new/, and tmp/ subdirectories.

Format quirks to handle

>From escaping. mbox format escapes lines beginning with From in message bodies by prepending >. Python's mailbox.mbox does not automatically undo this when reading. The conversion script must strip the leading > from any >From line in the message body.

Fetchmail envelope lines. When fetchmail delivered mail to the old mbox store it prepended a >From fetchmail <date> line before the RFC 2822 headers. This line is not a valid RFC 2822 header, so Python's email parser fails to parse any headers from the message and treats the entire content as a body. The result in Maildir is a message that starts with a blank line (the header/body separator for the empty headers section) followed by >From fetchmail... followed by the actual RFC 2822 headers inside the body. IMAP clients see such messages as having no Subject, no From, and no Date. The conversion script must detect and strip this envelope line before writing the Maildir file.

doveadm backup/import cannot be used because it requires the source and destination hierarchy separator to match. mbox defaults to /; Maildir++ uses .. Attempts to override this cause Dovecot to rename folders with underscores instead of dots.

Step 1 — Copy mbox files to the staging area

On the old server, the mail files live in ~/mail/. Copy them to the staging directory on the new server. The Dovecot container mounts /opt/mail/maildir as /var/mail, so the staging directory is accessible from the host at /opt/mail/maildir/mbox_<user>/.

# Run from the old server (or adjust rsync source accordingly)
rsync -av -e 'ssh -i /path/to/key' \
  ~/mail/ \
  root@217.154.207.148:/opt/mail/maildir/mbox_ml/

The INBOX mbox file lives at /var/mail/<user> on the old server, not inside ~/mail/. Copy it separately if you need it:

scp -i /path/to/key /var/mail/ml \
  root@217.154.207.148:/opt/mail/maildir/mbox_ml/INBOX

After copying, the staging directory should contain flat mbox files with dot-separated names:

/opt/mail/maildir/mbox_ml/
  Archive.de.friends.heiko
  Archive.uk.ex-family.masha
  companies.amazon
  Sent
  Trash
  ...

Fix ownership so Dovecot (UID/GID 5000) can read the files:

sudo chown -R 5000:5000 /opt/mail/maildir/mbox_ml/

Step 2 — Run the conversion script

Save the script below to the server and run it as root. It reads every mbox file in the staging directory and writes a Maildir++ folder for each one under /opt/mail/maildir/<user>/Maildir/.

sudo python3 /opt/mail/convert_mbox.py

Conversion script

#!/usr/bin/env python3
"""Convert flat dot-separated mbox files to Maildir++ folders.

Handles two common corruption patterns:
1. From_ envelope line leaked as first line of message.
2. Fetchmail >From envelope line before RFC 2822 headers — Python's email
   parser cannot parse headers from such messages and treats all content as
   body, producing an empty headers section followed by >From in the body.
"""
import mailbox, os, time, socket

# Adjust these paths for the target user.
MBOX_BASE   = '/opt/mail/maildir/mbox_ml'
MAILDIR_BASE = '/opt/mail/maildir/ml/Maildir'

# mbox files to skip (not real mail folders).
SKIP = {'INBOX', 'subscriptions', 'maildirfolder'}


def ensure_maildir_folder(folder_name):
    path = os.path.join(MAILDIR_BASE, folder_name)
    for sub in ('cur', 'new', 'tmp'):
        os.makedirs(os.path.join(path, sub), exist_ok=True)
    return path


def strip_envelope_line(raw):
    """Strip a From_/>From_ envelope line, plus any leading blank lines."""
    stripped = raw.lstrip(b'\r\n')
    if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
        nl = stripped.find(b'\n')
        return stripped[nl + 1:] if nl >= 0 else stripped
    return raw


def msg_to_bytes(msg):
    # unixfrom=False explicitly excludes the From_ separator line.
    raw = msg.as_bytes(unixfrom=False)

    # Case 1: From_ leaked as the very first line.
    if raw.startswith(b'From ') or raw.startswith(b'>From '):
        raw = strip_envelope_line(raw)

    # Case 2: empty-header message — fetchmail wrote a '>From fetchmail ...'
    # envelope line before the RFC 2822 headers. Python could not parse
    # headers so as_bytes() produces '\n>From fetchmail...\nReturn-Path:...'.
    # Strip the leading blank line and the envelope line.
    elif raw.startswith(b'\n') or raw.startswith(b'\r\n'):
        stripped = raw.lstrip(b'\r\n')
        if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
            nl = stripped.find(b'\n')
            raw = stripped[nl + 1:] if nl >= 0 else stripped

    # Unescape mbox >From escaping in the message body (>From -> From).
    lines = raw.split(b'\n')
    lines = [line[1:] if line.startswith(b'>From ') else line for line in lines]
    return b'\n'.join(lines)


total_msgs = 0
new_folders = []
hostname = socket.gethostname()

for fname in sorted(os.listdir(MBOX_BASE)):
    if fname in SKIP or fname.startswith('.'):
        continue
    fpath = os.path.join(MBOX_BASE, fname)
    if not os.path.isfile(fpath):
        continue
    # Skip non-mbox files (e.g. Dovecot index files accidentally copied over).
    try:
        with open(fpath, 'rb') as f:
            if f.read(5) != b'From ':
                continue
    except Exception:
        continue

    folder_name = '.' + fname   # Maildir++ requires a leading dot.
    folder_path = ensure_maildir_folder(folder_name)
    cur_dir = os.path.join(folder_path, 'cur')

    mbox = mailbox.mbox(fpath)
    count = 0
    for i, msg in enumerate(mbox):
        ts = int(time.time())
        unique = f"{ts}.P{os.getpid()}Q{i}.{hostname}"
        dest = os.path.join(cur_dir, f"{unique}:2,S")
        with open(dest, 'wb') as f:
            f.write(msg_to_bytes(msg))
        count += 1
    mbox.close()
    print(f"  {folder_name:55s} {count:5d} msgs")
    total_msgs += count
    new_folders.append(fname)   # IMAP name without leading dot.

# Write a subscriptions file so IMAP clients see all folders.
subs_path = os.path.join(MAILDIR_BASE, 'subscriptions')
with open(subs_path, 'w') as f:
    f.write('V\t2\n')
    for s in sorted(new_folders):
        f.write(s + '\n')

print(f"\nDone: {total_msgs} messages in {len(new_folders)} folders.")

Step 3 — Fix ownership and restart Dovecot

# Ensure Dovecot (UID/GID 5000) owns everything.
sudo chown -R 5000:5000 /opt/mail/maildir/ml

# Restart Dovecot to clear any cached folder lists.
docker restart dovecot

Step 4 — Clear Roundcube caches

Roundcube caches the IMAP folder list and message indices in PostgreSQL. If you have already logged in before the migration, stale cache entries will prevent the new folders from appearing.

docker exec mail-db psql -U roundcube -d roundcube \
  -c 'TRUNCATE cache, cache_index, cache_messages, cache_shared, cache_thread, session;'

Step 5 — Verify

Log in to Roundcube at https://mail.ladkau.de. The full folder tree should be visible. Spot-check a few messages in Trash (the largest folder and the one most affected by the fetchmail envelope issue) to confirm Subject, From, and Date are populated correctly.

To verify programmatically that no messages have the empty-header corruption:

sudo python3 - <<'EOF'
import os, random

maildir = '/opt/mail/maildir/ml/Maildir'
bad = good = 0
for folder in os.listdir(maildir):
    cur = os.path.join(maildir, folder, 'cur')
    if not os.path.isdir(cur):
        continue
    files = os.listdir(cur)
    for f in random.sample(files, min(10, len(files))):
        with open(os.path.join(cur, f), 'rb') as fh:
            start = fh.read(6)
        if start.startswith(b'\n') or start.startswith(b'From ') or start.startswith(b'>From'):
            bad += 1
        else:
            good += 1

print(f'Good: {good}, Bad: {bad}')
EOF

All messages should report as Good (bad count = 0).

Step 6 — Clean up staging area

Once the migration is confirmed working, remove the staging directory:

sudo rm -rf /opt/mail/maildir/mbox_ml/

INBOX

The mbox INBOX (/var/mail/ml on the old server) is skipped by the script because the Dovecot configuration on the new server stores INBOX as the root Maildir (/var/mail/ml/Maildir/), not as a subfolder. To migrate INBOX messages, either:

• Copy the INBOX mbox file to the staging area as a file named INBOX and remove INBOX from the SKIP set — but note the script will create .INBOX as a subfolder, not the real INBOX. Then use doveadm move INBOX to relocate the messages.
• Or use mb2md or a similar tool specifically designed for INBOX migration.

For most use cases, leaving INBOX empty and starting fresh is acceptable.