server_cloud_ladkau_de/docs/runbook-mail-import.md

# Mail Import Runbook

How to migrate existing mail from a Dovecot mbox-format server into the
Maildir++ format used by the mail role on cloud.ladkau.de.

## Background

The old server stores mail in mbox format:

```
mail_location = mbox:~/mail:INBOX=/var/mail/%u
```

Messages live in flat files under `~/mail/` with a dot-separated naming
convention (e.g. `companies.amazon`, `Archive.uk.friends.zoe`). The INBOX
lives at `/var/mail/<username>`.

The new server uses Maildir++:

```
mail_location = maildir:/var/mail/%u/Maildir
```

Folders are stored as dot-prefixed subdirectories of `~/<user>/Maildir/`
(e.g. `.companies.amazon/`, `.Archive.uk.friends.zoe/`), each containing
`cur/`, `new/`, and `tmp/` subdirectories.

## Format quirks to handle

**`>From ` escaping.** mbox format escapes lines beginning with `From ` in
message bodies by prepending `>`. Python's `mailbox.mbox` does not
automatically undo this when reading. The conversion script must strip the
leading `>` from any `>From ` line in the message body.

**Fetchmail envelope lines.** When fetchmail delivered mail to the old mbox
store it prepended a `>From fetchmail  <date>` line before the RFC 2822
headers. This line is not a valid RFC 2822 header, so Python's email parser
fails to parse any headers from the message and treats the entire content
as a body. The result in Maildir is a message that starts with a blank line
(the header/body separator for the empty headers section) followed by
`>From fetchmail...` followed by the actual RFC 2822 headers inside the body.
IMAP clients see such messages as having no Subject, no From, and no Date.
The conversion script must detect and strip this envelope line before writing
the Maildir file.

**doveadm backup/import cannot be used** because it requires the source and
destination hierarchy separator to match. mbox defaults to `/`; Maildir++
uses `.`. Attempts to override this cause Dovecot to rename folders with
underscores instead of dots.

## Step 1 — Copy mbox files to the staging area

On the old server, the mail files live in `~/mail/`. Copy them to the staging
directory on the new server. The Dovecot container mounts
`/opt/mail/maildir` as `/var/mail`, so the staging directory is accessible
from the host at `/opt/mail/maildir/mbox_<user>/`.

```bash
# Run from the old server (or adjust rsync source accordingly)
rsync -av -e 'ssh -i /path/to/key' \
  ~/mail/ \
  root@217.154.207.148:/opt/mail/maildir/mbox_ml/
```

The `INBOX` mbox file lives at `/var/mail/<user>` on the old server, not
inside `~/mail/`. Copy it separately if you need it:

```bash
scp -i /path/to/key /var/mail/ml \
  root@217.154.207.148:/opt/mail/maildir/mbox_ml/INBOX
```

After copying, the staging directory should contain flat mbox files with
dot-separated names:

```
/opt/mail/maildir/mbox_ml/
  Archive.de.friends.heiko
  Archive.uk.ex-family.masha
  companies.amazon
  Sent
  Trash
  ...
```

Fix ownership so Dovecot (UID/GID 5000) can read the files:

```bash
sudo chown -R 5000:5000 /opt/mail/maildir/mbox_ml/
```

## Step 2 — Run the conversion script

Save the script below to the server and run it as root. It reads every mbox
file in the staging directory and writes a Maildir++ folder for each one
under `/opt/mail/maildir/<user>/Maildir/`.

```bash
sudo python3 /opt/mail/convert_mbox.py
```

### Conversion script

```python
#!/usr/bin/env python3
"""Convert flat dot-separated mbox files to Maildir++ folders.

Handles two common corruption patterns:
1. From_ envelope line leaked as first line of message.
2. Fetchmail >From envelope line before RFC 2822 headers — Python's email
   parser cannot parse headers from such messages and treats all content as
   body, producing an empty headers section followed by >From in the body.
"""
import mailbox, os, time, socket

# Adjust these paths for the target user.
MBOX_BASE   = '/opt/mail/maildir/mbox_ml'
MAILDIR_BASE = '/opt/mail/maildir/ml/Maildir'

# mbox files to skip (not real mail folders).
SKIP = {'INBOX', 'subscriptions', 'maildirfolder'}


def ensure_maildir_folder(folder_name):
    path = os.path.join(MAILDIR_BASE, folder_name)
    for sub in ('cur', 'new', 'tmp'):
        os.makedirs(os.path.join(path, sub), exist_ok=True)
    return path


def strip_envelope_line(raw):
    """Strip a From_/>From_ envelope line, plus any leading blank lines."""
    stripped = raw.lstrip(b'\r\n')
    if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
        nl = stripped.find(b'\n')
        return stripped[nl + 1:] if nl >= 0 else stripped
    return raw


def msg_to_bytes(msg):
    # unixfrom=False explicitly excludes the From_ separator line.
    raw = msg.as_bytes(unixfrom=False)

    # Case 1: From_ leaked as the very first line.
    if raw.startswith(b'From ') or raw.startswith(b'>From '):
        raw = strip_envelope_line(raw)

    # Case 2: empty-header message — fetchmail wrote a '>From fetchmail ...'
    # envelope line before the RFC 2822 headers. Python could not parse
    # headers so as_bytes() produces '\n>From fetchmail...\nReturn-Path:...'.
    # Strip the leading blank line and the envelope line.
    elif raw.startswith(b'\n') or raw.startswith(b'\r\n'):
        stripped = raw.lstrip(b'\r\n')
        if stripped.startswith(b'From ') or stripped.startswith(b'>From '):
            nl = stripped.find(b'\n')
            raw = stripped[nl + 1:] if nl >= 0 else stripped

    # Unescape mbox >From escaping in the message body (>From -> From).
    lines = raw.split(b'\n')
    lines = [line[1:] if line.startswith(b'>From ') else line for line in lines]
    return b'\n'.join(lines)


total_msgs = 0
new_folders = []
hostname = socket.gethostname()

for fname in sorted(os.listdir(MBOX_BASE)):
    if fname in SKIP or fname.startswith('.'):
        continue
    fpath = os.path.join(MBOX_BASE, fname)
    if not os.path.isfile(fpath):
        continue
    # Skip non-mbox files (e.g. Dovecot index files accidentally copied over).
    try:
        with open(fpath, 'rb') as f:
            if f.read(5) != b'From ':
                continue
    except Exception:
        continue

    folder_name = '.' + fname   # Maildir++ requires a leading dot.
    folder_path = ensure_maildir_folder(folder_name)
    cur_dir = os.path.join(folder_path, 'cur')

    mbox = mailbox.mbox(fpath)
    count = 0
    for i, msg in enumerate(mbox):
        ts = int(time.time())
        unique = f"{ts}.P{os.getpid()}Q{i}.{hostname}"
        dest = os.path.join(cur_dir, f"{unique}:2,S")
        with open(dest, 'wb') as f:
            f.write(msg_to_bytes(msg))
        count += 1
    mbox.close()
    print(f"  {folder_name:55s} {count:5d} msgs")
    total_msgs += count
    new_folders.append(fname)   # IMAP name without leading dot.

# Write a subscriptions file so IMAP clients see all folders.
subs_path = os.path.join(MAILDIR_BASE, 'subscriptions')
with open(subs_path, 'w') as f:
    f.write('V\t2\n')
    for s in sorted(new_folders):
        f.write(s + '\n')

print(f"\nDone: {total_msgs} messages in {len(new_folders)} folders.")
```

## Step 3 — Fix ownership and restart Dovecot

```bash
# Ensure Dovecot (UID/GID 5000) owns everything.
sudo chown -R 5000:5000 /opt/mail/maildir/ml

# Restart Dovecot to clear any cached folder lists.
docker restart dovecot
```

## Step 4 — Clear Roundcube caches

Roundcube caches the IMAP folder list and message indices in PostgreSQL. If
you have already logged in before the migration, stale cache entries will
prevent the new folders from appearing.

```bash
docker exec mail-db psql -U roundcube -d roundcube \
  -c 'TRUNCATE cache, cache_index, cache_messages, cache_shared, cache_thread, session;'
```

## Step 5 — Verify

Log in to Roundcube at https://mail.ladkau.de. The full folder tree should be
visible. Spot-check a few messages in Trash (the largest folder and the one
most affected by the fetchmail envelope issue) to confirm Subject, From, and
Date are populated correctly.

To verify programmatically that no messages have the empty-header corruption:

```bash
sudo python3 - <<'EOF'
import os, random

maildir = '/opt/mail/maildir/ml/Maildir'
bad = good = 0
for folder in os.listdir(maildir):
    cur = os.path.join(maildir, folder, 'cur')
    if not os.path.isdir(cur):
        continue
    files = os.listdir(cur)
    for f in random.sample(files, min(10, len(files))):
        with open(os.path.join(cur, f), 'rb') as fh:
            start = fh.read(6)
        if start.startswith(b'\n') or start.startswith(b'From ') or start.startswith(b'>From'):
            bad += 1
        else:
            good += 1

print(f'Good: {good}, Bad: {bad}')
EOF
```

All messages should report as Good (bad count = 0).

## Step 6 — Clean up staging area

Once the migration is confirmed working, remove the staging directory:

```bash
sudo rm -rf /opt/mail/maildir/mbox_ml/
```

## INBOX

The mbox INBOX (`/var/mail/ml` on the old server) is skipped by the script
because the Dovecot configuration on the new server stores INBOX as the root
Maildir (`/var/mail/ml/Maildir/`), not as a subfolder. To migrate INBOX
messages, either:

- Copy the INBOX mbox file to the staging area as a file named `INBOX` and
  remove `INBOX` from the `SKIP` set — but note the script will create
  `.INBOX` as a subfolder, not the real INBOX. Then use `doveadm move INBOX`
  to relocate the messages.
- Or use `mb2md` or a similar tool specifically designed for INBOX migration.

For most use cases, leaving INBOX empty and starting fresh is acceptable.