Problems with restored emails from old server (dovecot - zlib plugin used)

heart1010 · December 1, 2023, 1:03pm

I moved to a new server, hestiacp is running already on the new server.
Now I copied over (scp) one backup (from say user1) from old to the new server and restored that backup.

The domain from user1 is still hosted on old provider so I’ve changed on their dashboard all DNS entries with the the ip from old server to my ip from the new server.

Everything looks ok but when I login to roundcube now I can see all the old emails but I can’t see the content of all those emails.

Here is the dovecot log - what I can see there is still the domain from my old hestia domain/server on many mail files, e.g.

still Error: Mailbox INBOX: UID=2115: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701162695.M673499P3941921.*****cp.my_old_hestia_domain.de******

# tail -f /var/log/dovecot.log
Dec 01 13:52:11 imap(info@domain_from_user.de)<645454><31uePHILMOx/AAAB>: Error: Mailbox INBOX: UID=2115: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701162695.M673499P3941921.cp.my_old_hestia_domain.de,S=88498,W=90293:2,S) failed: Cached message size larger than expected (88498 > 11272, box=INBOX, UID=2115) (read reason=mail stream)
Dec 01 13:52:11 imap(info@domain_from_user.de)<645453><1UCePHILJOx/AAAB>: Info: Disconnected: Logged out in=437 out=1346 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Dec 01 13:52:11 imap(info@domain_from_user.de)<645454><31uePHILMOx/AAAB>: Error: Mailbox INBOX: Deleting corrupted cache record uid=2115: UID 2115: Broken physical size in mailbox INBOX: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701162695.M673499P3941921.cp.my_old_hestia_domain.de,S=88498,W=90293:2,S) failed: Cached message size larger than expected (88498 > 11272, box=INBOX, UID=2115)
Dec 01 13:52:11 imap(info@domain_from_user.de)<645454><31uePHILMOx/AAAB>: Error: Mailbox INBOX: UID=2115: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701162695.M673499P3941921.cp.my_old_hestia_domain.de,S=88498,W=90293:2,S) failed: Cached message size larger than expected (88498 > 11272, box=INBOX, UID=2115)
Dec 01 13:52:11 imap(info@domain_from_user.de)<645454><31uePHILMOx/AAAB>: Info: Disconnected: FETCH failed: Mailbox INBOX: UID=2115: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701162695.M673499P3941921.cp.my_old_hestia_domain.de,S=88498,W=90293:2,S) failed: Cached message size larger than expected (88498 > 11272, box=INBOX, UID=2115) in=746 out=82145 deleted=0 expunged=0 trashed=0 hdr_count=129 hdr_bytes=44341 body_count=0 body_bytes=0
Dec 01 13:52:17 imap-login: Info: Login: user=<info@domain_from_user.de>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=645458, secured, session=<uQQAPXILPux/AAAB>
Dec 01 13:52:17 imap(info@domain_from_user.de)<645458><uQQAPXILPux/AAAB>: Error: Mailbox INBOX: UID=2106: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701010887.M883423P3564250.cp.my_old_hestia_domain.de,S=134545,W=137021:2,S) failed: Cached message size larger than expected (134545 > 17684, box=INBOX, UID=2106) (read reason=mail stream)
Dec 01 13:52:17 imap(info@domain_from_user.de)<645458><uQQAPXILPux/AAAB>: Error: Mailbox INBOX: Deleting corrupted cache record uid=2106: UID 2106: Broken physical size in mailbox INBOX: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701010887.M883423P3564250.cp.my_old_hestia_domain.de,S=134545,W=137021:2,S) failed: Cached message size larger than expected (134545 > 17684, box=INBOX, UID=2106)
Dec 01 13:52:17 imap(info@domain_from_user.de)<645458><uQQAPXILPux/AAAB>: Error: Mailbox INBOX: UID=2106: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701010887.M883423P3564250.cp.my_old_hestia_domain.de,S=134545,W=137021:2,S) failed: Cached message size larger than expected (134545 > 17684, box=INBOX, UID=2106)
Dec 01 13:52:17 imap(info@domain_from_user.de)<645458><uQQAPXILPux/AAAB>: Info: Disconnected: FETCH failed: Mailbox INBOX: UID=2106: read(/home/baeckerei/mail/domain_from_user.de/info/cur/1701010887.M883423P3564250.cp.my_old_hestia_domain.de,S=134545,W=137021:2,S) failed: Cached message size larger than expected (134545 > 17684, box=INBOX, UID=2106) in=634 out=2594 deleted=0 expunged=0 trashed=0 hdr_count=1 hdr_bytes=601 body_count=0 body_bytes=0
Dec 01 13:53:11 imap-login: Info: Login: user=<info@domain_from_user.de>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=645465, secured, session=<iHsxQHILvN1/AAAB>
Dec 01 13:53:11 imap(info@domain_from_user.de)<645465><iHsxQHILvN1/AAAB>: Info: Disconnected: Logged out in=211 out=1178 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
^C

I did a diff doveconf -n diff from old server (left) to the new hestia server (right).
Looks like I’ve added the zlib plugin there. So is this the reason I can’t read the old mails?

sahsanu · December 1, 2023, 2:20pm

I don’t know whether adding zlib plugin would fix it, I’ve never used that plugin but I had a similar problem regarding mismatched mail sizes and I fixed it using perl tool maildir-size-fix.pl so you could give it a try.

github.com

dovecot/tools/blob/main/maildir-size-fix.pl

#!/usr/bin/env perl

# v1.1 by Timo Sirainen, public domain

# some maildir filenames created by .. courier? maildrop? .. has file sizes
# that don't match the file's real size. this script finds such files and
# renames them. if they're found from dovecot-uidlist, they're also renamed
# within there, so their IMAP UID doesn't change either. the POP3 UIDL
# is also optionally preserved then.

use File::Basename;
use strict;

my $maildirlock_path = "/usr/local/libexec/dovecot/maildirlock";

# If UIDLs are based on filename and no P<uidl> entry already exist for
# a message, write a P<original filename> entry so it doesn't change when
# renaming a file.
my $preserve_pop3_uidl = 1; # -p / -np
# If filename doesn't already have a S=size, add it.

This file has been truncated. show original

Before executing maildir-size-fix.pl script, create a backup of user mail box.

Note: remember to replace domain_from_user.de by your actual domain

cd /root/
wget https://raw.githubusercontent.com/dovecot/tools/main/maildir-size-fix.pl
perl maildir-size-fix.pl /home/baeckerei/mail/domain_from_user.de/info/

Cheers,
sahsanu

sahsanu · December 1, 2023, 2:26pm

Hi again,

I think that yes, zlib is the issue and you won’t fix it with the script I was talking in my previous post.

heart1010 · December 1, 2023, 3:13pm

So what is the best thing to do now… both servers are still there (old and new one) so:

(old server) Can I simply stop dovecot, remove all zlib plugin occurences in conf file and restart dovecot again. Will this ‘decompress all old emails’? And then do a hestia backup and restore that backup on the new server.
(new server) Set all the zlib plugin settings also on the new server in the hope everything should work then?
Other option(s)?

eris · December 1, 2023, 4:06pm

Try first option 2 … That is probally the easiest method …

heart1010 · December 1, 2023, 4:57pm

I’ve tried to uncompress all those old mail files (XZ format) I’ve restored from old server and the first tests looking good - roundcube shows the content of all the emails now

#!/bin/bash

cd /home/baeckerei/mail/domain_from_user/

IFS=$'\n'
for i in $(find . -type f); do
   if file "$i" | grep XZ >/dev/null;
   then
        echo "$i"" >>> File is xz comporessed >>> decompressing!!!"
        mv "$i" "$i".tar.xz
        unxz "$i".tar.xz
        mv "$i".tar "$i"
    fi
done

system · December 31, 2023, 4:58pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.