Hello,
I have a server with about 125GB of space used only for emails in different accounts.
I’ve been researching how to optimize the available space and I found out that I can add the zlib plugin to dovecot to compress the messages.
In some tests I’ve achieved a 50% saving. Which is very good.
The big problem - I guess - is that the zlib plugin only compresses messages with the S= parameter in the filename and with Hestia the messages we receive without the size.
Why does this happen? Only the messages I send are being compressed, and they have S= in the filename
Here is the configuration I added to each file:
/etc/dovecot/conf.d/15-lda.conf
mail_plugins = $mail_plugins zlib
/etc/dovecot/conf.d/20-imap.conf:
mail_plugins = $mail_plugins zlib quota imap_quota
/etc/dovecot/conf.d/20-pop3.conf:
mail_plugins = $mail_plugins zlib quota
/etc/dovecot/conf.d/90-plugin.conf
plugin {
zlib_save = gz
zlib_save_level = 6
}
Is there anything I can do about the configuration?
Should I try to compress the email files manually, or some other solution to compress the emails?
Since disk space is relatively expensive, I think it would be interesting to implement some solution for this.
Thanks
Example of filenames I’m getting for sent and received mail:
Sent:
1724538681.M484376P15826.myserver.com,S=355,W=366:2,S
Received:
1724538709.M937155P15841.myserver.com:2,S
Hi @danielbr86,
I suppose Exim is doing the local delivery so try this, edit file /etc/exim4/exim4.conf.template and add maildir_tag directive.
local_delivery:
driver = appendfile
maildir_format
maildir_use_size_file
maildir_tag = ,S=${message_size}
user = ${extract{2}{:}{${lookup{$local_part}lsearch{/etc/exim4/domains/${lookup{$domain}dsearch{/etc/exim4/domains/}}/passwd}}}}
group = mail
create_directory
Remember to restart Exim after the change:
systemctl restart exim4
Note: I use dovecot sieve so the delivery is performed by dovecot-lda and it always adds the ,S=size tag to the message’s file names.
Hello, thank you for your help. I will test it.
Do you know if it is possible and easy to add Sieve after installing Hestia?
You are welcome.
Yes, you can execute this script:
/usr/local/hestia/install/upgrade/manual/install_sieve.sh
Just in case, first backup /etc/exim4/ and /etc/dovecot/
Thank you again.
Editing /etc/exim4/exim4.conf.template made the received email filenames now have S=, but as you pointed out my exim4 is sending directly via local_delivery, without going through dovecot-lda. This way the messages are not automatically compressed.
By installing sieve according to your instructions should solve this, right?
No, that is only to add the size tag, as far as I know you can compress the mail adding a directive to exim but I can’t remember which one. Also, dovecot-lda doesn’t compress the mails, at least not by default and I don’t know how to compress them automatically.
I found an old script which I used to zlib compress all my old emails - use it at your own risk!!! If the mails are already compressed etc. you end up with ‘corrupt / not readable emails’!
#!/bin/bash
cd /home/<your_user>/mail/<your_email_domain.com>/
systemctl stop dovecot.service
IFS=$'\n'
for i in $(find . -type f); do
if file "$i" |grep SMTP >/dev/null;
then
zstd -9 "$i"
rm "$i"
mv "$i".zst "$i"
fi
done
systemctl start dovecot.service
systemctl status dovecot.service
my config file /etc/dovecot/conf.d/90-zlib.conf
# https://doc.dovecot.org/settings/plugin/zlib-plugin/#plugin-zlib
# https://doc.dovecot.org/3.0/configuration_manual/mail_compress_plugin/
# https://github.com/styelz/dovecot-maildir-compress
# https://github.com/hestiacp/hestiacp/issues/2165
plugin {
zlib_save_level = 6
zlib_save = zstd
}
protocol lda {
mail_plugins = $mail_plugins zlib
}
protocol imap {
mail_plugins = $mail_plugins zlib imap_zlib
}
protocol pop3 {
mail_plugins = $mail_plugins zlib
}
protocol lmtp {
mail_plugins = $mail_plugins zlib
}
I thought that adding the zlib plugin to /etc/dovecot/conf.d/15-lda.conf would do this function. I’ll do some testing here and bring a feedback.
Thank you. In your case, are new emails received automatically compressed?
I actually already compressed the old emails. I created the script below, it checks if the file is already compressed with gzip and ignores it if that is the case. For those who want to try, please review for your use case and do so at your own risk.
Remember that you need to configure dovecot to read compressed emails as mentioned, otherwise they cannot be read.
compress-emails.sh:
#!/bin/bash
# Init vars
dir_base="/home/*"
# ? function
show_usage() {
echo "Usage: $0 [-u user]"
echo " -u Define base dir (default: /home/*)"
exit 1
}
# process arguments
while getopts ":u:" opt; do
case $opt in
u)
dir_base="/home/$OPTARG"
;;
\?)
show_usage
;;
esac
done
shift $((OPTIND -1))
# set dirs to process
dirs="$dir_base/mail/*/*/cur $dir_base/mail/*/*/new $dir_base/mail/*/*/tmp $dir_base/mail/*/*/.*/cur $dir_base/mail/*/*/.*/new $dir_base/mail/*/*/.*/tmp"
# compress mail files
for base_dir in $dirs; do
for dir in "$base_dir" "$base_dir"/.[!.]*; do
if [ -d "$dir" ] && [[ "$dir" != */./* ]]; then
while IFS= read -r -d '' filename; do
# verify is it is already compressed
if file "$filename" | grep -q 'gzip compressed data'; then
echo "The file '$filename' is already compressed. Skipping."
continue
fi
echo "Compressing '$filename' file."
# compress
gzip "$filename"
# rename removing .gz extension
mv "${filename}.gz" "${filename}"
done < <(find "$dir" -type f -print0)
fi
done
done
echo "Compressing done!"
Usage:
# Compress all emails on the server
./compress-emails.sh
# or compress all emails for a single user (/home/[username])
./compress-emails.sh -u username
Hi danielbr86,
Actually your script is not the best way to compress existing emails. There is a lot that goes on behind the scenes in dovecot that your script does not account for.
Looking at the documentation for dovecot, a better approach is to use the doveadm tool to migrate the mail accounts.
First create a designated folder to backup the email account you want to compress. You want the permissions to be admin:mail - 0750.
/home/admin/backup_mail
Then use the doveadm command like so to backup the mail account.
doveadm backup -u [email protected] maildir:/home/admin/backup_mail
Then delete all the data from the mail account [email protected].
doveadm expunge -u [email protected] mailbox '*' all
Then restore from the backup location.
doveadm import -u [email protected] maildir:/home/admin/backup_mail "" ALL
Now rebuild indexes.
doveadm force-resync -u [email protected] '*'
doveadm fts rescan -u [email protected]
Now check to see if you have all your mail.
This process uses dovecot’s native plugin zlib to compress each email.
To know if an email is compressed simply use the file command on the file to get info. It will tell you the type of compression used if it is compressed.
Note, if you are using the quota plugin and your config is using quota = maildir:User quota then you have the old size of the mailbox still. You need to use quota = dirsize:User quata to have the sytem calculate the actual size instead of relying on the old filename which has the old file size. Yes, this uses more resources.
Good luck to everyone!
2 Likes