As it happens, yesterday I went ahead and deleted all the users from the new Hestia installation and spent most of the day experimenting with this. By late last night I finally got it all working beautifully. I can create mail folders with both periods and slashes in the names.
First, I’ll show all the modifications I made to various config files, then I’ll give a full explanation, then show the results…
MODIFICATIONS
In /etc/dovecot/dovecot.conf, added the following lines at the top, just before the !include lines:
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_plugins = listescape
and changed separator in the namespace to a tilde:
separator = "~"
In /etc/dovecot/conf.d/20-imap.conf, changed the mail_plugins line to read as follows (so that it would include the “listescape” from dovecot.conf):
mail_plugins = $mail_plugins quota imap_quota
In /etc/dovecot/conf.d/10-mail.conf, changed mail_location to the following:
mail_location = maildir:%h/mail/%d/%n/store:LAYOUT=fs:FULLDIRNAME=0_FolderContent:UTF-8:INDEX=%h/mail/%d/%n/index:CONTROL=%h/mail/%d/%n/control:VOLATILEDIR=%h/mail/%d/%n/volatile
In /etc/exim4/exim4.conf.template:
under “local_delivery” transport, changed directory to the following, and copied the “quota_directory” line from local_spam_delivery:
directory = "${extract{5}{:}{${lookup{$local_part}lsearch{/etc/exim4/domains/$domain/passwd}}}}/mail/$domain/$local_part/store/INBOX/0_FolderContent"
quota_directory = "${extract{5}{:}{${lookup{$local_part}lsearch{/etc/exim4/domains/$domain/passwd}}}}/mail/$domain/$local_part"
under “local_spam_delivery” transport, changed directory to the following:
directory = "${extract{5}{:}{${lookup{$local_part}lsearch{/etc/exim4/domains/$domain/passwd}}}}/mail/$domain/$local_part/store/Junk/0_FolderContent"
EXPLANATION
Problem 1: Periods in Folder Names – As I said in the previous post, Dovecot’s default Maildir format is “LAYOUT=Maildir++”, which stores all mail subfolders as single filesystem directories named with the full path of the mail subfolders (with names separated by periods). So for example, if you had a “Correspondence” mail folder with a “Work” subfolder and subfolders with people’s names within that, they would be stored as follows:
/home/user/mail/domain/user/.Correspondence/
/home/user/mail/domain/user/.Correspondence.Work/
/home/user/mail/domain/user/.Correspondence.Work.Rufus/
/home/user/mail/domain/user/.Correspondence.Work.Delbert/
Consequently, the user cannot create mail folders containing periods, as they often would when they want to have a typical abbreviation in the name. So a folder like the following would not be allowed: “/home/user/mail/domain/user/.Correspondence.Work.Mr. Rogers/”. If, however, you specify “LAYOUT=fs” for the Maildir format in Dovecot’s mail_location directive, it will store each mail folder as a regular directory in the filesystem. That alleviates the restriction on periods, so “/home/user/mail/domain/user/Correspondence/Work/Mr. Rogers/” will work fine. Now the above example will be stored as follows:
/home/user/mail/domain/user/Correspondence/
/home/user/mail/domain/user/Correspondence/Work/
/home/user/mail/domain/user/Correspondence/Work/Rufus/
/home/user/mail/domain/user/Correspondence/Work/Delbert/
/home/user/mail/domain/user/Correspondence/Work/Mr.\ Rogers/
Problem 2: Slashes in Folder Names – Let’s face it, the slash is a commonly used punctuation mark: he/she, 2018/2019, this/that. It’s kind of absurd that in the year 2020 people still can’t use slashes in mail folder names. Luckily, Dovecot has a solution for that. First, enable the “listescape” plugin that comes with Dovecot, which “escapes” special characters in filesystem directory names. Second, change the mail folder separator to something far less common, such as a backslash or tilde. I chose tilde because I’ve read that backslash sometimes causes problems.
Problem 3: Folder Name Conflicts – When all the metadata files are stored in the same directories as the actual mail folders and messages—and hidden from view by Dovecot—the potential is always there that a user may try to create a folder with the same name as one of the hidden files/directories. Luckily, Dovecot offers several ways to deal with that:
First, In every mail folder, Dovecot creates three subfolders for its own internal use: cur, new, tmp. Both “new” and “tmp” could conceivably be problematic. In the mail_location directive, you can use the FULLDIRNAME option to tell Dovecot to put those three directories into a single directory with a name that no one would most likely try to use as their own subfolder name. The Dovecot documentation gives an example of “mAildir”. Meh, that seems odd. I’d rather name it something meaningful, like “FolderContent”. And I added “0_” to the beginning of it so that (1) it would always be output at the top of the directory list and (2) to make it far less likely that anyone would try to create their own mail folder by that name. One side effect of using the FULLDIRNAME option is that, rather than creating a “0_FolderContent” directory at the root of the tree, Dovecot instead creates a directory named “INBOX” with its own 0_FolderContent directory. That’s why I had to add that to the “directory” lines in exim4.conf.template.
Second, Dovecot allows you to make a clean separation between the locations where it stores the actual mailbox data, the indexes, and other “control” files that Dovecot uses. These locations are specified in the mail_location directive in /etc/dovecot/conf.d/10-mail.conf with options named INDEX, CONTROL, and VOLATILEDIR. Doing this has several advantages: (1) the directory storing the actual email messages doesn’t have extraneous, “hidden” files in it that aren’t part of the user’s mail content; (2) you have the option of putting the indexes on fast SSD storage (or even a ramdisk, I suppose); (3) the base directory is then left open for other config files, such as sieve filters, rather than putting them as hidden files in the directory where the mail folders are actually stored. So with this new config, a user’s mail directory contains the following four directories: control, index, store, volatile. I’m going to be experimenting with getting sieve to work with RoundCube, so this will allow me to add a “sieve” directory alongside these others.
And a few final side notes:
I just added log_timestamp = "%Y-%m-%d %H:%M:%S " because I hate it when log file times aren’t in ISO format :-).
And in dovecot.conf, I changed mailbox Junk to auto=subscribe and set mailbox Spam to auto=no, just because in my experience, “Junk” is the most common, standard, default name for the junk folder in email clients.
RESULTS
SO, I’ve created a “test” user to try these things out. Here’s what the directories look like for that user:
$ ll /home/example/mail/example.com/test/
drwx------ 8 example mail 188 2020-03-23 12:50:17 control/
drwx------ 8 example mail 142 2020-03-23 12:49:45 index/
drwx------ 8 example mail 85 2020-03-23 12:49:45 store/
drwx------ 2 example mail 6 2020-03-23 12:50:17 volatile/
$ ll /home/example/mail/example.com/test/store
drwx------ 4 example mail 48 2020-03-23 12:50:17 Archive/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Drafts/
drwx------ 3 example mail 29 2020-03-23 12:46:35 INBOX/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Junk/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Sent/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Trash/
$ ll /home/example/mail/example.com/test/store/INBOX/0_FolderContent/
drwx------ 2 example mail 60 2020-03-23 12:49:45 cur/
-rw-rw---- 1 example mail 17 2020-03-23 12:48:42 maildirsize
drwx------ 2 example mail 6 2020-03-23 12:49:45 new/
drwx------ 2 example mail 6 2020-03-23 12:48:42 tmp/
In RoundCube, I created a folder named “2017/2018” in the Archive folder. Thanks to the Dovecot “listescape” plugin, here’s how it created the actual filesystem directory:
$ ll /home/example/mail/example.com/test/store/Archive/
drwx------ 5 example mail 39 2020-03-23 12:49:45 0_FolderContent/
drwx------ 3 example mail 29 2020-03-23 12:50:17 '2017\2f2018'/
drwx------ 3 example mail 29 2020-03-23 13:37:28 'Lou. Area Humanists'/
And here are the metadata directories:
$ ll /home/example/mail/example.com/test/control/
drwx------ 4 example mail 48 2020-03-23 12:50:17 Archive/
-rw------- 1 example mail 8 2020-03-23 12:50:17 dovecot-uidvalidity
-r--r--r-- 1 example mail 0 2020-03-23 12:46:35 dovecot-uidvalidity.5e78f601
drwx------ 3 example mail 29 2020-03-23 12:49:45 Drafts/
drwx------ 3 example mail 29 2020-03-23 12:46:35 INBOX/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Junk/
-rw------- 1 example mail 10 2020-03-23 12:49:44 maildirsize
drwx------ 3 example mail 29 2020-03-23 12:49:45 Sent/
-rw------- 1 example mail 56 2020-03-23 12:50:17 subscriptions
drwx------ 3 example mail 29 2020-03-23 12:49:45 Trash/
$ ll /home/example/mail/example.com/test/index/
drwx------ 4 example mail 48 2020-03-23 12:50:17 Archive/
-rw------- 1 example mail 3.8K 2020-03-23 12:50:17 dovecot.list.index.log
-rw------- 1 example mail 144 2020-03-23 12:50:17 dovecot.mailbox.log
drwx------ 3 example mail 29 2020-03-23 12:49:45 Drafts/
drwx------ 3 example mail 29 2020-03-23 12:46:35 INBOX/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Junk/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Sent/
drwx------ 3 example mail 29 2020-03-23 12:49:45 Trash/
All of the mail from the Zimbra server is currently being transferred and hasn’t run into any problems.
For the sake of completeness, if anyone is interested, here’s the imapsync command I’m using to transfer a user account. I’ve used this many times over the past 20 years or so to transfer back and forth between old Linux servers with everything in mbox files, Cyrus servers, Exchange, Zimbra, ISPConfig, and now this.
mkdir /var/tmp/imapsync
imapsync --host1 SOURCEIPADDRESS --port1 993 --ssl1 --user1 USERNAME --password1 'PASSWORD' \
--host2 localhost --port2 993 --ssl2 --user2 'USERNAME' --password2 'PASSWORD' \
--nofoldersizes --nofoldersizesatend --skipsize --nocheckmessageexists \
--delete2 --useuid --tmpdir /var/tmp/imapsync
Make sure you have enough space on the drive where you put the tmpdir. It will create a cache directory so that you can keep re-syncing and it won’t have to resync everything all over again, only the changes.
Hope everyone’s staying safe.
Cheers
–Dan