FTS Solr support in Dovecot?

[andrewnewby]

Hi,

While this isn’t directly Hestia related, I thought I’d ask here as you guys may know more! I’ve got some large mailboxes, and I want to setup Solr FTS plugin so that its faster to search (and better). So I’ve been reading:

https://doc.dovecot.org/configuration_manual/fts/solr/

That talks of Solr 7.7, which is super old. So I’ve installed Solr 8.9.0:

sudo wget https://archive.apache.org/dist/lucene/solr/8.9.0/solr-8.9.0.tgz
sudo tar xzf solr-8.9.0.tgz
sudo bash solr-8.9.0/bin/install_solr_service.sh solr-8.9.0.tgz
sudo systemctl status solr
sudo systemctl enable solr
sudo -u solr /opt/solr/bin/solr create -c dovecot

That created the files ok. Then I copied the schema/solrconfig files they provide (all be it for Solr 7.7, but it seems to load ok with no errors)

Then I installed:

apt-get install dovecot-solr

…but now I’m stuck! I tried adding this in the very top of dovcot.conf:

mail_plugins = $mail_plugins fts fts_solr

plugin {
  fts_autoindex = yes
  fts = solr
  fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ debug
}

It loads (with no errors), but not sure its working. If I stick it in ./conf.d/90-plugin.conf, as I would have expected to do - I get:

Dec 17 07:44:48 east.newbyhost.com dovecot[125293]: doveconf: Warning: /etc/dovecot/conf.d/90-plugin.conf line 13: Global setting mail_plugins won’t change the setting inside an earlier filter at /etc/dovecot/conf.d/15-lda.conf line 47

I guess one of my questions is - that it says

Dovecot is not compiled with Solr FTS support by default. To enable it, you need to add the --with-solr parameter to your invocation of the configure script. You will also need to have libexpat installed, including development headers (typically from a separate development package). Configuration will fail if --with-solr is enabled while libexpat headers cannot be found. Older versions of Dovecot also required libcurl for Solr support, but recent versions of Dovecot include a custom HTTP client.

How does Hestia build Dovecot? With that flag? I’m hoping I can make a how-to if I can get this working, so others can make use of it

Thanks!

Andy

[eris]

We don’t build our own dovecote we use the packages from Ubuntu / Debian

[andrewnewby]

Ah ok. I wonder if that makes a difference I don’t see anything related to solr (apart from the bits I added);

root@east:/etc/dovecot# grep -r solr
conf.d/90-plugin.conf:mail_plugins = $mail_plugins fts fts_solr
conf.d/90-plugin.conf:  fts = solr
conf.d/90-plugin.conf:  fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ debug
dovecot.conf:# mail_plugins = $mail_plugins fts fts_solr
dovecot.conf:#   fts = solr
dovecot.conf:#   fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ debug

Not too sure if it would be in that folder. I feel like I’m so close to figuring this out its frustrating! Manually trying to rescan with debug gives me:

doveadm -D fts rescan -u [email protected]
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
doveadm([email protected])<131039><>: Debug: auth-master: userdb lookup([email protected]): Started userdb lookup
doveadm([email protected])<131039><>: Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Connecting
doveadm([email protected])<131039><>: Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Client connected (fd=10)
doveadm([email protected])<131039><>: Debug: auth-master: userdb lookup([email protected]): auth USER input: [email protected] uid=1011 gid=8 home=/home/hatlamp quota_rule=*:storage=0M
doveadm([email protected])<131039><>: Debug: auth-master: userdb lookup([email protected]): Finished userdb lookup ([email protected] uid=1011 gid=8 home=/home/hatlamp quota_rule=*:storage=0M)
doveadm([email protected])<131039><>: Debug: Added userdb setting: plugin/quota_rule=*:storage=0M
doveadm([email protected]): Debug: Effective uid=1011, gid=8, home=/home/hatlamp
doveadm([email protected]): Debug: Namespace : type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/hatlamp/mail/hatlamp.com/hello
doveadm([email protected]): Debug: maildir++: root=/home/hatlamp/mail/hatlamp.com/hello, index=, indexpvt=, control=, inbox=/home/hatlamp/mail/hatlamp.com/hello, alt=
doveadm([email protected]): Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Disconnected: Connection closed (fd=10)

It just doesn’t seem to do anything (no increase in the Solr docs indexed). I’m familiar with Solr, as I’ve used it on different projects as a search engine. Just not so familiar with how Dovecot interacts with it

[eris]

It might be possible the Ubuntu / Debian doesn’t build with Solr support in that case you need to build dovecote your self.

How ever I have no experience with it.

[eris]

@andrewnewby

fts_solr

Compiling

Dovecot is not compiled with Solr FTS support by default. To enable it, you need to add the --with-solr parameter to your invocation of the configure script. You will also need to have libexpat installed, including development headers (typically from a separate development package). Configuration will fail if --with-solr is enabled while libexpat headers cannot be found. Older versions of Dovecot also required libcurl for Solr support, but recent versions of Dovecot include a custom HTTP client.

As expected…

[andrewnewby]

Ah ok - so I was right thinking that Dovecot needs to be compiled with the --with-solr flag? (rather than just via an apt-get). I’m just having a look through hst-install-ubuntu.sh to see how its even installed in the first place?

UPDATE: I was reading somewhere that the packaged versions DO come with --with-solr enabled… not sure how I would check thats the case? (this post was from 2014!)

[andrewnewby]

I’m sure something else just isn’t right. Totally ignoring Solr for a moment, and just trying to list the status of the mailboxes:

doveadm -Dv mailbox status  -u [email protected] all
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
doveadm mailbox status [-u <user>|-A] [-S <socket_path>] <fields> <mailbox> [...]

Complains its the wrong syntax?

[andrewnewby]

Ok so I think I’m pretty much done

It turns out the Dovecot package that is installed does work with Solr. You just have to enable it (and install the dovecot-solr package). So this is pretty much it:

sudo wget https://archive.apache.org/dist/lucene/solr/8.9.0/solr-8.9.0.tgz
sudo tar xzf solr-8.9.0.tgz
sudo bash solr-8.9.0/bin/install_solr_service.sh solr-8.9.0.tgz
sudo systemctl status solr
sudo systemctl enable solr
sudo -u solr /opt/solr/bin/solr create -c dovecot

apt-get install dovecot-solr

wget -O/var/solr/data/dovecot/conf/solrconfig.xml https://raw.githubusercontent.com/dovecot/core/main/doc/solr-config-7.7.0.xml
wget -O/var/solr/data/dovecot/conf/schema.xml https://raw.githubusercontent.com/dovecot/core/main/doc/solr-schema-7.7.0.xml 
ln -s /var/solr/data/dovecot/conf/schema.xml /var/solr/data/dovecot/conf/managed-schema

Add this to the top of /etc/dovecot/dovcot.conf:

mail_plugins = $mail_plugins fts fts_solr

plugin {
  fts_autoindex = yes
  fts = solr
  fts_solr = url=http://127.0.0.1:8983/solr/dovecot/ debug
}

Add firewall ALLOW entries for TCP 8983, for local IP (127.0.0.1), and another rule for your own IP

Without setting up a brand new server to test all those commands exactly, that should do it. The issue I was having with it nort being indexed was a PITA. it turns out all I needed to do was send in an email, and that would trigger it to start indexing that account!!!

Hopefully this helps someone else I may write a .sh script when time permits, to make this easier and faster in the future when pushing out onto other servers (and so others can make use of it as well). Any tips and suggestions would be welcome of course!