I found the culprit! It was in /etc/exim4/exim4.conf.template all along.
Here there are multiple ACLs checking the passwd file for data still, I just need to replace those checks with LDAP lookups and I’m set.
I don’t expect anyone here to know how to do that, I’ll ask around. If you know though, a pointer in the right direction would be more than welcome!
I’ll close this topic now, but it might be useful for future reference in case anyone else wants to swap out authentication backends for email.