Seems like apache is the issue:
$ sudo lsof | awk '{print $1}' | sort | uniq -c | sort -h | tail -5
2809 exim4
8138 named
15657 nginx
24092 mariadbd
65246 apache2
But raising the limit in the apache2 Service config did not help…
And I am not sure whether such a number of open files is even reasonable?