I’m facing two problems at the moment, and I’m not sure the best way forward.
I am routing all my email from Hestia through Amazon SES for deliverability purposes. Come to my surprise, I check my SES account and there have been 1440 emails sent. So I ruled out a hack and I’m assuming that it is CRON doing this because before I turned off notifications using the “turn off notifications” button, I was getting sent an email every minute telling me that a cron job was processed etc.
My exim server is getting absolutely battered by an Iranian brute force attacker, here are some logs I found after checking mainlog in /var/log/exim4. Because of so many requests and I think the firewall is working overtime, thus making my CPU skyrocket to 100% rendering my server inoperable for a few seconds and sometimes minutes.
2020-07-04 06:24:43 no host name found for IP address 126.96.36.199
2020-07-04 06:24:44 no host name found for IP address 188.8.131.52
2020-07-04 06:24:44 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[184.108.40.206] input=“QUIT\r\n”
2020-07-04 06:24:44 no host name found for IP address 220.127.116.11
2020-07-04 06:24:46 dovecot_login authenticator failed for (User) [18.104.22.168]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:46 dovecot_login authenticator failed for (User) [22.214.171.124]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:47 no host name found for IP address 126.96.36.199
2020-07-04 06:24:47 dovecot_login authenticator failed for (User) [188.8.131.52]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:47 no host name found for IP address 184.108.40.206
2020-07-04 06:24:48 dovecot_login authenticator failed for (User) [220.127.116.11]: 535 Incorrect authentication data ([email protected]becauseonly2linksallowed)
2020-07-04 06:24:48 no host name found for IP address 18.104.22.168
2020-07-04 06:24:49 no host name found for IP address 22.214.171.124
2020-07-04 06:24:50 dovecot_login authenticator failed for (User) [126.96.36.199]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:51 no host name found for IP address 188.8.131.52
2020-07-04 06:24:52 no host name found for IP address 184.108.40.206
2020-07-04 06:24:52 dovecot_login authenticator failed for (User) [220.127.116.11]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:52 no host name found for IP address 18.104.22.168
2020-07-04 06:24:54 dovecot_login authenticator failed for (User) [22.214.171.124]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:54 dovecot_login authenticator failed for (User) [126.96.36.199]: 535 Incorrect authentication data ([email protected]d)
2020-07-04 06:24:55 dovecot_login authenticator failed for (User) [188.8.131.52]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:55 dovecot_login authenticator failed for (User) [184.108.40.206]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:56 no host name found for IP address 220.127.116.11
2020-07-04 06:24:58 dovecot_login authenticator failed for (User) [18.104.22.168]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:59 dovecot_login authenticator failed for (User) [22.214.171.124]: 535 Incorrect authentication data ([email protected])
2020-07-04 06:24:59 no host name found for IP address 126.96.36.199
These days you just can’t put a server online without first having applied a comprehensive multi-level security policy, both preventive (e.g. firewall with geoIP blocks and throttling) and reactive (e.g. fail2ban, Postfix tarpit etc).
Regarding hackers: 4 months ago I put a new testing / staging server online (note: no sites, no domains); within 48hrs it was receiving over 200,000/day smtp auth connections from a /24 from Iran (note: they rotated through all IPs so they could circumvent the throttling rules I used), trying to guess username/passwords.
To the right words what @kpv already wrote, can you describe more about the cron emails you seem to send? Usualy, if the configuration is right, you should not have unusual cron messages expect for probaly failed lets encrypt renewals.
I’ve never really worked with Fail2Ban much before, but I blocked majority of the one ASN’s range and it slowed down a lot, but now some of the IP’s are using residential networks which is harder for me to block.
Am I doing this wrong?
These are some of the initial ones that got sent to my email
In here: https://pastebin.com/Z5pHFCvH
Now when I check the mainlog, these are the logs that it gives me when it actually goes through SES, shortened down to two because these are the two most recurring logs
CHECK PASTEBIN HERE https://pastebin.com/Z5pHFCvH
When I first checked it too, the frozen messages were piled up and I found a way to clear that.
I’m not sure how to find it now, but it was basically showing some of the successful cron’s coming from root user after I disabled notifications on the other user.
Had to put pastebin bc it wasn’t allowing me to send two links
Is there anything that I can do about this?
When I add it to iptables, it seems to be getting automatically removed and there’s a ton of IP ranges to ban. So how do I go about doing this? Is there a way to implement a publically available blocklist with all these SMTP bruteforce attackers? Is there a way to bulk add these IP ranges to the firewall and let it SAVE permanently?
hmm, that’s probably just background noise.
however if you implement measures on the application layer that run checks on each incoming request that obviously might be a bad idea as you yourself add to the load of your server.
let me give you an example: with good intention you add too many DNSBL blacklists in your mail config. each incoming request to your server is eventually going to be checked against each of them causing a multiple of outgoing requests and work to the system. bad idea. rather stick to just a few.
try to filter at the earliest point possible, which usually is iptables. fail2ban will add failed login attempts automatically to a certain extend, so you don’t even need to interfere.
if you want to add things on top of that, I suggest looking into ipset and blacklists. nice starting point: https://libraries.io/github/trick77/ipset-blacklist
for the cron mails: if you set up a cronjob that runs every minute and don’t surpress eventual output properly, of course you will get a mail every minute that php7.3 artisan call does not look like something hestia does but rather a custom thing you added yourself
The CRON seems to be fixed after changing the output of each CRON command, including the root ones.
The reason i want to blacklist these IP’s is because when CLAMAV runs, it maxes out the server with 100% CPU usage and it renders is unusable for a good few minutes before it relaxes itself. So I thought this would be a good way to combat that, perhaps I have another issue with ClamAV that I don’t know about?
Edit: Also, iptables clears the firewall every now and then for some reason?
yes blacklisting the IPs before the requests reach clamav is a good idea. it seems weird though, that so much stuff gets that far anyway as login/auth errors should be rejected before clam even gets invoked.
also fail2ban will act on these login fails and block the ips automatically, that’s why I mentioned it earlier.
are you eventually using catchall for mails instead of just adding the really needed mail-addresses with a few aliases? if so I’d definitely turn that off to not receive all the shit for non existing mail-addresses which will be 101% spam anyway.
iptables does not flush itself, but fail2ban only bans temporarily and releases the blocks after a while. that makes sense, because often IPs can originate from highjacked devices and could be used by other individuals later that got nothing to do with it. you can however tweak the fail2ban config to your needs
if you change the firewall settings in HestiaCP the firewall rules might be resetted though.
There’s only a few emails on the server itself, 3-4 max. It uses Amazon SES so all the the addresses that they are trying would never work because I’ve got it routing through Amazon SES and that blocks the email that isn’t verified.
Question: Would it be better for me to use CSF instead of IPTables + Fail2Ban? Is there any pros to that?
The problem I’m having with using the HestiaCP firewall is that I have to add each IP range individually one at a time, and choose a service for it.
Is there a way to block them all at once for all services?
Also, is there a way to block ASN’s?
you have incoming mail routed through SES? how is it hammering your server then and why would clamav have a problem with the load? I think there is something else off with your setup then
I am not a fan of CSF but that’s most likely because I haven’t used it much but rather stick to iptables as I know that much better. for better blocks see the link I posted above, that’s a script that pulls in blocklists (you can define which ones to use) on a cron schedule and adds all these IPs to an ipset which works as a global blocklist on your server. no manual work involved other than picking good blocklists for your use case (I recommend looking at firehol datasets for that)
yes, use command line with the correct iptables command
Outgoing mail, not incoming.
I have been adding them through IPTables, however the same IP’s even though they were blocked last week, started attacking again today which is my concern that it was getting removed?
so how is SES outgoing to help you with attackers an random mail-names if you use catchall on a mail-domain?
I think we are talking different things here and need to take a step back and look at your full mail-process if you want to optimize it
you are seeing high load with clamav, which means that obviously a high amount of mails come that far into your system, that clamav gets triggered. at that point SES is not involved, so you would want to narrow down on the cause of that much mail getting that far.
if it would be brute force / login attempts to exim or dovecot, fail2ban would take care of them after a few tries and these IPs would pile up being banned automatically in iptables anyway (and clamav for sure would not see these)
that takes me to the assumption that it’s just spam mails that are thrown at your server, most likely with random names that don’t exist - however these too should be rejected before clamav gets involved - unless the domain is set up for catchall.
if you now combine that catchall with a forward to an external mail address this surely would go back through SES (and eventually clamav a 2nd time) and maybe SES even filters out shit - however that doesn’t help your system at that late time in the whole process…
maybe I am not drawing it correctly and your process looks indeed different, however, I think you get the idea. as said before, you want to filter at the earliest time possible to keep the stuff away from reaching your system at all. SES therefore does not help you at any point, as it’s handling whatever (outgoing) mail far too late.