Fail2Ban and IPtables NOT blocking successfully

Community Support Firewall
1

Hello great HestiaCP community. I’ve recently been working on adding a few additional jails to fail2ban to stop some of the common probing attacks I have been seeing hit my websites and EXIM server.

I have managed to successfully add the fail2ban jails and I see that they are working. They are successfully identifying the errors in the apache server logs, they are successfully banning the IP, and it’s successfully adding the IP to the iptables and ip6tables. However, the server is NOT blocking the IPs. I don’t understand how this could be. But I tested it several times by triggering the fail2ban jail rules with my IP Address. I can see that my IP address is banned and it’s also added to the appropriate IPTables jail chain.

In fact, I noticed that even the default Hestia jails, which are also banning IPs, are being added to iptables, but the banned IPs are still allowed to hit the server and continue their probing. But I can clearly see that the IP is listed in iptables and blocked to all destinations on all ports.

Anyway else seeing this. It’s almost as if the iptables firewall is not even be used, but I can see that it’s active, running on both the HestiaCP admin panel as well as directly from checking via terminal.

Could someone help me understand why the IPs are not being blocked? Thank you in advance!

2

If it helps, here is the output of iptables -S

-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N f2b-apache-primary
-N f2b-exim-reject
-N fail2ban-FTP
-N fail2ban-HESTIA
-N fail2ban-MAIL
-N fail2ban-RECIDIVE
-N fail2ban-SSH
-N fail2ban-WEB
-N hestia
-A INPUT -p tcp -m multiport --dports 25,465,465,587 -j f2b-exim-reject
-A INPUT -p tcp -m multiport --dports 0:65535 -j f2b-apache-primary
-A INPUT -p tcp -m tcp --dport 8083 -j fail2ban-HESTIA
-A INPUT -p tcp -m tcp --dport 21 -j fail2ban-FTP
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-WEB
-A INPUT -p tcp -m multiport --dports 25,465,587,110,995,143,993 -j fail2ban-MAIL
-A INPUT -p tcp -m multiport --dports 1:65535 -j fail2ban-RECIDIVE
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s (INTERNALIPADDRESS)/32 -j ACCEPT
-A INPUT -s (INTERNALIPADDRESS)/32 -j ACCEPT
-A INPUT -s (PUBLICIPADDRESS/32 -j ACCEPT
-A INPUT -s 127.0.0.1/32 -j ACCEPT
-A INPUT -p tcp -m set --match-set BlockMalicious src -j DROP
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 21,12000:12100 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 25,465,587 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 110,995 -j DROP
-A INPUT -p tcp -m multiport --dports 143,993 -j DROP
-A INPUT -p tcp -m tcp --dport 8083 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A f2b-apache-primary -s 155.248.180.30/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-apache-primary -s 20.190.217.226/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-apache-primary -j RETURN
-A f2b-exim-reject -j RETURN
-A fail2ban-FTP -j RETURN
-A fail2ban-HESTIA -j RETURN
-A fail2ban-RECIDIVE -s 87.120.93.10/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 80.231.174.107/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 60.54.218.240/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 208.109.188.104/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 149.56.128.35/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 143.110.249.252/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 143.110.187.22/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 118.37.157.169/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -s 103.190.29.88/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 172.174.177.141/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 182.229.10.141/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 159.89.192.100/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 139.59.11.227/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 189.181.212.129/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -j RETURN
3

And here is the ip6tables output using ip6tables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N f2b-apache-primary
-A INPUT -p tcp -m multiport --dports 0:65535 -j f2b-apache-primary
-A f2b-apache-primary -s 2600:8801:117:000f:c611:9994:a64e:17ca/128 -j REJECT --reject-with icmp6-port-unreachable
-A f2b-apache-primary -j RETURN
4

Okay, so I figured out the cause. It was due to Cloudflare. I use Cloudflare’s built in proxy service, so my server IP is not shown to the visitors. You can enable this within Cloudflare in their DNS settings.

However, I still don’t understand why having the proxy setting on, prevents the server and iptables from blocking the IP address. If anyone can help me understand that, I would appreciate it. It seems even though I am using Cloudflare, my server can see the external user’s IP address that is probing my server and it successfully identifies the IP, bans it, and then adds it to iptables to block. However, it doesn’t get blocked due to the Cloudflare’s proxy. This is the part I don’t understand.

Anyway, the way I solved this was by using Cloudflare’s API. This website explains it quite well and it’s super quick and easy.

It just involves updating the Cloudflare action file in fail2ban, adding your global API key and email address and then adding in the cloudflare action into each of your fail2ban jails. Once I did that, my problem was solved. In this case, the IP gets banned and it calls the Cloudflare API to ban the IP at the Cloudflare level.

Anyway, I hope this helps the next person that is scratching their head wondering why the hell their server isn’t blocking the IPs. I went through and checked everything but Cloudflare, which is why it took me so longer to figure this out. My logic was, since the server can see the external user’s IP and successfully ban it, it can’t possibly by something related to Cloudflare. But apparently it is and it has to do with their proxy service. I still don’t understand why, but happy that I got this figured out and my server is back in business blocking these annoying aholes.

1 Like
5

Hi @user7632726

The server receives headers sent by Cloudflare that contain the real IP, but those IPs are not directly connecting to your server, only Cloudflare’s IPs are.

The reason you see the real IP in the logs is that Nginx has been compiled with the realip modules.

$ nginx -V 2>&1 | tr ' ' '\n' | grep realip
--with-http_realip_module
--with-stream_realip_module

Additionally, Hestia configures Nginx to display the real IP address for connections coming from the Cloudflare CDN

$ cat /etc/nginx/conf.d/cloudflare.inc
# Cloudflare IP Ranges

# IPv4
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;

# IPv6
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2a06:98c0::/29;
set_real_ip_from 2c0f:f248::/32;

real_ip_header CF-Connecting-IP;

So you see the real IP in the logs because real_ip_header directive uses the CF-Connecting-IP header, which is sent by Cloudflare, to display the client IP in the log.

1 Like
6

Amazing! Thank you for the explanation @sahsanu! Now I fully understand. And thank you and the HestiaCP team for building the HestiaCP in a way that considers this. I wasn’t even aware that such considerations and actions were taken to ensure compatibility with Cloudflare, as I assume if it had not been configured that way, the fail2ban rules I had setup would have been blocking Cloudflare IPs instead and that would have caused a completely different issue.

Ok, so now I understand why iptables wasn’t blocking. And I assume the action I took to use the Cloudflare API and fail2ban’s cloudflare action, is the appropriate way to block these bots and TAs. Thanks again!

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.