Iptables not blocking latest DDoS attackers IP's

[neven]

Hi
I was checking high server loads due to these new DDoS attacks, and noticed my iptables has no effect on some of them.
Everything seems fine. iptables runs correctly, v-update-firewall works and updates correctly, no errors or warnings anywhere…
When I try to block random IP address, It works as expected.

But in this example:
I’m blocking web access from 3.124.100.133 and the output from “iptables -L -v” is

Chain fail2ban-WEB (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all – any any ec2-3-124-100-133.eu-central-1.compute.amazonaws.com anywhere reject-with icmp-port-unreachable
447K 46M RETURN all – any any anywhere anywhere

this IP connects to port 443 without issues, with 50 or so connections.

This is the first time I’m seeing iptables resolving from IP’s PTR record. Could this be the problem?

Or I’m missing something else?

[sahsanu]

@neven

This is the first time I’m seeing iptables resolving from IP’s PTR record. Could this be the problem?

iptables will try to resolve ips, if you don’t want to resolve them add -n to the command:

iptables -L -v -n

Maybe you added the rule after other rules, how did you add the ip? In Hestia you should add something like this:

v-add-firewall-ban 3.124.100.133 WEB

or if you want to ban the ip for more time:

v-add-firewall-ban 3.124.100.133 RECIDIVE

Could you please show the output of this command?

iptables -S

[neven]

Thanks for helping. I don’t think there’s an issue with the config, because I’m successfully blocking other IP’s both from web interface and cli.

Here is the config. I’ve removed identifying data. (IP’s and ports)

-P INPUT DROP
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N f2b-sshd
-N fail2ban-DNS
-N fail2ban-FTP
-N fail2ban-HESTIA
-N fail2ban-MAIL
-N fail2ban-RECIDIVE
-N fail2ban-SSH
-N fail2ban-WEB
-N hestia
-A INPUT -p tcp -m multiport --dports 1:65535 -j fail2ban-RECIDIVE
-A INPUT -p tcp -m multiport --dports 25,465,587,110,995,143,993 -j fail2ban-MAIL
-A INPUT -p tcp -m tcp --dport 21 -j fail2ban-FTP
-A INPUT -p tcp -m tcp --dport mysshport -j fail2ban-SSH
-A INPUT -p tcp -m tcp --dport myhestiaport -j fail2ban-HESTIA
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-WEB
-A INPUT -p udp -m udp --dport 53 -j fail2ban-DNS
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s serverlocalip1/32 -j ACCEPT
-A INPUT -s serverlocalip2/32 -j ACCEPT
-A INPUT -s serverpublicip1/32 -j ACCEPT
-A INPUT -s serverpublicip2/32 -j ACCEPT
-A INPUT -s 127.0.0.1/32 -j ACCEPT
-A INPUT -p tcp -m set --match-set firehollevel2 src -j DROP
-A INPUT -p tcp -m set --match-set spammers src -j DROP
-A INPUT -p tcp -m set --match-set bruteforce src -j DROP
-A INPUT -p tcp -m set --match-set torexits src -j DROP
-A INPUT -p tcp -m set --match-set Blacklist src -j DROP
-A INPUT -p tcp -m tcp --dport -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 21,12000:12100 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 25,465,587 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 110,995 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 143,993 -j ACCEPT
-A INPUT -p tcp -m tcp --dport -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A f2b-sshd -s bruteforcerIP1 -j REJECT --reject-with icmp-port-unreachable
loads more
-A f2b-sshd -s bruteforcerIP453 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A fail2ban-DNS -j RETURN
-A fail2ban-FTP -j RETURN
-A fail2ban-HESTIA -s bruteforcerIP1 -j REJECT --reject-with icmp-port-unreachable
loads more
-A fail2ban-HESTIA -s bruteforcerIP345 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-HESTIA -j RETURN
-A fail2ban-MAIL -j RETURN
-A fail2ban-RECIDIVE -s bruteforcerIP1 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-RECIDIVE -j RETURN
-A fail2ban-SSH -s bruteforcerIP1 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -j RETURN
-A fail2ban-WEB -s bruteforcerIP1 -j REJECT --reject-with icmp-port-unreachable
loads more
-A fail2ban-WEB -s bruteforcerIP1235 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-WEB -j RETURN

[sahsanu]

I suppose you forgot to change the ssh and hestia ports by mysshport and myhestiaport in two rules and that is the reason they look strange

-A INPUT -p tcp -m tcp --dport -j ACCEPT

Also, I suppose the ip 3.124.100.133 is in a fail2ban- to rule.

You are using f2b-sshd chain but none of your rules use that chain but that shouldn’t be a problem.

Anyways, I can’t see any issue with your rules. I think you are using several public ips but also, that shouldn’t be a problem, the iptables rules should affect all your public ips.

Sorry, but I’ve no idea what’s the reason for that ip still reaching your web server.

[neven]

@sahsanu

I suppose you forgot to change the ssh and hestia ports by mysshport and myhestiaport in two rules and that is the reason they look strange

-A INPUT -p tcp -m tcp --dport -j ACCEPT

Hahaha… No I was putting mysshport etc between “<” and “>” signs and the form filtered it out.

@sahsanu

You are using f2b-sshd chain but none of your rules use that chain but that shouldn’t be a problem.

Yes. Thanks. It must be a leftover from an old server.

3.124.100.133 is RECIDIVE and WEB chains. Also, it’s entire 14 bit range is in one of the blocklists…
And still i get 35 unique connections open to https port…

netstat -tn 2>/dev/null | grep :443 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -nr | head
35 3.124.100.133

It’s not a problem. I’ll block it at the gateway, but it’s just freaking me out. I’ve never had issues with iptables before.

[sahsanu]

@neven

netstat -tn 2>/dev/null | grep :443 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -nr | head
35 3.124.100.133

It’s a bit strange but maybe those connections were already ESTABLISHED so iptables is still allowing them…

[neven]

Found the cause… But not the cure.

I was watching the output of iftop and noticed that the offending IP is connecting to unusual server IP.

So I tried to delete this IP and got an error: can’t delete primary IP address

WTF? It was a temporary and never a primary ip address!

And ofcourse in /etc/netplan/60-hestia.yaml there is a wrong ip address configured.
Real primary IP adress is configured as additional IP in /etc/netork/interfaces

Both files were created by Hestia, so I’m not sure if changing anything there will have a lasting effect.

running v-update-sys-ip with the right IP does not seem to do anything.

Any idea how to switch the primary IP correctly?

[sahsanu]

First, I would check what are the ips Hestia is using:

v-list-sys-ips

or for more info:

v-list-sys-ips json

Regarding the primary address, this is what Hestia does to check whether the ip you want to remove is a primary address:

1.- Gets the default interface

ip -d -j route show | jq -r '.[] | if .dst == "default" then .dev else empty end'

Let’s say the interface is ens18

2.- With the default interface it gets the primery address:

ip -4 -d -j addr show "ens18" | jq -r '.[].addr_info[] | if .scope == "global" then .local else empty end' | head -n1

3.- Then compares the primary ip it got from previous command with the ip that you want to delete and if it is the same, it refuses to delete it.

WARNING: I didn’t test it so you should make a snapshot of your server just in case you want to try it because things could get really bad.

I would check that that ip is not used by any of your domains and if it is not used, maybe you should delete the ip manually, something like this:

ip addr del "YOURIP/CIDR" dev "INTERFACE"

Example:

ip addr del "203.0.113.0/24" dev "ens18"

And the I would try to remove the ip again:

v-delete-sys-ip YOURIP

Again, if you have doubts, don’t do that without a current snapshot of your server.

I’m pretty sure @eris has more info/knowdledge about how to do it the right way (Hestia way )

Good luck,
sahsanu

[neven]

@sahsanu

2.- With the default interface it gets the primery address:

ip -4 -d -j addr show "ens18" | jq -r '.[].addr_info[] | if .scope == "global" then .local else empty end' | head -n1

I looked at the jason and.

This one is currently primary. (And the one I want to delete.):
{“family”:“inet”,“local”:“WRONG_IP”,“prefixlen”:32,“scope”:“global”,“label”:“eth0”,“valid_life_time”:4294967295,“preferred_life_time”:4294967295}

And this one is additional, but should be primary:
{“family”:“inet”,“local”:“GOOD_IP”,“prefixlen”:32,“broadcast”:“GOOD_IP”,“scope”:“global”,“dynamic”:true,“label”:“eth0”,“valid_life_time”:81216,“preferred_life_time”:81216}

Server is extremely large production server so snapshotting and restoring is only for last resort situations. Also I already started to drink, so no more shell commands for me tonight.

Maybe eris has a quick way to switch the primary IP. (Btw curent primary IP is used by 0 vhosts, so later on it can be deleted.)
Still, I really appreciate your help!