Hacky New Year Everyone

pluto · January 1, 2021, 3:30am

From one of my Hestia servers …

Status for the jail: recidive
| |- Currently failed: 28
| |- Total failed: 6262
|- Currently banned: 719
|- Total banned: 1048
`- Banned IP list: 46.101.184.178 157.245.184.148 103.226.250.169 138.68.4.8 193.112.39.2 83.103.150.72 120.132.6.27 175.24.36.108 182.253.80.134 47.103.24.120 221.226.48.102 124.205.84.20 182.61.42.249 121.201.61.205 94.191.107.157 58.220.39.205 122.155.11.89 165.22.59.208 120.92.74.249 153.127.37.59 160.124.50.93 101.32.187.120 182.61.149.138 220.132.68.51 115.236.119.139 139.59.248.103 49.234.18.158 153.127.38.250 94.43.85.6 35.209.209.15 128.199.167.163 128.199.170.33 119.28.4.87 62.28.217.62 111.40.217.92 35.200.114.187 49.234.124.225 27.254.95.199 111.229.160.86 106.13.36.215 202.61.128.174 122.144.221.149 123.207.23.120 189.206.165.62 118.27.11.126 129.28.103.85 113.53.238.195 67.205.164.197 49.234.119.42 197.153.99.104 128.78.170.162 115.159.110.41 165.169.241.28 167.172.156.12 167.71.102.201 190.120.254.76 161.35.121.130 41.77.146.98 163.44.168.207 213.251.184.102 193.56.28.30 167.71.209.158 152.32.188.109 94.57.252.152 91.121.134.201 142.4.212.121 200.108.135.82 183.91.4.119 121.201.26.46 118.98.96.184 106.55.168.173 181.223.88.209 68.183.221.115 211.24.73.223 189.112.126.171 219.240.99.120 139.59.70.186 157.230.90.18 103.151.182.6 35.200.241.227 197.5.145.106 89.232.192.40 188.234.247.110 106.13.47.10 206.189.33.197 45.82.137.35 124.156.135.221 62.234.6.147 119.28.55.81 120.88.46.226 106.52.133.87 106.12.45.110 159.89.197.1 51.91.76.3 178.128.107.65 111.231.132.94 129.211.18.180 183.111.96.44 115.42.127.133 142.93.8.99 200.56.17.5 106.52.152.237 119.45.26.117 89.97.218.142 193.112.16.245 91.232.197.100 142.44.211.57 138.197.171.79 219.232.48.190 138.68.251.70 197.5.145.69 112.15.71.71 200.150.77.93 125.212.225.167 112.192.160.70 106.13.201.127 107.175.150.83 106.53.117.149 101.32.208.125 51.79.84.101 153.36.233.60 206.189.130.119 49.235.86.133 67.211.208.83 49.232.196.185 111.229.232.74 103.246.240.26 182.254.155.85 132.232.10.144 51.38.191.126 77.27.168.117 51.161.70.84 120.52.93.191 39.145.126.86 54.38.186.60 109.167.231.99 138.68.254.64 167.172.204.125 194.65.73.101 187.72.177.131 41.230.14.107 188.166.240.30 51.75.202.218 128.199.249.43 119.45.231.71 23.133.5.154 124.205.84.15 188.166.185.157 220.225.126.55 36.250.229.92 42.192.165.111 200.91.192.60 152.32.144.204 159.65.153.147 130.61.100.68 47.146.33.128 46.101.101.183 178.38.71.231 178.128.14.102 115.254.63.50 52.151.24.212 183.195.121.197 212.47.238.207 49.233.173.90 2.233.125.227 165.227.7.187 177.128.216.2 111.93.71.219 157.230.7.236 45.4.5.221 38.95.167.13 34.80.223.251 42.192.86.140 139.155.255.117 134.209.148.107 159.203.190.189 111.231.108.204 222.82.253.106 34.82.27.159 200.73.128.175 91.103.248.23 78.131.56.112 134.175.236.132 51.178.50.20 111.231.201.210 198.211.99.24 106.252.164.246 210.205.8.250 177.19.226.178 45.116.179.183 103.242.56.137 104.199.144.232 50.255.64.233 111.230.181.82 123.207.254.14 188.166.186.253 41.223.142.211 190.221.46.78 128.199.142.33 166.111.152.230 62.201.96.142 106.53.153.95 209.97.169.116 191.239.249.178 150.158.4.230 192.144.190.244 103.80.110.180 102.39.67.51 129.28.176.136 167.99.143.45 116.6.18.115 107.173.149.104 103.219.112.48 74.208.28.198 119.90.61.10 106.12.165.253 167.99.66.2 117.51.137.82 212.129.53.188 58.33.197.82 189.8.108.50 192.144.215.146 182.254.247.245 51.15.136.191 155.94.144.75 159.65.1.41 49.235.38.46 24.47.90.156 129.226.68.181 49.234.111.90 209.198.180.142 34.101.136.173 211.218.192.149 201.174.182.158 183.82.101.32 106.12.173.47 190.52.131.234 193.112.250.77 186.206.129.160 139.59.118.3 122.201.113.87 101.36.122.11 123.1.154.200 176.31.253.42 91.150.172.58 159.65.136.44 51.83.134.233 175.24.187.87 179.217.74.100 117.186.96.54 146.59.154.68 68.183.145.59 60.250.164.169 188.254.0.116 192.241.202.169 134.122.19.255 62.234.68.31 119.63.84.130 221.141.253.171 200.27.212.22 46.35.19.18 111.231.165.121 38.114.119.41 61.48.115.254 106.52.33.247 45.232.75.253 45.6.72.20 142.93.141.192 134.122.90.115 49.233.159.163 124.225.35.30 167.71.153.244 51.68.190.223 165.22.99.216 68.183.238.175 181.48.155.149 158.69.78.207 161.35.148.75 116.112.92.5 192.99.247.102 170.150.72.28 213.27.189.252 151.228.255.154 119.28.239.30 101.33.124.71 188.166.187.179 216.80.102.155 46.218.85.69 106.52.140.195 106.54.67.25 178.128.147.244 220.123.241.30 46.228.93.242 141.157.202.138 35.220.253.166 121.165.140.242 118.70.33.125 116.85.66.176 93.107.187.162 122.165.149.75 49.234.62.183 113.161.234.67 40.73.120.132 14.207.2.17 49.232.153.128 122.51.168.254 36.89.213.100 180.106.151.38 118.89.108.37 87.110.181.30 150.109.113.116 167.71.112.14 198.211.116.15 182.61.21.155 195.164.152.106 152.136.254.204 177.141.52.197 157.231.113.130 60.250.94.66 106.13.75.187 117.114.138.246 186.249.231.234 209.105.243.145 159.65.227.64 157.230.48.243 58.0.70.14 128.199.112.240 221.138.17.156 49.233.49.109 81.68.207.135 1.214.245.27 81.68.83.82 91.121.30.186 164.132.230.98 195.70.59.121 149.202.175.11 81.68.142.128 117.51.158.223 120.246.124.254 124.152.118.194 81.70.18.171 142.93.174.237 190.0.8.134 60.13.230.199 124.126.18.130 113.161.73.210 109.167.200.10 138.197.130.138 81.71.68.119 157.230.39.223 112.15.9.120 206.174.214.90 213.87.101.176 115.137.112.89 188.166.219.160 152.200.143.218 83.212.79.78 129.211.14.195 121.229.4.236 106.52.179.63 106.13.99.107 182.61.25.156 104.131.84.225 185.43.149.146 211.159.171.238 83.17.166.241 45.230.102.203 168.196.180.66 151.0.149.34 120.132.13.131 212.64.74.214 206.72.198.54 140.143.228.67 121.204.153.151 185.239.106.97 49.233.53.111 151.253.125.137 139.155.239.5 122.155.174.36 159.89.130.178 79.129.22.230 211.112.187.197 103.89.176.73 188.247.90.252 218.207.126.155 62.234.78.62 52.230.5.100 167.71.210.7 124.29.236.163 95.56.228.2 120.53.225.81 115.78.96.99 106.13.102.197 101.32.204.213 180.96.11.20 106.53.114.90 187.134.6.186 180.76.231.121 101.32.29.69 123.177.19.13 118.69.183.237 186.146.76.2 49.235.70.92 175.100.138.49 122.152.211.187 49.235.121.129 185.123.164.54 195.9.225.238 210.206.92.137 111.230.231.196 103.123.8.75 159.148.65.178 138.197.66.68 68.183.94.63 104.214.51.167 68.183.218.177 119.45.212.145 119.45.199.200 206.189.213.126 58.233.251.109 35.203.2.131 150.158.153.133 177.12.227.131 117.66.172.199 49.255.93.10 49.234.99.246 58.20.54.143 165.232.122.135 52.117.5.212 206.189.225.85 185.123.164.52 49.235.1.23 34.96.167.190 134.73.73.117 98.1.76.155 120.48.7.156 180.76.238.69 106.52.181.236 87.251.122.178 54.232.169.18 182.253.122.21 119.29.216.238 163.172.29.120 123.234.7.109 122.51.199.69 111.229.26.25 189.126.202.121 175.24.62.199 111.231.62.217 51.68.91.191 103.8.119.166 104.248.176.46 119.147.42.82 168.196.96.37 138.197.19.166 14.118.212.107 119.28.61.72 101.32.34.76 175.24.67.124 148.70.129.112 101.36.126.34 123.206.62.112 117.141.253.80 176.235.192.133 104.41.34.120 104.131.46.166 94.57.252.133 106.55.34.241 49.233.117.138 173.249.57.42 123.206.219.211 121.204.164.89 88.49.31.242 183.230.91.108 122.51.248.146 103.19.110.39 202.144.157.70 42.192.38.253 67.207.94.180 45.250.37.174 218.25.130.220 208.109.11.34 143.110.191.207 212.70.149.54 64.227.124.181 177.200.82.126 128.199.152.105 206.189.171.239 178.128.52.226 81.68.126.101 188.166.227.153 201.151.6.30 45.55.176.173 106.55.161.202 178.128.52.193 1.36.210.62 35.245.33.180 164.90.215.39 167.114.237.46 107.23.8.251 200.73.130.226 209.97.163.175 81.69.47.35 61.189.43.58 106.52.98.89 157.230.96.51 27.156.4.179 101.36.112.191 200.206.81.154 37.59.58.8 124.167.226.214 49.232.99.75 128.199.143.19 101.99.81.44 180.250.247.45 81.68.217.130 45.32.125.199 114.143.100.102 167.160.184.74 113.108.88.69 119.45.138.160 52.170.248.124 177.37.71.40 206.189.180.178 140.249.17.94 157.245.81.223 212.47.241.15 123.30.249.49 212.70.149.85 182.61.40.124 68.202.138.246 106.13.125.64 164.132.107.245 119.29.195.187 49.232.194.141 203.213.66.170 197.248.2.229 106.52.135.118 161.35.51.18 116.247.81.99 1.85.0.98 122.51.189.179 180.76.171.20 36.91.175.11 5.88.135.45 128.199.177.52 186.147.160.189 118.25.133.121 220.86.227.220 64.227.25.222 64.52.85.184 106.12.242.58 161.97.81.176 49.247.204.78 62.162.111.153 106.75.9.115 161.35.70.5 49.232.135.77 114.67.77.148 139.59.1.159 111.229.191.150 183.6.107.20 142.4.9.189 49.232.146.132 120.53.249.99 37.152.185.237 103.90.228.121 49.234.55.143 180.215.216.226 167.99.78.74 218.56.11.236 157.245.178.61 106.13.137.49 51.68.58.73 144.7.116.1 139.59.81.182 152.136.229.197 81.68.100.138 206.189.47.102 103.130.214.181 20.191.119.145 69.47.161.24 62.234.213.198 221.139.0.56 193.112.108.135 222.199.52.3 188.131.173.58 119.45.22.71 119.29.182.185 217.182.78.195 159.65.245.182 153.120.166.129 172.81.242.201 128.199.171.106 188.166.20.136 14.32.164.26 206.189.200.15 154.8.148.102 221.7.194.253 104.41.9.98 49.233.15.54 150.136.172.1 142.93.217.163 129.226.177.170 191.235.111.159 198.211.109.242 101.32.26.159 176.25.37.117 203.195.157.137 41.111.133.103 165.227.62.103 220.132.75.140 87.225.104.160 167.172.98.89 182.23.93.140 42.118.242.189 69.51.16.248 119.28.74.68 222.240.228.67 46.101.4.230 51.83.41.120 123.127.244.100 59.120.113.240 51.15.118.15 178.62.96.23 125.124.209.123 119.45.193.252 111.125.70.22 218.78.50.164 178.62.224.24 51.254.102.19 119.28.75.244 118.24.236.121 196.15.211.91 103.44.254.177 49.233.204.30 132.232.43.111 60.190.6.134 113.141.165.235 138.197.222.97 121.135.237.32 45.7.196.77 150.109.148.108 119.45.9.65 81.69.255.130 161.35.72.39 68.183.115.108 218.92.175.102 119.45.246.214 129.211.42.153 123.206.255.181 113.31.117.79 81.69.15.6 117.27.147.135 81.182.254.124 201.48.192.60 61.135.136.184 120.48.21.252 66.110.114.178 152.136.110.246 183.232.56.19 180.76.232.167 20.64.0.165 206.189.151.151 111.229.158.93 119.28.178.61 120.204.196.137 104.131.63.47 40.70.12.248 192.144.160.53 123.130.112.86 58.87.69.15 61.155.233.227 62.234.8.154 103.242.166.5 119.45.134.38 120.195.23.27 134.122.69.7 149.202.169.202 119.45.50.126 114.67.199.133 83.48.101.184 149.56.110.240 160.251.11.43 179.66.108.217 106.55.13.61 174.138.16.90 106.12.220.84 95.111.231.86 59.8.91.185 159.65.152.201 49.232.166.52 192.144.237.48 187.255.153.91 106.52.96.173 187.12.167.85 80.211.28.26 186.4.176.242 5.160.243.153 87.188.125.211 91.121.30.154 109.194.174.78 68.183.140.19 109.86.226.133 37.59.123.166 109.244.35.32 159.65.84.183 139.199.25.110 119.45.175.143 106.245.95.116 178.124.177.180 125.212.233.50 148.223.120.122 104.131.181.76 119.29.13.97 122.170.109.61 134.175.89.31 122.170.5.123 218.82.136.106

Those are the repeat offenders!

schiwe · January 1, 2021, 6:27am

This is a big list indeed
What is the service they have banned on?

pluto · January 1, 2021, 9:24am

Mainly SSH. And this is running on a non-standard port too! Its common this time of year for hackers to throw everything they have at servers while the server admins are off duty. Just a reminder to keep vigilant!

schiwe · January 1, 2021, 4:40pm

Well, SSH (in my opinion) Should be closed for the public.

Lupu · January 1, 2021, 9:42pm

Are you using the IP Lists (ipset) function added ~recently in Hestia? (Server-Firewall-Manage-iplist)

If not you can take a look here: https://github.com/hestiacp/hestiacp/pull/819 and use them to setup iptables DENY/ALLOW rules and reduce most of these attacks

pluto · January 2, 2021, 6:11am

I’m not complaining. I understand this is the way of the internet, having been defending servers for over 20 years. I was just surprised at the ferocity of the attack this year, and thought I’d let everyone know.
schiwe: Ideally yes, but unfortunately often not possible, eg if using ansible. SSH itself is safe enough when configured for no passwords, key auth only, but the main problem is log noise.
Lupu: Unfortunately the IP addresses change daily. I fail2ban recidive jail does as good a job as any of banning them, and I increased the ban time to 10 days. And my personal IP address which I access from is dynamic, so I can’t really DENY all and then ALLOW me. Have thought about port knocking occasionally, or having a central SSH jump box occasionally. But never got around to it.

Happy New Year.

Lupu · January 2, 2021, 9:49am

Happy new year.

It seems like you are not using the ipset blocklist, you will be surprised how much of the noise goes away after setting it up

My config looks like this:

country-ro : Only connection from IPs in my country are allowed (Romania) to 22(SSH) and 8083(Hestia)
blacklist : Compiled list of known attackers / spammers, currently ~70k prefixes listed, all traffic from these IPs are dropped

Hestia

$> v-list-firewall-ipset 
LISTNAME    IP_VERSION  AUTOUPDATE  SUSPENDED  SOURCE                                                            TIME      DATE
----        ------      -----       ----       --                                                                ----      ----
country-ro  v4          yes         no         http://ipverse.net/ipblocks/data/countries/ro.zone                00:01:02  2021-01-02
blacklist   v4          yes         no         script:/usr/local/hestia/install/deb/firewall/ipset/blacklist.sh  00:01:19  2021-01-02

$> v-list-firewall
RULE  ACTION  PROTO  PORT            IP                SPND  DATE
----  ------  -----  ----            --                ----  ----
2     ACCEPT  TCP    8083            ipset:country-ro  no    2020-06-03
..
13    ACCEPT  TCP    22              ipset:country-ro  no    2020-06-02
..
15    DROP    TCP    0               ipset:blacklist   no    2020-06-20

Stats

fail2ban-RECIDIVE and fail2ban-SSH iptables chains are empty pretty much all the time (JUMP and RETURN packets are also equal)

$> iptables -n -v -L
Chain INPUT (policy DROP 447K packets, 27M bytes)
 pkts bytes target     prot opt in     out     source               destination         
30283 5115K fail2ban-HESTIA  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8083
 772K  302M fail2ban-SSH  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22
 107M  227G fail2ban-RECIDIVE  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 1:65535     
 565K   27M DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set blocklist src
   87  5076 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 match-set country-ro src
 1726 95320 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8083 match-set country-ro src

Chain fail2ban-RECIDIVE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 107M  227G RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain fail2ban-SSH (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 772K  302M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain fail2ban-HESTIA (1 references)
 pkts bytes target     prot opt in     out     source               destination         
30283 5115K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

# Output from this server
$> grep -i "Failed" /var/log/auth.log{,.1} | wc -l
0

# Output from an older Vestacp server without ipset protection 
$> grep -i "Failed" /var/log/auth.log{,.1} | wc -l
196947

pluto · January 3, 2021, 5:36am

I see! I didn’t realise there were country based lists available that I could use. I will indeed look into this approach for some of my servers. Thanks for taking the time to explain.

kpv · January 5, 2021, 6:15pm

I use a very similar approach to Lupu’s, only allowing inbound traffic to sensitive ports (like ssh tcp/22 and hestia tcp/8083) from my country by “white-listing” certain IP ranges using country geoip ipsets.

However the iptables ruleset posted above by Lupu is not optimized (which may not be a problem if you don’t have much traffic or have a beefy server), since each and every packet from every interface goes through several chains.

Lupu · January 6, 2021, 4:22am

No realy, netfilter rules are not evaluated on every packet, but only when a new conntrack connection is established the following packets are permitted trough the ‘accept established,related’ rule.

Secondly, we are not using iptables to traverse the prefixes in blacklists in this case, but ipset lists which are using a very optimised data structure that does not have to check/traverse all the prefixes when doing a lookup.

Difference in lookup cost O(n) -iptables vs O(1) -ipset

pluto · January 6, 2021, 6:06am

I’ve been reading around, and it seems that the performance overhead on a ipset rule with x IP addresses is about the same as an IPtables rule, which is quite impressive.

Lupu, quick question … I’m now curious as to what that blacklist script is blocking. Is there a way to add logging to that rule on the fly? (And turn it off again after a couple of days). I’m familiar with how to tinker with rsyslog, but I’m hesitant to mess with firewalls unless I know exactly what I’m doing … Been locked out once too often.

andrewnewby · January 6, 2021, 9:51am

I’m trying to work out if I am missing a step here. I have enabled 3 lists:

 v-list-firewall-ipset
LISTNAME        IP_VERSION  AUTOUPDATE  SUSPENDED  SOURCE                                                            TIME      DATE
----            ------      -----       ----       --                                                                ----      ----
spammer1        v4          yes         no         script:/usr/local/hestia/install/deb/firewall/ipset/blacklist.sh  00:10:07  2021-01-06
russiaspammers  v4          yes         no         http://ipverse.net/ipblocks/data/countries/ru.zone                00:10:08  2021-01-06
china           v4          yes         no         http://ipverse.net/ipblocks/data/countries/cn.zone                00:10:08  2021-01-06

Yet I don’t see it in iptables:

iptables --list -n | grep src

I also don’t see it on:

v-list-firewall
RULE  ACTION  PROTO  PORT            IP             SPND  DATE
----  ------  -----  ----            --             ----  ----
1     ACCEPT  ICMP   0               0.0.0.0/0      no    2014-09-16
2     ACCEPT  TCP    9183            0.0.0.0/0      no    2014-05-25
3     ACCEPT  TCP    143,993         0.0.0.0/0      no    2014-05-25
4     ACCEPT  TCP    110,995         0.0.0.0/0      no    2014-05-25
5     ACCEPT  TCP    25,465,587      0.0.0.0/0      no    2018-11-07
6     ACCEPT  TCP    53              0.0.0.0/0      no    2014-05-25
7     ACCEPT  UDP    53              0.0.0.0/0      no    2014-05-25
8     ACCEPT  TCP    21,12000-12100  0.0.0.0/0      no    2014-05-25
9     ACCEPT  TCP    80,443          0.0.0.0/0      no    2014-09-24
10    ACCEPT  TCP    22              0.0.0.0/0      no    2014-09-16
13    ACCEPT  TCP    2812            81.174.134.33  no    2020-12-09
14    ACCEPT  TCP    35621,35623     0.0.0.0/0      no    2020-12-23
15    ACCEPT  UDP    35622           0.0.0.0/0      no    2020-12-12
16    ACCEPT  TCP    55413,55415     0.0.0.0/0      no    2020-12-15
17    ACCEPT  TCP    55414           81.174.134.33  no    2020-12-15

Have I missed something?

Lupu · January 6, 2021, 10:34am

You have to add the firewall rule now, in which you specify the Port,protocol(tcp/udp) and the ACTION (allow/deny). Use the Ip dropdown to select one of the defined lists

eris · January 6, 2021, 10:41am

See here for an example

andrewnewby · January 6, 2021, 11:17am

Ahhh nice - not sure why I assumed it would just add it on its own (especially seeing as I’ve had to add my own srcset for maltrail)

Will give that a go in a bit

andrewnewby · January 6, 2021, 1:59pm

How do you set wildcard for the ports? Leave empty, or 0?

eris · January 6, 2021, 2:01pm

All ports is 0

schiwe · January 6, 2021, 5:01pm

I recommend using firehol level 3 to block any malicious IPs get access to the server. *Be careful with the list. It might have a false positive IPs.

AlexZ · January 12, 2021, 8:57am

Why not change SSH port to smth like 22122 =) I did an no ssh-attempts at all.

Raphael · January 12, 2021, 10:32am

I just limit the ssh access to my trusted networks, changing ssh port to something else will not prevent port scanners - but will prevent the “standard attackers”.