Hacky New Year Everyone

From one of my Hestia servers …

Status for the jail: recidive
| |- Currently failed: 28
| |- Total failed: 6262
|- Currently banned: 719
|- Total banned: 1048
`- Banned IP list:

Those are the repeat offenders!

1 Like

This is a big list indeed :wink:
What is the service they have banned on?

Mainly SSH. And this is running on a non-standard port too! Its common this time of year for hackers to throw everything they have at servers while the server admins are off duty. Just a reminder to keep vigilant!

Well, SSH (in my opinion) Should be closed for the public.

Are you using the IP Lists (ipset) function added ~recently in Hestia? (Server-Firewall-Manage-iplist)

If not you can take a look here: https://github.com/hestiacp/hestiacp/pull/819 and use them to setup iptables DENY/ALLOW rules and reduce most of these attacks

1 Like

I’m not complaining. I understand this is the way of the internet, having been defending servers for over 20 years. I was just surprised at the ferocity of the attack this year, and thought I’d let everyone know.
schiwe: Ideally yes, but unfortunately often not possible, eg if using ansible. SSH itself is safe enough when configured for no passwords, key auth only, but the main problem is log noise.
Lupu: Unfortunately the IP addresses change daily. I fail2ban recidive jail does as good a job as any of banning them, and I increased the ban time to 10 days. And my personal IP address which I access from is dynamic, so I can’t really DENY all and then ALLOW me. Have thought about port knocking occasionally, or having a central SSH jump box occasionally. But never got around to it.

Happy New Year.

Happy new year.

It seems like you are not using the ipset blocklist, you will be surprised how much of the noise goes away after setting it up :wink:

My config looks like this:

  • country-ro : Only connection from IPs in my country are allowed (Romania) to 22(SSH) and 8083(Hestia)
  • blacklist : Compiled list of known attackers / spammers, currently ~70k prefixes listed, all traffic from these IPs are dropped


$> v-list-firewall-ipset 
LISTNAME    IP_VERSION  AUTOUPDATE  SUSPENDED  SOURCE                                                            TIME      DATE
----        ------      -----       ----       --                                                                ----      ----
country-ro  v4          yes         no         http://ipverse.net/ipblocks/data/countries/ro.zone                00:01:02  2021-01-02
blacklist   v4          yes         no         script:/usr/local/hestia/install/deb/firewall/ipset/blacklist.sh  00:01:19  2021-01-02
$> v-list-firewall
RULE  ACTION  PROTO  PORT            IP                SPND  DATE
----  ------  -----  ----            --                ----  ----
2     ACCEPT  TCP    8083            ipset:country-ro  no    2020-06-03
13    ACCEPT  TCP    22              ipset:country-ro  no    2020-06-02
15    DROP    TCP    0               ipset:blacklist   no    2020-06-20


fail2ban-RECIDIVE and fail2ban-SSH iptables chains are empty pretty much all the time (JUMP and RETURN packets are also equal)

$> iptables -n -v -L
Chain INPUT (policy DROP 447K packets, 27M bytes)
 pkts bytes target     prot opt in     out     source               destination         
30283 5115K fail2ban-HESTIA  tcp  --  *      *              tcp dpt:8083
 772K  302M fail2ban-SSH  tcp  --  *      *              tcp dpt:22
 107M  227G fail2ban-RECIDIVE  tcp  --  *      *              multiport dports 1:65535     
 565K   27M DROP       tcp  --  *      *              match-set blocklist src
   87  5076 ACCEPT     tcp  --  *      *              tcp dpt:22 match-set country-ro src
 1726 95320 ACCEPT     tcp  --  *      *              tcp dpt:8083 match-set country-ro src

Chain fail2ban-RECIDIVE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 107M  227G RETURN     all  --  *      *             

Chain fail2ban-SSH (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 772K  302M RETURN     all  --  *      *             

Chain fail2ban-HESTIA (1 references)
 pkts bytes target     prot opt in     out     source               destination         
30283 5115K RETURN     all  --  *      *  
# Output from this server
$> grep -i "Failed" /var/log/auth.log{,.1} | wc -l

# Output from an older Vestacp server without ipset protection 
$> grep -i "Failed" /var/log/auth.log{,.1} | wc -l


I see! I didn’t realise there were country based lists available that I could use. I will indeed look into this approach for some of my servers. Thanks for taking the time to explain.

I use a very similar approach to Lupu’s, only allowing inbound traffic to sensitive ports (like ssh tcp/22 and hestia tcp/8083) from my country by “white-listing” certain IP ranges using country geoip ipsets.

However the iptables ruleset posted above by Lupu is not optimized (which may not be a problem if you don’t have much traffic or have a beefy server), since each and every packet from every interface goes through several chains.

No realy, netfilter rules are not evaluated on every packet, but only when a new conntrack connection is established the following packets are permitted trough the ‘accept established,related’ rule.

Secondly, we are not using iptables to traverse the prefixes in blacklists in this case, but ipset lists which are using a very optimised data structure that does not have to check/traverse all the prefixes when doing a lookup.

Difference in lookup cost O(n) -iptables vs O(1) -ipset

I’ve been reading around, and it seems that the performance overhead on a ipset rule with x IP addresses is about the same as an IPtables rule, which is quite impressive.

Lupu, quick question … I’m now curious as to what that blacklist script is blocking. Is there a way to add logging to that rule on the fly? (And turn it off again after a couple of days). I’m familiar with how to tinker with rsyslog, but I’m hesitant to mess with firewalls unless I know exactly what I’m doing … Been locked out once too often. :slight_smile:

1 Like

I’m trying to work out if I am missing a step here. I have enabled 3 lists:

LISTNAME        IP_VERSION  AUTOUPDATE  SUSPENDED  SOURCE                                                            TIME      DATE
----            ------      -----       ----       --                                                                ----      ----
spammer1        v4          yes         no         script:/usr/local/hestia/install/deb/firewall/ipset/blacklist.sh  00:10:07  2021-01-06
russiaspammers  v4          yes         no         http://ipverse.net/ipblocks/data/countries/ru.zone                00:10:08  2021-01-06
china           v4          yes         no         http://ipverse.net/ipblocks/data/countries/cn.zone                00:10:08  2021-01-06

Yet I don’t see it in iptables:

iptables --list -n | grep src

I also don’t see it on:

RULE  ACTION  PROTO  PORT            IP             SPND  DATE
----  ------  -----  ----            --             ----  ----
1     ACCEPT  ICMP   0           no    2014-09-16
2     ACCEPT  TCP    9183        no    2014-05-25
3     ACCEPT  TCP    143,993      no    2014-05-25
4     ACCEPT  TCP    110,995      no    2014-05-25
5     ACCEPT  TCP    25,465,587      no    2018-11-07
6     ACCEPT  TCP    53          no    2014-05-25
7     ACCEPT  UDP    53          no    2014-05-25
8     ACCEPT  TCP    21,12000-12100      no    2014-05-25
9     ACCEPT  TCP    80,443      no    2014-09-24
10    ACCEPT  TCP    22          no    2014-09-16
13    ACCEPT  TCP    2812    no    2020-12-09
14    ACCEPT  TCP    35621,35623      no    2020-12-23
15    ACCEPT  UDP    35622       no    2020-12-12
16    ACCEPT  TCP    55413,55415      no    2020-12-15
17    ACCEPT  TCP    55414   no    2020-12-15

Have I missed something?

You have to add the firewall rule now, in which you specify the Port,protocol(tcp/udp) and the ACTION (allow/deny). Use the Ip dropdown to select one of the defined lists

See here for an example

Ahhh nice - not sure why I assumed it would just add it on its own :smile: (especially seeing as I’ve had to add my own srcset for maltrail)

Will give that a go in a bit

How do you set wildcard for the ports? Leave empty, or 0?

All ports is 0


I recommend using firehol level 3 to block any malicious IPs get access to the server. *Be careful with the list. It might have a false positive IPs.

1 Like

Why not change SSH port to smth like 22122 =) I did an no ssh-attempts at all.

I just limit the ssh access to my trusted networks, changing ssh port to something else will not prevent port scanners - but will prevent the “standard attackers”.