SYN Flood attack on hestiacp

Hello,

for the past few weeks I’ve been struggling with ddos attacks on client server.

Right now they’re trying synflood method on port 8443.

Site is going down every few minutes.

Normally it’s 300 connections ++++

Firewall rules:

/etc/sysctl.conf

vm.vfs_cache_pressure=50
vm.dirty_background_ratio = 5
vm.dirty_ratio = 80
vm.max_map_count = 655300
fs.file-max = 2097152
fs.nr_open = 2097152

kernel.printk = 4 4 1 7
kernel.panic = 10
kernel.sysrq = 0
kernel.shmmax = 4294967296
kernel.shmall = 4194304
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
net.core.netdev_max_backlog = 262144
net.core.rmem_default = 31457280
net.core.rmem_max = 67108864
net.core.wmem_default = 31457280
net.core.wmem_max = 67108864
net.core.somaxconn = 65535
net.core.optmem_max = 25165824
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv4.neigh.default.gc_interval = 5
net.ipv4.neigh.default.gc_stale_time = 120
net.netfilter.nf_conntrack_max = 10000000
net.netfilter.nf_conntrack_tcp_loose = 0
net.netfilter.nf_conntrack_tcp_timeout_established = 1800
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 20
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 20
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 10
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_no_pmtu_disc = 1
net.ipv4.route.flush = 1
net.ipv4.route.max_size = 8048576
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 4096 87380 33554432
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 400000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1

Any idea what might help in this scenario?

Regards

Hi @Anotherbanana

You could also try to limit the number of SYN packets using iptables.

Create the file /usr/local/hestia/data/firewall/custom.sh and add these to that file:

#!/usr/bin/env bash
IPTABLES="/usr/sbin/iptables"
"$IPTABLES" -F syn_flood 2>/dev/null
"$IPTABLES" -X syn_flood 2>/dev/null
"$IPTABLES" -N syn_flood
"$IPTABLES" -I INPUT -p tcp --syn -j syn_flood
"$IPTABLES" -A syn_flood -m limit --limit 2/s --limit-burst 10 -j RETURN
"$IPTABLES" -A syn_flood -j DROP

Note: the important part here is --limit 2/s --limit-burst 10 and that means that after 10 syn packets received from the same ip it will block the ip if it send again 2 packets whitin a second. This can be modified to fit your needs.

Save the file, add execution perms to custom.sh script and update firewall rules:

chmod +x /usr/local/hestia/data/firewall/custom.sh
v-update-firewall

Let us know whether it helps.

I developed a script that analyzes the IP addresses accessing the system. This script checks each IP, and if there are more than 50 connections from the same address, it classifies the IP accordingly. It identifies whether the IP is national, international, or local (from the internal network).

  • National IP: The address is added to a specific list for national IPs.
  • International IP: The address is included in a list for international IPs.
  • Local IP: The address is placed in a list for local IPs.

This script can be set to run automatically using cron, at intervals of 5 minutes or less, while respecting the query policy of the freeipapi.com service.

#!/bin/bash

# Checks if jq is installed
if ! command -v jq &> /dev/null
then
    echo "jq is not installed. Please install jq to use this script."
    exit 1
fi

# Defines the output files
br_file="/brIp.txt"
inter_file="/interIp.txt"
local_file="/ipLocal.txt"

# List of IPs to ignore
ignored_ips=("8.8.8.8" "8.8.4.4")

# Collects the IPs from the netstat command output
ips=$(netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | awk '$1 > 50 {print $2}' | grep -vE "$(ip addr show | awk '/inet / {split($2, a, "/"); print a[1]}' | paste -sd '|' -)")

for ip_address in $ips; do
  # Checks if the IP is in the ignore list
  skip=false
  for ignored_ip in "${ignored_ips[@]}"; do
    if [ "$ip_address" == "$ignored_ip" ]; then
      #echo "IP $ip_address is in the ignore list. Skipping check."
      skip=true
      break
    fi
  done

  if [ "$skip" == true ]; then
    continue
  fi

  # Checks if the IP is already in one of the lists
  if grep -q "$ip_address" "$br_file" || grep -q "$ip_address" "$inter_file" || grep -q "$ip_address" "$local_file"; then
    #echo "IP $ip_address has already been checked. Skipping check."
    continue
  fi

  # Uses curl to make the HTTP GET request
  url="https://freeipapi.com/api/json/$ip_address"
  response=$(curl -s $url)

  # Checks the request status
  if [ $? -ne 0 ]; then
    echo "Failed to retrieve information for IP $ip_address. Skipping classification."
    continue
  fi

  # Extracts the country code from the JSON response using jq if the request was successful
  country_code=$(echo $response | jq -r '.countryCode')

  # Classifies the IP based on the country code
  if [ "$country_code" == "BR" ]; then
    echo "$ip_address" >> "$br_file"
  elif [ "$country_code" == "-" ]; then
    echo "$ip_address" >> "$local_file"
  else
    echo "$ip_address" >> "$inter_file"
  fi

  echo "IP $ip_address successfully classified."
done

After implementing the script, you can add the path of the generated list directly to Ipset in the Hestia panel.

Then, create a firewall rule to block this list.

{06CCEA41-D418-4C24-A26D-F0627DCB256A}

I hope this was helpful.

2 Likes

Hello,

thanks for your input. This solution banned redis server unfortunately. I’ll try with @molero.renan solution now, but it’ll take me a while to understand the code. Will share result later.

1 Like

I’m not sure how to use this script. I placed it in my home directory.
Did chmod +x script.sh
but there’s no output whatsoever. I’m not sure where I’m wrong.
jq installed by default on ubuntu if I’m not wrong.

What I did so far is banned 8443 port completely. Banned http 1.0, and http 1.1 completely.
Don’t know why, but when I was doing this from hestiacp firewall panel it wasn’t blocking this port.
I have to change it in hestiacp firewall and did v-update-firewall. Not it’s blocked and when needed I’ll open this port again, and close after.

Right now they’re trying with simple ddos on homepage and I’m banning them using nginx rate limiting.

I’m still interested in making your script @molero.renan working. Any follow up advice for me?

Regards

Are these paths correct?

Defines the output files

br_file=“/brIp.txt”
inter_file=“/interIp.txt”
local_file=“/ipLocal.txt”

You need to define these paths and create these files if they don’t already exist. Remember, the script will only query IPs with more than 50 simultaneous connections. For testing, you may reduce this threshold.

Hello

Applied both methods (adjusted them a little) and still no luck. I’m getting over 250 connections and server is timing out.

I can’t understand why it’s going down with only 250connections +.

Below you can find my /etc/sysctl.conf
Maybe someone have idea how to tweak this.
They’re only ddosing 443 port and I can’t close it. All traffic is going through Cloudflare but IPs don’t have any pattern.

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 3 4 1 3

###################################################################
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
# See http://lwn.net/Articles/277146/



# Note: This may impact IPv6 TCP sessions too

#HERE

#net.ipv4.tcp_syncookies=1
#net.ipv4.tcp_max_syn_backlog = 2048
#net.ipv4.tcp_synack_retries = 3



# Uncomment the next line to enable packet forwarding for IPv4
#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#  Enabling this option disables Stateless Address Autoconfiguration
#  based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1


###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1
#

###################################################################
# Magic system request Key
# 0=disable, 1=enable all, >1 bitmask of sysrq functions
# See https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
# for what other values do
#kernel.sysrq=438

#net.core.somaxconn = 65535
#net.core.netdev_max_backlog = 65535




#START


net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv=30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=180
net.ipv4.tcp_fin_timeout=20
net.ipv4.tcp_synack_retries=5
net.ipv4.tcp_keepalive_time=1800
net.ipv4.tcp_sack=0
net.ipv4.tcp_max_tw_buckets=1440000
net.ipv4.tcp_max_syn_backlog=4096
net.ipv4.tcp_keepalive_intvl=40
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.inet_peer_gc_maxtime=240
net.ipv4.inet_peer_maxttl=500
net.ipv4.inet_peer_minttl=80
net.ipv4.ip_local_port_range=32768 61001
net.ipv4.tcp_dsack=0
net.ipv4.tcp_ecn=0
net.ipv4.tcp_fack=0
net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_max_orphans=524288
net.ipv4.tcp_no_metrics_save=1
net.ipv4.tcp_retries2=10
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_syncookies=2
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_window_scaling=1
fs.file-max = 100000


@molero.renan
I edited your script to be like:

ips=$(netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | awk '$1 > 5 {print $2}' | grep -vE "$(ip addr show | awk '/inet / {split($2, a, "/"); print a[1]}' | paste -sd '|' -)")

and yours @sahsanu to

#!/usr/bin/env bash
IPTABLES="/usr/sbin/iptables"
"$IPTABLES" -F syn_flood 2>/dev/null
"$IPTABLES" -X syn_flood 2>/dev/null
"$IPTABLES" -N syn_flood
"$IPTABLES" -I INPUT -p tcp --syn -j syn_flood
"$IPTABLES" -A syn_flood -m limit --limit 5/s --limit-burst 30 -j RETURN
"$IPTABLES" -A syn_flood -j DROP

Port 443 is used for the HTTPS certificate and cannot be blocked.

When you run the code I sent you, is it adding the IPs to the lists?

br_file="/brIp.txt"
inter_file="/interIp.txt"
local_file="/ipLocal.txt"

If yes, did you add the list you want to block in the iptables as shown in the image I sent you?

Yes, it’s adding IPs to lists:

root@redacted:~# cat interIp.txt 
162.158.222.165
162.158.222.212
162.158.222.213
162.158.222.32
162.158.222.33
162.158.222.91
162.158.33.213
172.70.162.121
172.70.163.167
172.70.58.3
172.70.58.4
172.70.58.43
172.70.58.44
172.70.91.34
146.70.119.49
172.70.86.119
172.68.205.35

It’s cloudflares IPs though.

Did I mention it’s overkill server for just one wordpress site?
It’s 72 cores, 94gb ram dedicated server with only 1 site on it.

It’s total overkill so I’m wondering why it’s timing out with only 250 + connections.
I’m checking connections with tcptrack and with this script

#!/bin/sh
OS=$(uname)

case "$OS" in
    'SunOS')
            AWK=/usr/bin/nawk
            ;;
    'Linux')
            AWK=/bin/awk
            ;;
    'AIX')
            AWK=/usr/bin/awk
            ;;
esac

netstat -an | $AWK -v start=1 -v end=65535 ' $NF ~ /TIME_WAIT|ESTABLISHED/ && $4 !~ /127\.0\.0\.1/ {
    if ($1 ~ /\./)
            {sip=$1}
    else {sip=$4}

    if ( sip ~ /:/ )
            {d=2}
    else {d=5}

    split( sip, a, /:|\./ )

    if ( a[d] >= start && a[d] <= end ) {
            ++connections;
            }
    }
    END {print connections}'

250+ connections is causing timeout for site

Wow. 72cores for 1 site.

That’s some serious overkill.

I’ll try to measure EXACTLY what your “Port Scan” script is doing.
Simple math shows that 250 connections TIMES 65,000 ports is a LOT more traffic than you think.

Are you trying to tell me that it’s 250*65000?
16250000 connections?

How to fight with this kind of traffic?

My cloud provider is putting firewall in front of this server right now, but it’ll take time ( till Monday).

Is there any other way? Port 80 and 443 is only accessible via cloudflare.

Regards