Okay, so I figured out the cause. It was due to Cloudflare. I use Cloudflare’s built in proxy service, so my server IP is not shown to the visitors. You can enable this within Cloudflare in their DNS settings.
However, I still don’t understand why having the proxy setting on, prevents the server and iptables from blocking the IP address. If anyone can help me understand that, I would appreciate it. It seems even though I am using Cloudflare, my server can see the external user’s IP address that is probing my server and it successfully identifies the IP, bans it, and then adds it to iptables to block. However, it doesn’t get blocked due to the Cloudflare’s proxy. This is the part I don’t understand.
Anyway, the way I solved this was by using Cloudflare’s API. This website explains it quite well and it’s super quick and easy.
It just involves updating the Cloudflare action file in fail2ban, adding your global API key and email address and then adding in the cloudflare action into each of your fail2ban jails. Once I did that, my problem was solved. In this case, the IP gets banned and it calls the Cloudflare API to ban the IP at the Cloudflare level.
Anyway, I hope this helps the next person that is scratching their head wondering why the hell their server isn’t blocking the IPs. I went through and checked everything but Cloudflare, which is why it took me so longer to figure this out. My logic was, since the server can see the external user’s IP and successfully ban it, it can’t possibly by something related to Cloudflare. But apparently it is and it has to do with their proxy service. I still don’t understand why, but happy that I got this figured out and my server is back in business blocking these annoying aholes.