How do you handle bad bots?

1

I am currently getting hammered by bad bots, I use a WordPress plugin bad bots pro on some of my websites but still having a hard time dealing with them. I would prefer something that is not PHP related, and it’s handled by the server directly. Some of my websites are full dynamic, so I am unable to use fastcgi cache. Also I am unable to use a security CDN, so what do you use on the server side?

2

I would maybe start with the ipset blacklist and see, if you have less bot activity: Firewall | Hestia Control Panel

1 Like
3

This might be a bit of overkill, but it works at the nginx level.

The install script doesn’t interfere with hestia config, but of course backup your /etc/nginx directory first.
Once installed globally you enable it for each site that needs it with a single config line. Or is it two. Anyway, not hard.

2 Likes
4

Simpler approach might be to have a rule like this and insert it where needed.

if ($http_user_agent ~* “SemrushBot|Semrush|AhrefsBot|MJ12bot|YandexBot|+++++add your own here++++|linkfluence.com|TweetmemeBot|LinkisBot”) { return 403; }

5

IPset blacklist and eventually have a look at 8G Firewall (Apache only until now, NGINX a little later - use 7G Firewall for NGINX until then)

3 Likes
6

Cloudflare is highly customizable and has fairly easy WAF rules.

7

To handle the bad bots at the server is to use the server side security tool like Fail2Ban. It can monitor the log files and for suspicious activity and automatically block the IP address for malicious behavior including those associated with bad bots.

8

I have 7G nginx installed serverwide. It is highly advisable.

9

Hi Pluto, does this work fine along with 7G Firewall?

For simplicity I am considering to put both to work.

10

I haven’t tried them together. I’d say use one or the other, as there is a lot of duplication between the two. 7g/8g is probably good enough for most uses, and pretty easy to troubleshoot when you activate the logging. I needed this recently when the 8G version I installed on a server started blocking all URLs with ‘login’ in them, which seemed like a bit of an oversight.
The bad bot scripts are more complicated, and introduce different levels of treatment, such as rate-limiting some bots and completely banning others, so if this is something that appeals to you, then maybe switch to that. But my feeling is that running both would be unncessarily complicating things.

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.