I’m hoping someone can help explain what causes this and how to resolve. I’ve noticed this error message in my apache log file (/var/log/apache2/error.log) since I built a new server with HestiaCP. These errors have appeared consistently over the past 3 weeks. They all appear to originate from either Russia or China, so I assume they are bots or someone trying to hack the server.
Can someone please help me understand what causes this and if it’s normal or how to address it. Thanks!
But honestly, I can’t follow or understand what they are talking about.
In case it helps, all of the websites that run on my server are Wordpress.
I’m looking for someone that has experienced this issue and knows how to resolve it. I would greatly appreciate if someone can provide the steps to resolve this issue. E.g. Is there a setting that needs to be changed on the server, some text that needs to be added to a .htaccess file. The linked article is a bit all over the place and references both APACH + NGINX and Apache only and is not very clear on what the actual issue is.
yes. random bots throwing random requests at your server/domains hoping for finding some known vulnerability. they don’t care what you have on there, they are simply trying to find one of a million holes. welcome to the public internet. get used to it. ignore it. make sure you don’t use insecure software/plugins/etc.
if you want to cut down the noise a bit, you can try adding some ipsets against bad bots and known malicious IPs. there also are some filters you might add to fail2ban that can be triggered by such errors and ban IPs for a while.
the issue still will always simply be that running a public server/website attracts that kind of blind attacks. you cannot ban them all anyway.
Which speaks about an issue with .htaccess files and after an upgrade of php-fpm. I never had these errors before in my logs when I was running VestaCP on Apache. The HestiaCP server is slightly different in that it’s using php-fpm. So that’s why I think it may be something related to the .htaccess files or the configuration since I moved to this new server.
I came across another forum post on DigitalOcean, also referencing a potential issue with configuration issues with Apache or .htaccess file: Understanding apache.error.log message? | DigitalOcean. Unfortuantely, no answer was provided there.
So I’m just trying to rule this out. I can’t really mess with the .htaccess file as its all auto generated from Wordpress. So I’m trying to understand what’s causing this and how to solve it. If it’s legit traffic, I would hate to have it not serving the pages to valid users.
If it’s just noise, I can definitely ignore it, but it seems that the other people commenting on these posts are saying it’s an apache configuration issue. That’s why I’m trying to rule that out.
the thing the guy is doing there is simply changing the priorities of the rewrite rules by moving them around and he has a certain path involved, which might really exist on his system and the error in his specific case most likely was really related to that.
in your case you still see random foreign requests searching for anything. yes of course the error message might have changed due to changing the way apache is handling requests for php files now. when it was prefork, apache would already try to find the file and then run it internally via mod-php. now it simply passes the request on to php-fpm and won’t care much about the rest.
I am pretty sure it’s not. simply match the timestamp and IP towards the requests in the related access.log to see what the full request was. most likely you will see random stuff that cannot exist on your system or is at least not directly accessible for a reason…
@falzo - thank you for the very informative response. A lot of good knowledge in there. Actually, that was the other question I forgot to ask. How do I figure out where this error is coming from. There is not much detail in the error itself and I can’t seem to tell which domain/site is even being accessed that is causing this error. The error is in the /var/log/apache2/error.log
From the above entry, how can I tell which access.log to check?
cat /var/log/apache2/access.log shows an empty file. So I assume you mean I should check the domain/website log files.
I realized these files are in the domains folder under sitename.tld.log, but is there a way to know which ones I should check based on the above error message? Or I just need to go through and check one by one. It seems there should be a better way.
the your.domain.com.log files in the domains folder are the access logs per domain…
apart from that the easiest way to start investigating is running something like:
find /var/log/apache2 -type f -exec grep '12.34.56.78' {} \; -print
this should get you all the entries related to an offending IP and also print which file they are in. once you found that you can analyse in more detail…
PS: in general, the related access log should be on the same level as the error log where you found the message. so it’s either under domains in the according domain.error.log or if in the general error.log in apache2 folder the request probably wasn’t even specific to an existing web-domain.
So it seems there is no entry for that IP address in any of the access logs, only in the error logs. Also I made sure to test the find command by using an IP address I see hitting one of my domains in one of the domain.tld.log file and it came back with the expected results.
So it seems there is no access logged by this IP address. Why would that be? I will test a few others from the error.log file that I found to see if they also turn up nothing.
But as you say, if it’s just noise and I can ignore it (i.e. it’s not an issue with configuration or .htaccess files), I will just ignore it going forward when I see it in log files.
if it is not in the error log of the domain itself, it is far away from having anything to do with your htaccess. because that means exactly that: it does not even have a domain related vhost involved.
it is probably some request hitting the IP directly and not even a domain name. or an old domain name of a former owner of that IP that might still point there but does not exist in your setup. that’s why it ends up in the general apache2 log.
I just checked and /var/log/apache2/access.log is empty on my servers too. logging at that level is probably just not configured or turned off on purpose because everything there is wrong traffic anyway. to be fair I am too lazy to dig that up now. it’s noise and I rather prefer to not have that noise fill up my logfiles on top.
Thanks @falzo! I will ignore it. I appreciate yours and eris wealth of knowledge.
And that makes more sense, given I changed the IP address of the server, when I built the new DigitalOcean droplet for this HestiaCP server.
It’s a bit late now, but if I were to do it again, I think it would have been better to build the server and then restore the image onto the existing server IP address. As I have had that IP for many years. I notice that Gmail is rate limiting some of the email I am forwarding over to them. I don’t know if this is because of the new IP address on the server or other things I need to check, but perhaps it’s related.
Anyway, another battle for another day. Thanks so much!
most likely. IPs earn a reputation (good or bad) over time, especially when it comes to mail. forwarding on top is a rather bad practise nowadays because your system will also forward spam mails and for gmail the origin of these spam mails are your IP and therefore it will hurt its reputation.
but as you said, that definitely is a totally different battle and an even more complex one.
Indeed, that’s why I typically implement spam assassin with strict rules and I have the server delete anything with a spam score over 10. This way it doesn’t get forwarded on to gmail. I had that battle in the past and I think it resolved itself. I’m guessing it could be just because this is a new IP address and I need to improve the reputation.
Btw, while I was investigating the above issue, I noticed that my apache error.log was flooded with these phpmyadmin failed to load errors. There is more than 100 of them and all within seconds of each other.
I assume this is some kind of brute-force attack? Is this just noise or something I should look into? Thanks!
[Mon May 03 23:57:10.064910 2021] [proxy_fcgi:error] [pid 508395:tid 139630275958528] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:11.928087 2021] [proxy_fcgi:error] [pid 508395:tid 139630423525120] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:17.449325 2021] [proxy_fcgi:error] [pid 508396:tid 139630275958528] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:20.438889 2021] [proxy_fcgi:error] [pid 508396:tid 139630166853376] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:23.523171 2021] [proxy_fcgi:error] [pid 508395:tid 139630250780416] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:30.932029 2021] [proxy_fcgi:error] [pid 508396:tid 139630431917824] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:33.716355 2021] [proxy_fcgi:error] [pid 508396:tid 139630259173120] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:34.829349 2021] [proxy_fcgi:error] [pid 508395:tid 139630183638784] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:35.815561 2021] [proxy_fcgi:error] [pid 508396:tid 139630267565824] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:37.018177 2021] [proxy_fcgi:error] [pid 508396:tid 139630301136640] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:40.856900 2021] [proxy_fcgi:error] [pid 508395:tid 139630284351232] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:42.908818 2021] [proxy_fcgi:error] [pid 508396:tid 139630225602304] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
[Mon May 03 23:57:43.865481 2021] [proxy_fcgi:error] [pid 508396:tid 139630175246080] [client 78.138.43.43:0] AH01071: Got error 'PHP message: phpmyadmin: Failed to load /var/lib/phpmyadmin/config.inc.php Check group www-data has read access and open_basedir restrictions.'
