Running Hestia v1.8.11 + Debian 10 (Buster). After apt update/upgrade all wordpress sites show Internal Server Error. /var/log/nginx/error.log reveals “110: Connection time out while connecting to upstream…” errors.
Apache2 status shows running. Nginx status shows running.
We are in full panic mode over here. Lots of sites down. Any help is appreciated. Where to look???
Further research and testing point to PHP-FPM being the issue. Sites run HTML files fine, but any simple/basic .php page is throwing the internal server error. Any advice on this?
Take a look to this post just in case you are also using mpm_prefork instead of mpm_event in Apache2
Hey sahsanu, thanks for chiming in. Is there a way to detect if I am using mpm_prefork? All of the sites on this server are actually on PHP7.4.
In the other post are all the commands but I paste them here too:
The last apache2ctl command is the one to see the modules used by Apache2:
php -v
lsof -Pn +c0 -i:9000 -sTCP:LISTEN
find /etc/php/ -name "www.conf"
ls -l /etc/apache2/conf.d/
dpkg -l | grep libapache2-mod-php
apache2ctl -M
php -v
PHP 7.4.33 (cli) (built: Sep 4 2023 08:15:35) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies
lsof -Pn +c0 -i:9000 -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
php-fpm7.4 723 root 16u IPv4 17151 0t0 TCP 127.0.0.1:9000 (LISTEN)
find /etc/php/ -name "www.conf"
/etc/php/7.4/fpm/pool.d/www.conf
ls -l /etc/apache2/conf.d/
total 16
-rw-r--r-- 1 root root 540 May 22 2023 172.26.8.223.conf
drwxr-xr-x 2 root root 4096 Mar 7 15:50 domains
-rw-r--r-- 1 root root 130 Feb 17 2021 hestia-event.conf
-rw-r--r-- 1 root root 1408 Nov 25 2021 phpmyadmin.inc
dpkg -l | grep libapache2-mod-php
[returned nothing]
apache2ctl -M
bash: apache2ctl: command not found
/usr/sbin/apache2ctl -M
Loaded Modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
fcgid_module (shared)
filter_module (shared)
status_module (shared)
mime_module (shared)
mpm_event_module (shared)
negotiation_module (shared)
proxy_module (shared)
proxy_fcgi_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
suexec_module (shared)
All looks good, show the output of this command:
grep -i '^include' /etc/apache2/apache2.conf
grep -i '^include' /etc/apache2/apache2.conf
Include mods-enabled/*.load
Include mods-enabled/*.conf
Include ports.conf
IncludeOptional conf.d/*.conf
IncludeOptional conf.d/domains/webmail.*.conf
IncludeOptional conf.d/domains/*.conf
I’m noticing the absence of “php_module (shared)” from the apache2ctl command. Issue?
The apache2.conf looks good too. Regarding php module, you don’t need it.
PHP is clearly the issue. If I put this into a site’s root, it won’t run…
<!DOCTYPE html>
<html>
<head>
<title>PHP Test</title>
</head>
<body>
<?php echo '<p>Hello World</p>'; ?>
</body>
</html>
But standard, vanilla HTML works fine.
lsof -Pn +c0 -i -sTCP:LISTEN | grep -Ei 'apache|nginx'
lsof -Pn +c0 -i -sTCP:LISTEN | grep -Ei 'apache|nginx'
nginx 990 root 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 990 root 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 990 root 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
apache2 5625 www-data 3u IPv4 1189773 0t0 TCP 127.0.0.1:8081 (LISTEN)
apache2 5625 www-data 4u IPv4 1189775 0t0 TCP 172.26.8.223:8443 (LISTEN)
apache2 5625 www-data 5u IPv4 1189777 0t0 TCP 172.26.8.223:8080 (LISTEN)
apache2 5635 www-data 3u IPv4 1189773 0t0 TCP 127.0.0.1:8081 (LISTEN)
apache2 5635 www-data 4u IPv4 1189775 0t0 TCP 172.26.8.223:8443 (LISTEN)
apache2 5635 www-data 5u IPv4 1189777 0t0 TCP 172.26.8.223:8080 (LISTEN)
apache2 5636 www-data 3u IPv4 1189773 0t0 TCP 127.0.0.1:8081 (LISTEN)
apache2 5636 www-data 4u IPv4 1189775 0t0 TCP 172.26.8.223:8443 (LISTEN)
apache2 5636 www-data 5u IPv4 1189777 0t0 TCP 172.26.8.223:8080 (LISTEN)
nginx 5798 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5798 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5798 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5799 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5799 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5799 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5800 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5800 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5800 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5801 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5801 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5801 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5802 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5802 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5802 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5803 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5803 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5803 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5804 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5804 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5804 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
nginx 5805 www-data 108u IPv4 21575 0t0 TCP 172.26.8.223:80 (LISTEN)
nginx 5805 www-data 109u IPv4 21576 0t0 TCP 172.26.8.223:443 (LISTEN)
nginx 5805 www-data 110u IPv4 21577 0t0 TCP 127.0.0.1:8084 (LISTEN)
apache2 15547 root 3u IPv4 1189773 0t0 TCP 127.0.0.1:8081 (LISTEN)
apache2 15547 root 4u IPv4 1189775 0t0 TCP 172.26.8.223:8443 (LISTEN)
apache2 15547 root 5u IPv4 1189777 0t0 TCP 172.26.8.223:8080 (LISTEN)
hestia-nginx 23671 root 6u IPv4 100809 0t0 TCP *:8083 (LISTEN)
hestia-nginx 23673 admin 6u IPv4 100809 0t0 TCP *:8083 (LISTEN)
I should also add, the Hestia CP web control panel is working flawlessly. Which I think is PHP, no?
Looks fine too.
apache2ctl configtest
nginx -t
systemctl status --no-pager -l php7.4-fpm
systemctl status --no-pager -l nginx
systemctl status --no-pager -l apache2
And show the complete error you see in nginx error.log, also, check if you see errors inside /var/log/apache2/
Yes, but it is using its own nginx and php version.
/usr/sbin/apache2ctl configtest
Syntax OK
/usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
systemctl status --no-pager -l php7.4-fpm
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-03-07 17:24:51 EST; 2h 17min ago
Docs: man:php-fpm7.4(8)
Main PID: 723 (php-fpm7.4)
Status: "Processes active: 0, idle: 0, Requests: 0, slow: 0, Traffic: 0req/sec"
Tasks: 1 (limit: 4915)
Memory: 28.7M
CGroup: /system.slice/php7.4-fpm.service
└─723 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
Mar 07 17:24:50 [mydomain].com systemd[1]: Starting The PHP 7.4 FastCGI Process Manager...
Mar 07 17:24:51 [mydomain].com systemd[1]: Started The PHP 7.4 FastCGI Process Manager.
systemctl status --no-pager -l nginx
● nginx.service - nginx - high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-03-07 17:24:52 EST; 2h 18min ago
Docs: https://nginx.org/en/docs/
Process: 5754 ExecReload=/bin/sh -c /bin/kill -s HUP $(/bin/cat /var/run/nginx.pid) (code=exited, status=0/SUCCESS)
Main PID: 990 (nginx)
Tasks: 10 (limit: 4915)
Memory: 94.5M
CGroup: /system.slice/nginx.service
├─ 990 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
├─5798 nginx: worker process
├─5799 nginx: worker process
├─5800 nginx: worker process
├─5801 nginx: worker process
├─5802 nginx: worker process
├─5803 nginx: worker process
├─5804 nginx: worker process
├─5805 nginx: worker process
└─5806 nginx: cache manager process
Mar 07 17:40:14 [mydomain].com systemd[1]: /lib/systemd/system/nginx.service:8: PIDFile= references path below legacy directory /var/run/, updating /var/run/nginx.pid → /run/nginx.pid; please update the unit file accordingly.
Mar 07 17:40:14 [mydomain].com systemd[1]: /lib/systemd/system/nginx.service:8: PIDFile= references path below legacy directory /var/run/, updating /var/run/nginx.pid → /run/nginx.pid; please update the unit file accordingly.
Mar 07 17:40:14 [mydomain].com systemd[1]: /lib/systemd/system/nginx.service:8: PIDFile= references path below legacy directory /var/run/, updating /var/run/nginx.pid → /run/nginx.pid; please update the unit file accordingly.
Mar 07 17:40:15 [mydomain].com systemd[1]: /lib/systemd/system/nginx.service:8: PIDFile= references path below legacy directory /var/run/, updating /var/run/nginx.pid → /run/nginx.pid; please update the unit file accordingly.
Mar 07 18:07:28 [mydomain].com systemd[1]: Reloading nginx - high performance web server.
Mar 07 18:07:28 [mydomain].com systemd[1]: Reloaded nginx - high performance web server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloading nginx - high performance web server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloaded nginx - high performance web server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloading nginx - high performance web server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloaded nginx - high performance web server.
systemctl status --no-pager -l apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-03-07 17:41:44 EST; 2h 2min ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 15511 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Process: 5492 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
Main PID: 15547 (apache2)
Tasks: 56 (limit: 4915)
Memory: 33.8M
CGroup: /system.slice/apache2.service
├─ 5625 /usr/sbin/apache2 -k start
├─ 5635 /usr/sbin/apache2 -k start
├─ 5636 /usr/sbin/apache2 -k start
└─15547 /usr/sbin/apache2 -k start
Mar 07 17:41:44 [mydomain].com systemd[1]: Starting The Apache HTTP Server...
Mar 07 17:41:44 [mydomain].com systemd[1]: Started The Apache HTTP Server.
Mar 07 18:07:28 [mydomain].com systemd[1]: Reloading The Apache HTTP Server.
Mar 07 18:07:28 [mydomain].com systemd[1]: Reloaded The Apache HTTP Server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloading The Apache HTTP Server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloaded The Apache HTTP Server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloading The Apache HTTP Server.
Mar 07 18:07:29 [mydomain].com systemd[1]: Reloaded The Apache HTTP Server.
Seeing lots of errors in the /var/log/nginx/error.log like this:
2024/03/07 17:27:56 [error] 991#991: *79 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.166.204, server: _, request: "GET /FdLf HTTP/1.1", upstream: "http://172.26.8.223:8080/FdLf", host: "54.166.112.170:80"
2024/03/07 17:28:56 [error] 991#991: *106 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.166.204, server: _, request: "GET /x3nD HTTP/1.1", upstream: "http://172.26.8.223:8080/x3nD", host: "54.166.112.170:80"
2024/03/07 18:21:13 [error] 5798#5798: *1292 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.152.66, server: _, request: "GET /VhUI HTTP/1.1", upstream: "http://172.26.8.223:8080/VhUI", host: "54.166.112.170:80"
2024/03/07 18:22:13 [error] 5798#5798: *1320 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.152.66, server: _, request: "GET /JbhI HTTP/1.1", upstream: "http://172.26.8.223:8080/JbhI", host: "54.166.112.170:80"
2024/03/07 19:16:26 [error] 5798#5798: *2771 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.165.113, server: _, request: "GET /dqgqoeCXckuwPtxov HTTP/1.1", upstream: "http://172.26.8.223:8080/dqgqoeCXckuwPtxov", host: "54.166.112.170"
2024/03/07 19:17:26 [error] 5798#5798: *2800 upstream timed out (110: Connection timed out) while connecting to upstream, client: 106.75.165.113, server: _, request: "GET /dqgqoeCXckuwPtxov HTTP/1.1", upstream: "http://172.26.8.223:8080/dqgqoeCXckuwPtxov", host: "54.166.112.170"
Last handful of errors in /var/log/apache2/error.log:
[Thu Mar 07 18:07:28.773287 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00493: SIGUSR1 received. Doing graceful restart
[Thu Mar 07 18:07:28.852777 2024] [ssl:warn] [pid 15547:tid 139756538131584] AH01909: 172.26.8.223:443:0 server certificate does NOT include an ID which matches the server name
[Thu Mar 07 18:07:28.853213 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00489: Apache/2.4.58 (Debian) mod_fcgid/2.3.9 OpenSSL/1.1.1n configured -- resuming normal operations
[Thu Mar 07 18:07:28.853230 2024] [core:notice] [pid 15547:tid 139756538131584] AH00094: Command line: '/usr/sbin/apache2'
[Thu Mar 07 18:07:29.277176 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00493: SIGUSR1 received. Doing graceful restart
[Thu Mar 07 18:07:29.355669 2024] [ssl:warn] [pid 15547:tid 139756538131584] AH01909: 172.26.8.223:443:0 server certificate does NOT include an ID which matches the server name
[Thu Mar 07 18:07:29.356070 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00489: Apache/2.4.58 (Debian) mod_fcgid/2.3.9 OpenSSL/1.1.1n configured -- resuming normal operations
[Thu Mar 07 18:07:29.356088 2024] [core:notice] [pid 15547:tid 139756538131584] AH00094: Command line: '/usr/sbin/apache2'
[Thu Mar 07 18:07:29.696956 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00493: SIGUSR1 received. Doing graceful restart
[Thu Mar 07 18:07:29.775644 2024] [ssl:warn] [pid 15547:tid 139756538131584] AH01909: 172.26.8.223:443:0 server certificate does NOT include an ID which matches the server name
[Thu Mar 07 18:07:29.776057 2024] [mpm_event:notice] [pid 15547:tid 139756538131584] AH00489: Apache/2.4.58 (Debian) mod_fcgid/2.3.9 OpenSSL/1.1.1n configured -- resuming normal operations
[Thu Mar 07 18:07:29.776074 2024] [core:notice] [pid 15547:tid 139756538131584] AH00094: Command line: '/usr/sbin/apache2'
I really appreciate all of your help! I am very confused as to what happened…
Are those [ssl:warn] errors in the apache2 error log the culprit?