500 Internal Server Error after Ubuntu update

himi · January 30, 2023, 8:06pm

Hello, I had HestiaCP setup and working perfectly for few of my websites, including SSL and everything on Ubuntu server. Unfortunately I ran Ubuntu update & upgrade earlier today and now I’m getting the mentioned error: “500 Internal Server Error”. Hestia CP itself runs fine but all websites are showing the site with yellow exclamation mark:

I admit I was a bit careless during the update, there were some updates regarding apache2 and other.

This is the complete list of all updates:

Start-Date: 2023-01-30  19:09:30
Commandline: apt-get upgrade
Requested-By: ubuntu (1001)
Upgrade: openssh-client:arm64 (1:8.9p1-3, 1:8.9p1-3ubuntu0.1), php8.0-soap:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), kbd:arm64 (2.3.0-3ubuntu4, 2.3.0-3ubuntu4.22.04), libsasl2-modules:arm64 (2.1.27+dfsg2-3ubuntu1, 2.1.27+dfsg2-3ubuntu1.1), libsasl2-2:arm64 (2.1.27+dfsg2-3ubuntu1, 2.1.27+dfsg2-3ubuntu1.1), php8.0-common:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-curl:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-imap:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-intl:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), openssh-server:arm64 (1:8.9p1-3, 1:8.9p1-3ubuntu0.1), libsasl2-modules-db:arm64 (2.1.27+dfsg2-3ubuntu1, 2.1.27+dfsg2-3ubuntu1.1), php8.0-opcache:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), apache2-suexec-custom:arm64 (2.4.54-1+ubuntu22.04.1+deb.sury.org+1, 2.4.55-1+ubuntu22.04.1+deb.sury.org+2), python-apt-common:arm64 (2.3.0ubuntu2.1, 2.4.0), apache2-data:arm64 (2.4.54-1+ubuntu22.04.1+deb.sury.org+1, 2.4.55-1+ubuntu22.04.1+deb.sury.org+2), php8.0-gd:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), apache2-bin:arm64 (2.4.54-1+ubuntu22.04.1+deb.sury.org+1, 2.4.55-1+ubuntu22.04.1+deb.sury.org+2), open-vm-tools:arm64 (2:11.3.5-1ubuntu4.1, 2:12.1.0-1~ubuntu0.22.04.1), update-manager-core:arm64 (1:22.04.9, 1:22.04.10), python3-apt:arm64 (2.3.0ubuntu2.1, 2.4.0), systemd-hwe-hwdb:arm64 (249.11.1, 249.11.2), php8.0-mbstring:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), apache2-utils:arm64 (2.4.54-1+ubuntu22.04.1+deb.sury.org+1, 2.4.55-1+ubuntu22.04.1+deb.sury.org+2), grub2-common:arm64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), php8.0-mysql:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-pspell:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), openssh-sftp-server:arm64 (1:8.9p1-3, 1:8.9p1-3ubuntu0.1), php8.0-readline:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), grub-common:arm64 (2.06-2ubuntu7, 2.06-2ubuntu7.1), php8.0-bcmath:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), python3-update-manager:arm64 (1:22.04.9, 1:22.04.10), apache2:arm64 (2.4.54-1+ubuntu22.04.1+deb.sury.org+1, 2.4.55-1+ubuntu22.04.1+deb.sury.org+2), php8.0-bz2:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-cli:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-ldap:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-fpm:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), php8.0-xml:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1), libpq5:arm64 (14.5-0ubuntu0.22.04.1, 14.6-0ubuntu0.22.04.1), php8.0-zip:arm64 (1:8.0.26-1+ubuntu22.04.1+deb.sury.org+1, 1:8.0.27-2+ubuntu22.04.1+deb.sury.org+1)
End-Date: 2023-01-30  19:13:03

So yes, some misconfiguration has happened and I cannot figure out what happened, why and where to fix it.

Following is the setting same for all websites in Hestia CP:

nginx log showed this after the update but no more:

2023/01/30 19:23:59 [emerg] 1190#1190: bind() to 10.0.0.227:80 failed (98: Address already in use)
2023/01/30 19:23:59 [emerg] 1190#1190: bind() to 10.0.0.227:443 failed (98: Address already in use)
2023/01/30 19:23:59 [emerg] 1190#1190: still could not bind()

Error log for one of the websites shows (in Hestia CP - binoculars - Error.log):

2023/01/30 21:01:34 [error] 8327#8327: *568 connect() failed (111: Connection refused) while connecting to upstream, client: 216.179.117.119, server: mywebsite.com, request: "GET /en/ HTTP/1.1", upstream: "https://10.0.0.227:8443/en/", host: "mywebsite.com"
2023/01/30 21:02:39 [error] 8327#8327: *575 connect() failed (111: Connection refused) while connecting to upstream, client: 38.97.137.99, server: mywebsite.com, request: "GET /en/ HTTP/1.1", upstream: "https://10.0.0.227:8443/en/", host: "mywebsite.com"
2023/01/30 21:03:17 [error] 8327#8327: *578 connect() failed (111: Connection refused) while connecting to upstream, client: 216.144.248.20, server: mywebsite.com, request: "GET / HTTP/1.1", upstream: "https://10.0.0.227:8443/", host: "mywebsite.com", referrer: "https://mywebsite.com"
2023/01/30 21:03:19 [error] 8327#8327: *580 connect() failed (111: Connection refused) while connecting to upstream, client: 37.143.116.232, server: mywebsite.com, request: "GET / HTTP/2.0", upstream: "https://10.0.0.227:8443/", host: "www.mywebsite.com"

systemctl status apache2 shows:

 apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-01-30 19:41:30 CET; 1h 27min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 1138 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
        CPU: 67ms

Jan 30 19:41:30 hcp.mydomain.com apachectl[1199]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using hcp.mydomain.com. Set the 'ServerName' directive globally to suppress this message
Jan 30 19:41:30 hcp.mydomain.com apachectl[1199]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
Jan 30 19:41:30 hcp.mydomain.com apachectl[1199]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
Jan 30 19:41:30 hcp.mydomain.com apachectl[1199]: no listening sockets available, shutting down
Jan 30 19:41:30 hcp.mydomain.com apachectl[1199]: AH00015: Unable to open logs
Jan 30 19:41:30 hcp.mydomain.com apachectl[1138]: Action 'start' failed.
Jan 30 19:41:30 hcp.mydomain.com apachectl[1138]: The Apache error log may have more information.
Jan 30 19:41:30 hcp.mydomain.com systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Jan 30 19:41:30 hcp.mydomain.com systemd[1]: apache2.service: Failed with result 'exit-code'.
Jan 30 19:41:30 hcp.mydomain.com systemd[1]: Failed to start The Apache HTTP Server.

Could you please help me and advise what should I check and what to fix?
I am already considering reinstalling the whole server, use Hestia backup and restore the data, as the last resort.

Many thanks!

eris · January 30, 2023, 8:12pm

himi · January 30, 2023, 8:17pm

Yes, exactly my case. Running HCP in Oracle cloud, I most likely did the same mistake, selecting the wrong option. Getting the same message.

How to resolve this?

Raphael · January 30, 2023, 8:24pm

himi · January 30, 2023, 8:36pm

I removed the original file /etc/apache2/apache2.conf with the content of what you posted above but the result is still the same after server reboot:

systemctl status apache2.service

× apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-01-30 21:31:26 CET; 4min 12s ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 1196 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
        CPU: 70ms

Jan 30 21:31:26 hcp.mydomain.com apachectl[1219]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using hcp.mydomain.com. Set the 'ServerName' directive globally to>
Jan 30 21:31:26 hcp.mydomain.com apachectl[1219]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
Jan 30 21:31:26 hcp.mydomain.com apachectl[1219]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
Jan 30 21:31:26 hcp.mydomain.com apachectl[1219]: no listening sockets available, shutting down
Jan 30 21:31:26 hcp.mydomain.com apachectl[1219]: AH00015: Unable to open logs
Jan 30 21:31:26 hcp.mydomain.com apachectl[1196]: Action 'start' failed.
Jan 30 21:31:26 hcp.mydomain.com apachectl[1196]: The Apache error log may have more information.
Jan 30 21:31:26 hcp.mydomain.com systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Jan 30 21:31:26 hcp.mydomain.com systemd[1]: apache2.service: Failed with result 'exit-code'.
Jan 30 21:31:26 hcp.mydomain.com systemd[1]: Failed to start The Apache HTTP Server.

himi · January 30, 2023, 8:49pm

According to several other topics I found there’s a conflict of nginx and apache2, both want to use port 80 and 443.

Using this to find out what is using port 80:

sudo ss -4 -tlnp | grep 80
sudo ss -6 -tlnp | grep 80

Yes, it’s nginx but what is the correct configuration to get it working again?

eris · January 30, 2023, 8:56pm

Follow the steps in the the link I posted…

himi · January 30, 2023, 9:09pm

Oh, wow, many thanks! It worked!
What I did: listing all files in apache2 directory I noticed there are also two files apache2.conf.dpkg-old
and ports.conf.dpkg-old from the day I originally installed Hestia CP. So I simply removed the current apache2.conf and ports.conf. Then I used the dpkg-old files and used them as current ones.
Restarted the Apache and now it works.
Thank you! Thank you! Thank you!!!

MAN5 · February 5, 2023, 2:17pm

If you apache2 manual update (not by hestiaCP…)…
simply make it empty the file " ports.conf".

As of Hestiacp v1.6.14 - This file is just blank.