Error 500 regularly and temporaly solution

floown · September 21, 2020, 4:58pm

Hello,
I just made the update to 1.3.4 (and reboot because the kernel has updated too) but a error 500 occur on my domain (and his webmail)
Message :
“Internal Server Error Oops! Something went wrong.The server encountered an internal error or misconfiguration and was unable to complete your request.Error Code: 500”

No panic, I go https://hostname.domain.tld:8083/edit/web/?domain=domain.tld and control that php-fpm has selection for php 7.3, then save.

The 500 has gone.

I regularly have to make sure that this template is properly selected, and save, because this error 500 occurs very regularly.

What is it coming from, please?

OS : Ubuntu 20.04.01
Default HestiaCP installation

eris · September 21, 2020, 5:19pm

Does the template change when you have issue?

floown · September 21, 2020, 5:25pm

I have not yet optimized some elements (php.ini, etc) and the templates are the originals provided during installation.

falzo · September 22, 2020, 5:59pm

does not have to be an issue with the templates itself. maybe after the reboot one of the services like e.g. php-fpm does not get started for whatever reason. selecting/rebuilding the template issues a reload/restart and therefore this could solve the problem.

you should check your log files to find the real reason for the 500 if it happens more regularly.

Raphael · September 22, 2020, 6:11pm

Just for the records: the upgrade to 1.2.4 didnt run a rebuild config, it was just to replace the v-update-letsencrypt-ssl script - so probaly @falzo is right.

floown · September 22, 2020, 6:24pm

Hello,

Here are the last errors:
2020/09/21 18:48:21 [error] 7277#7277: *20 connect() failed (111: Connection refused) while connecting to upstream, client: 84.104.178.145, server: mydomain.tld, request: “GET /favicon.ico HTTP/2.0”, upstream: “https://161.145.112.84:8443/favicon.ico”, host: “www.mydomain.tld”
2020/09/22 17:34:53 [crit] 21007#21007: *669 SSL_write() failed while reading client request headers, client: 93.174.93.12, server: mydomain.tld, request: “GET / HTTP/1.1”
2020/09/22 19:52:34 [error] 6828#6828: *30 connect() failed (111: Connection refused) while connecting to upstream, client: 216.244.66.238, server: mydomain.tld, request: “GET / HTTP/1.1”, upstream: “https://161.145.112.84:8443/”, host: “mydomain.tld”
2020/09/22 20:03:48 [error] 6828#6828: *37 connect() failed (111: Connection refused) while connecting to upstream, client: 128.14.134.170, server: mydomain.tld, request: “GET / HTTP/1.1”, upstream: “https://161.145.112.84:8443/”, host: “163.172.106.81”
2020/09/22 20:11:55 [error] 6828#6828: *44 connect() failed (111: Connection refused) while connecting to upstream, client: 83.145.164.114, server: mydomain.tld, request: “GET / HTTP/2.0”, upstream: “https://161.145.112.84:8443/”, host: “mydomain.tld”
2020/09/22 20:12:03 [error] 6828#6828: *44 connect() failed (111: Connection refused) while connecting to upstream, client: 83.145.164.114, server: mydomain.tld, request: “GET / HTTP/2.0”, upstream: “https://161.145.112.84:8443/”, host: “mydomain.tld”
2020/09/22 20:15:46 [error] 6828#6828: *48 connect() failed (111: Connection refused) while connecting to upstream, client: 83.145.164.114, server: mydomain.tld, request: “GET / HTTP/2.0”, upstream: “https://161.145.112.84:8443/”, host: “www.mydomain.tld”

I have rebuild the site, and it works now. I monitor if the errors has gone. Thanks for your help and have a nice evening.

falzo · September 22, 2020, 6:27pm

yeah these look like nginx not being able to connect to apache2. so maybe your apache2 did not start after reboot? next time this happens check which processes are running or not, nginx / apache2 / php-fpm - to see what might be missing

maybe even reboot again and watch the output in a vnc if available. you should see if something gets stuck and journalctl -xe might give additional hints.

floown · October 19, 2020, 8:41am

Hello,

Then I have just noticed that the phenomenon occurred again, apache2 did not restart following the update that I have just carried out on my server. I just launched it via the control panel and it’s ok. It’s a default installation on a Ubuntu 20.04 LTS base, so with php-fpm. I had not noticed any error during the installation of Hestia.

How can I correct this behavior manually? Thank you.

eris · October 19, 2020, 8:50am

With out any log information we don’t know. systemctl status apache and journalctl -xe could be usefull

Maybe do some digging inside the logs…

floown · October 19, 2020, 8:59am

$ systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-10-19 10:34:31 CEST; 18min ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 1013548 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 1013562 (apache2)
Tasks: 56 (limit: 38214)
Memory: 12.1M
CGroup: /system.slice/apache2.service
├─1013562 /usr/sbin/apache2 -k start
├─1013564 /usr/sbin/apache2 -k start
├─1013566 /usr/sbin/apache2 -k start
└─1013567 /usr/sbin/apache2 -k start

Oct 19 10:34:31 hostname.domain.com systemd[1]: Starting The Apache HTTP Server…
Oct 19 10:34:31 hostname.domain.com systemd[1]: Started The Apache HTTP Server.

$ journalctl -xe
Oct 19 10:52:01 hostname.domain.com sudo[1017288]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 19 10:52:02 hostname.domain.com sudo[1017288]: pam_unix(sudo:session): session closed for user root
Oct 19 10:52:02 hostname.domain.com CRON[1017286]: pam_unix(cron:session): session closed for user admin
Oct 19 10:52:04 hostname.domain.com sshd[1017314]: Invalid user tin from 111.229.215.206 port 48694
Oct 19 10:52:04 hostname.domain.com sshd[1017314]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 10:52:04 hostname.domain.com sshd[1017314]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=111.229.215.206
Oct 19 10:52:06 hostname.domain.com sshd[1017314]: Failed password for invalid user tin from 111.229.215.206 port 48694 ssh2
Oct 19 10:52:11 hostname.domain.com sshd[1017382]: Unable to negotiate with 14.142.57.66 port 41570: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,dif>
Oct 19 10:52:28 hostname.domain.com named[536]: client @0x7f285c012cc0 159.138.80.11#24659 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:28 hostname.domain.com named[536]: client @0x7f285c012cc0 159.138.80.11#24660 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:28 hostname.domain.com named[536]: client @0x7f28740046e0 159.138.80.11#24661 (ns.domaindns.com): query (cache) ‘ns.domaindns.com/AAAA/IN’ denied
Oct 19 10:52:33 hostname.domain.com named[536]: client @0x7f28680211f0 114.119.176.141#32639 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:33 hostname.domain.com named[536]: client @0x7f285c012cc0 114.119.176.141#32640 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:33 hostname.domain.com named[536]: client @0x7f285800e650 114.119.176.141#32641 (ns.domaindns.com): query (cache) ‘ns.domaindns.com/AAAA/IN’ denied
Oct 19 10:52:38 hostname.domain.com named[536]: client @0x7f2860013dd0 159.138.80.11#24662 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:39 hostname.domain.com systemd-resolved[480]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature l>
Oct 19 10:52:39 hostname.domain.com systemd-resolved[480]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature l>
Oct 19 10:52:43 hostname.domain.com named[536]: client @0x7f2870000cd0 114.119.176.141#32642 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:53 hostname.domain.com named[536]: client @0x7f28680211f0 159.138.80.13#31992 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:53 hostname.domain.com named[536]: client @0x7f28640176a0 159.138.80.13#31993 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:54 hostname.domain.com named[536]: client @0x7f285c012cc0 159.138.80.13#32428 (ns.domaindns.com): query (cache) ‘ns.domaindns.com/AAAA/IN’ denied
Oct 19 10:52:54 hostname.domain.com named[536]: client @0x7f28680211f0 74.125.190.134#42303 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:54 hostname.domain.com named[536]: client @0x7f2860013dd0 74.125.190.151#54592 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:55 hostname.domain.com named[536]: client @0x7f28680211f0 159.138.80.13#32429 (www.domaindns.com): query (cache) ‘www.domaindns.com/A/IN’ denied
Oct 19 10:52:58 hostname.domain.com sshd[1017390]: Invalid user asb from 217.133.58.148 port 37413
Oct 19 10:52:58 hostname.domain.com sshd[1017390]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 10:52:58 hostname.domain.com sshd[1017390]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=217.133.58.148
Oct 19 10:53:00 hostname.domain.com sshd[1017390]: Failed password for invalid user asb from 217.133.58.148 port 37413 ssh2
Oct 19 10:53:17 hostname.domain.com sshd[1017521]: Unable to negotiate with 14.142.57.66 port 36194: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,dif>
Oct 19 10:53:24 hostname.domain.com sshd[1017523]: Invalid user ozi from 106.52.139.223 port 57656
Oct 19 10:53:24 hostname.domain.com sshd[1017523]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 10:53:24 hostname.domain.com sshd[1017523]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=106.52.139.223
Oct 19 10:53:25 hostname.domain.com sshd[1017523]: Failed password for invalid user ozi from 106.52.139.223 port 57656 ssh2
Oct 19 10:53:26 hostname.domain.com sshd[1017523]: Received disconnect from 106.52.139.223 port 57656:11: Bye Bye [preauth]
Oct 19 10:53:26 hostname.domain.com sshd[1017523]: Disconnected from invalid user ozi 106.52.139.223 port 57656 [preauth]
Oct 19 10:54:01 hostname.domain.com CRON[1017561]: pam_unix(cron:session): session opened for user admin by (uid=0)
Oct 19 10:54:01 hostname.domain.com CRON[1017562]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 19 10:54:01 hostname.domain.com sudo[1017563]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 19 10:54:01 hostname.domain.com sudo[1017563]: pam_unix(sudo:session): session closed for user root
Oct 19 10:54:01 hostname.domain.com CRON[1017561]: pam_unix(cron:session): session closed for user admin
Oct 19 10:54:20 hostname.domain.com sshd[1017589]: Invalid user lva from 165.227.96.127 port 44112
Oct 19 10:54:21 hostname.domain.com sshd[1017589]: pam_unix(sshd:auth): check pass; user unknown
Oct 19 10:54:21 hostname.domain.com sshd[1017589]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=165.227.96.127
Oct 19 10:54:22 hostname.domain.com sshd[1017589]: Failed password for invalid user lva from 165.227.96.127 port 44112 ssh2
Oct 19 10:54:23 hostname.domain.com sshd[1017589]: Received disconnect from 165.227.96.127 port 44112:11: Bye Bye [preauth]
Oct 19 10:54:23 hostname.domain.com sshd[1017589]: Disconnected from invalid user lva 165.227.96.127 port 44112 [preauth]

floown · October 19, 2020, 9:03am

The logctl entries are not good, it doesn’t cover the time of the update, and no restart. I need to specify which options to have it?

floown · December 9, 2020, 3:53pm

Ok, I have a warn issu related, perhaps.

I have peeled Apache logs and configuration, I have found that, after the reboot I done:
$ journalctl --no-pager -u apache2
– Reboot –
Dec 09 06:07:45 hostname.domain.com systemd[1]: Starting The Apache HTTP Server…
Dec 09 06:07:45 hostname.domain.com apachectl[568]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 160.160.160.60:8443
Dec 09 06:07:45 hostname.domain.com apachectl[568]: no listening sockets available, shutting down
Dec 09 06:07:45 hostname.domain.com apachectl[568]: AH00015: Unable to open logs
Dec 09 06:07:45 hostname.domain.com apachectl[533]: Action ‘start’ failed.
Dec 09 06:07:45 hostname.domain.com apachectl[533]: The Apache error log may have more information.
Dec 09 06:07:45 hostname.domain.com systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Dec 09 06:07:45 hostname.domain.com systemd[1]: apache2.service: Failed with result ‘exit-code’.
Dec 09 06:07:45 hostname.domain.com systemd[1]: Failed to start The Apache HTTP Server.

The port 8443 is already in use. Where I modify Apache (or what else?) to listen to another port?

Thanks.