SSL/TLS and Port 8443 connection issues on fresh HestiaCP installation - Ubuntu Server 22.04 LTS

FIRST PART

All,

Hope you are well and you can give me some hints.

Description:
I’m facing an issue with my production server, which has just been freshly installed and it’s running HestiaCP v1.8.12 on Ubuntu Server 22.04 LTS (fully patched).

Everything is properly set up and configured (including obviously all the necessary DNS records in my third-party DNS provider, so let me anticipate that this is not a DNS related issue), yet I’m unable to load the webmail web page https://webmail.mydomain.com (this is just a dummy name as you can see) for any newly created mail domain although a valid SSL certificate has been issued for each one of them with the built-in Let’s Encrypt integration confirming that there’s no DNS related issue on their FQDNs.

I also have a staging server with the very same version of HestiaCP running and, again, with Ubuntu Server 22.04 LTS (previously deployed but fully patched as well). Both of them are running in two different instances of Oracle OCI.

One more insight: all the rules created in the security list associated to the public subnet where my production server is hooked (within the Oracle related vcn) are the very same ones as the ones for the staging environment and I’m simply replacing my old instance of HestiaCP (which was working fine).

Let me anticipate also here that names you see in the terminal have been changed from the original ones but the rest of information are the original ones.

At the moment I only have two mail domains (I created the second one to see if the issue was just happening on the first one only but it’s not and I will reserve to create all the other ones once this issue is resolved):

The minute I click on the little icon to load the related web page:

image

This is one valid mail account for the first domain indicated in the above screenshot:

I see this:

Troubleshooting steps done so far

  1. Telnet to webmail.mymaildomain.com (fake name) are fine from any IPv4 public address in the world.

  2. Checked Nginx configuration file. I found this files om /etc/nginx/conf.d/domains:
    • webmail.mymaildomain.com.conf
    • webmail.mymaildomain.com.ssl.conf

  3. I opened the second one and I found this (not sure if there’s anything wrong):

# cat webmail.mymaildomain.com.ssl.conf
server {
        listen      10.0.1.10:443 ssl;
        server_name webmail.mymaildomain.com mail.mymaildomain.com;
        root        /var/lib/roundcube;
        index       index.php index.html index.htm;
        access_log  /var/log/nginx/domains/webmail.mymaildomain.com.log combined;
        error_log   /var/log/nginx/domains/webmail.mymaildomain.com.error.log error;

        ssl_certificate     /home/mymaildomain/conf/mail/mymaildomain.com/ssl/mymaildomain.com.pem;
        ssl_certificate_key /home/mymaildomain/conf/mail/mymaildomain.com/ssl/mymaildomain.com.key;
        ssl_stapling        on;
        ssl_stapling_verify on;

        # TLS 1.3 0-RTT anti-replay
        if ($anti_replay = 307) { return 307 https://$host$request_uri; }
        if ($anti_replay = 425) { return 425; }

        location ~ /\.(?!well-known\/) {
                deny all;
                return 404;
        }

        location ~ ^/(README.md|config|temp|logs|bin|SQL|INSTALL|LICENSE|CHANGELOG|UPGRADING)$ {
                deny all;
                return 404;
        }

        location / {
                alias /var/lib/roundcube/;

                try_files $uri $uri/ =404;

                proxy_pass https://10.0.1.10:8443;

                location ~* ^.+\.(ogg|ogv|svg|svgz|swf|eot|otf|woff|woff2|mov|mp3|mp4|webm|flv|ttf|rss|atom|jpg|jpeg|gif|png|webp|ico|bmp|mid|midi|wav|rtf|css|js|jar)$ {
                        expires 7d;
                        fastcgi_hide_header "Set-Cookie";
                }
        }

        location @fallback {
                proxy_pass https://10.0.1.10:8443;
        }

        location /error/ {
                alias /var/www/document_errors/;
        }

        proxy_hide_header Upgrade;

        include /home/mymaildomain/conf/mail/mymaildomain.com/nginx.ssl.conf_*;
  1. This is the content of the “/etc/apache2/sites-available” folder and there’s no virtual host file related to webmail.mymaildomain.com
# ll
total 28
drwxr-xr-x  2 root root 4096 Oct 15 08:05 ./
drwxr-xr-x 10 root root 4096 Oct 14 20:09 ../
-rw-r--r--  1 root root 1286 Jul 18 03:56 000-default.conf
-rw-r--r--  1 root root   20 Oct 14 20:01 default
-rw-r--r--  1 root root   20 Oct 14 20:01 default-ssl
-rw-r--r--  1 root root 4573 Jul 18 03:56 default-ssl.conf
  1. Checked the content of the “/var/lib/roundcube” folder and everything looks normal (as in my staging server):
# ll
total 408
drwxr-xr-x 11 hestiamail www-data   4096 Oct 14 20:08 ./
drwxr-xr-x 57 root       root       4096 Oct 14 20:08 ../
-rw-r--r--  1 hestiamail www-data   2553 Oct 14 20:08 .htaccess
-rw-r--r--  1 hestiamail www-data 214982 Oct 14 20:08 CHANGELOG.md
-rw-r--r--  1 hestiamail www-data  12661 Oct 14 20:08 INSTALL
-rw-r--r--  1 hestiamail www-data  35147 Oct 14 20:08 LICENSE
-rw-r--r--  1 hestiamail www-data   3853 Oct 14 20:08 README.md
-rw-r--r--  1 hestiamail www-data    967 Oct 14 20:08 SECURITY.md
drwxr-xr-x  7 hestiamail www-data   4096 Oct 14 20:08 SQL/
-rw-r--r--  1 hestiamail www-data   4657 Oct 14 20:08 UPGRADING
drwxr-xr-x  2 hestiamail www-data   4096 Oct 14 20:08 bin/
-rw-r--r--  1 hestiamail www-data    994 Oct 14 20:08 composer.json
-rw-r--r--  1 hestiamail www-data   1086 Oct 14 20:08 composer.json-dist
-rw-r--r--  1 hestiamail www-data  56784 Oct 14 20:08 composer.lock
lrwxrwxrwx  1 hestiamail www-data     15 Oct 14 20:08 config -> /etc/roundcube//
-rw-r--r--  1 hestiamail www-data  11199 Oct 14 20:08 index.php
drwxr-xr-x  2 hestiamail www-data   4096 Oct 14 20:08 logs/
drwxr-xr-x 37 hestiamail www-data   4096 Oct 14 20:08 plugins/
drwxr-xr-x  8 hestiamail www-data   4096 Oct 14 20:08 program/
drwxr-xr-x  3 hestiamail www-data   4096 Oct 14 20:08 public_html/
-rw-r--r--  1 hestiamail www-data     26 Oct 14 20:08 robots.txt
drwxr-xr-x  3 hestiamail www-data   4096 Oct 14 20:08 skins/
drwxr-xr-x  2 hestiamail www-data   4096 Oct 14 20:08 temp/
drwxr-xr-x 14 hestiamail www-data   4096 Oct 14 20:08 vendor/
  1. This is the list of ports where local processes are listening from in the affected server:
# netstat -ntupla | grep 443
tcp        0      0 10.0.1.10:8443          0.0.0.0:*               LISTEN      56337/apache2
tcp        0      0 10.0.1.10:443           0.0.0.0:*               LISTEN      51870/nginx: master
tcp        0      0 10.0.1.10:38962         147.152.230.136:443     ESTABLISHED 1296/gomon
tcp        0      0 10.0.1.10:443           80.223.61.141:34218     ESTABLISHED 101330/nginx: worke
tcp        0      0 10.0.1.10:443           89.13.88.231:1825       ESTABLISHED 101333/nginx: worke
tcp        0      0 10.0.1.10:443           89.13.88.231:9161       ESTABLISHED 101330/nginx: worke
  1. This is the output of the “openssl s_client -connect webmail.mymaildomain.com:443 -servername webmail.mymaildomain.com” command (obviously I have sanitized by changing a lot of characheters and strings with dummy ones):
# openssl s_client -connect webmail.mymaildomain.com:443 -servername webmail.mymaildomain.com
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R11
verify return:1
depth=0 CN = mail.mymaildomain.com
verify return:1

---
Certificate chain
 0 s:CN = mail.mymaildomain.com
   i:C = US, O = Let's Encrypt, CN = R11
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 14 21:05:53 2024 GMT; NotAfter: Jan 12 21:05:52 2025 GMT
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGDzCCBPegAwIBAgIs98df7897SD9F8076SKDJFS9879KJK3qGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjQxMDE0MjEwNTUzWhcNMjUwMTEyMjEwNTUyWjAeMRwwGgYDVQQD
ExNtYWlsLml2YW5vYnVmZmEuY29tMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIIC
MFc5YGtwEmg2p2nzwSebJqIQ6LtNmuH/J4RtPqRAcjANBgkqhkiG9w0BAQsFAAOC
AQEAkJ93U3ymXA85xfvHJKjDi5jSN1uFsWqBfWM811A7etsqwus0JXhv9yA8szx8
2m56C8BAcHndCYgLwI9DGp5ckSLtxtxnfHabC6MRcKGHwuZaQNRTq5RpoPUIIHGz
dGJRvtFYDE9AspLir/Ov5g+DJuGHVuRZcIRymW8y5xfOUeaoBzgQ3VWUdXVkI6e/
CgKCAgEAkk6RI4/SZ5SoA+/JsSRgcY2A7zju5JIaG1dXw+8iOqakJQNIZE5ct+kB
602/8L744T/1b2WhgOWzmzTXuAiHLVP+7qqfMeCeXGOuOw1/TssNkEq+XFEdtH9V
WoswjIEDAcbzBp72D+9uf4K117OR+ISfL5150OkXjlW55CTj7eTvgfHZ3UZo+Nxo
J4iwaVCXfMWt7UqkqM24ErJ6zRIpjrsn36fgufnf8rWn5X4Zn9q7lM/QaqxkpKoH
uyLjRTMmkoExL2GwEWNNDSQm7JJsexL0v09z0aEiYJ0kMZzMsQlL/RsHiwBAnwD2
YV7Ct4CYuKeatZYzIYrArjVssEpjSOFbyPNCCtDCKzJraLGNKONfLNT4a+fNFnB0
e1xAxLFHbT/+trnepr6+3vjyfNtuM+dKAXABL3nXrB/IfytDdNaFdgki+EfLpq7K
HwYDVR0jBBgwFoAUxc9GpOr0w8B6bJXELbBeki8m47kwVwYIKwYBBQUHAQEESzBJ
MCIGCCsGAQUFBzABhhZodHRwOi8vcjExLm8ubGVuY3Iub3JnMCMGCCsGAQUFBzAC
hhdodHRwOi8vcjExLmkubGVuY3Iub3JnLzA2BgNVHREELzAtghNtYWlsLml2YW5v
YnVmZmEuY29tghZ3ZWJtYWlsLml2YW5vYnVmZmEuY29tMBMGA1UdIAQMMAowCAYG
Z4EMAQIBMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAouMK5EXvva2bfjjtR2d3
/ncJ2estGW0i3IkBHucAMMklUlQ7kamC0jioHSfLlXr4tnE72edfMcRwG2ir6h1V
t+cEdNKqd8ZjzorRgO7vdMZoOGVuXTI+BqYvsKjCtITfreymavYUsN8UbfZ3k6/2
1gm4x14s2id5H2SaM7kPv+bRyjLCOHNK7Xsd2G32I5TU6CCfD/S5QZwIBti+pYLY
oue/4ehdyvWsuwu2J2FzcnRTZLEcRZzL3VKgrQYI6evR4BCKvrkCAwEAAaOCAjAw
ggIsMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
AwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUq+OUZSEA11hEPxHaQS/xjUHPO9Mw
U9eCW4SU1yteGyzEuVCkR+cAAAGSjQ+bUQAABAMARzBFAiEAlutKlCeuOGJIeE4D
0EDXBUOLl5DO/3Zqwp5EuXq2aFMCIFLL5dlTCtY5rl37CIg49Yd8cB+6K+D8YSyZ
x2z5eZb6AHcAzxFW7tUufK/zh1vZaS6b6RpxZ0qwF+ysAdJbd87MOwgAAAGSjQ+b
mgAABAMASDBGAiEA+r9HeIFVk97jmw4lAKWJGi1GhE3royUL1yXbDPVCNgoCIQCj
9XIQotqrKtE9M+lfH2AAPilEcrf05nNuZWY19iV0t8DIf4HbM32W90Scug+/ggAF
VNgazgHwpDNHk+CHB6A7odXbzVhFQ6hVkuL2SWMLG3rDhQEasMSdpkNTXmWK1XXk
FqeKUwSCBCrzvl7hyTxYAKKrsg==
-----END CERTIFICATE-----
subject=CN = mail.mymaildomain.com
issuer=C = US, O = Let's Encrypt, CN = R11
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3650 bytes and written 388 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 0DBFA547E14190151080BC86DBE752BBB1CF3CDBE4D78141D97384EB978CB98E
    Session-ID-ctx:
    Resumption PSK: 48E170CBC4AE6E65B268B79252C1328B02ECABE3778376F4DD265B03EC50C2A8
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - d5 d0 4a 9f f4 1b 48 d4-c6 7c a7 c8 98 4f c3 e5   ..J...H..|...O..
    0010 - a0 e2 9a 40 f6 0c c4 1f-53 de 09 78 87 d5 b0 b0   [email protected]....
    0020 - ef f5 e0 08 80 5f aa 42-6a 1e 55 a8 53 8e 55 d0   ....._.Bj.U.S.U.
    0030 - 17 57 10 e5 06 3b d3 89-8e 50 d0 49 94 a6 e1 9e   .W...;...P.I....
    0040 - 2f 29 ac 02 1a c0 40 d6-b9 58 10 84 c8 9f bb 5c   /)[email protected].....\
    0050 - 90 b9 74 7d 71 07 8d d0-5f 9d 72 89 80 3f be 45   ..t}q..._.r..?.E
    0060 - 27 30 25 d7 7a ad 95 9e-8d 25 8b e9 26 a5 e1 b9   '0%.z....%..&...
    0070 - 60 a9 ed 41 3e 93 83 cd-70 74 4e 53 2c 2e 03 77   ..A>...ptNS,..w
    0080 - 2f 68 cf 3d df b3 85 9b-75 3e 0c 48 08 a1 d9 af   /h.=....u>.H....
    0090 - b4 ec 5f 87 92 c1 b5 f3-49 86 22 d7 a8 44 02 2c   .._.....I."..D.,
    00a0 - b4 5f f6 80 09 91 10 04-3b 1d 42 cd 73 5d 8a a4   ._......;.B.s]..
    00b0 - c0 4f e9 4d 6e 04 6c 88-c3 68 49 e8 82 4b 5a fa   .O.Mn.l..hI..KZ.
    00c0 - 2c 3e 55 83 34 08 3c f4-df 5c a5 b1 75 22 4f 90   ,>U.4.<..\..u"O.
    00d0 - 74 91 f4 ca f2 04 50 b3-f7 3b 57 c7 b5 2f 39 1b   t.....P..;W../9.
    00e0 - 99 9f 55 cd 12 ca cd e4-70 22 ea 6d d0 a7 dd 04   ..U.....p".m....

    Start Time: 1728976446
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 16384
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: DF37D1CE1CA98A645DDF0FADD445106CC83C33883D7891CBF513DFC5026DE69D
    Session-ID-ctx:
    Resumption PSK: 44E24566E036A99C1AACDCC94CF34BFC496881D6AD9154908B1E6BBB8ADDF856
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - d5 d0 4a 9f f4 1b 48 d4-c6 7c a7 c8 98 4f c3 e5   ..J...H..|...O..
    0010 - b7 76 a1 dc 7f ce 70 f0-5e 90 0c 87 b2 73 d0 21   .v....p.^....s.!
    0020 - b3 0c e2 44 db 41 5c 5f-5b 84 5f 14 d1 19 07 80   ...D.A\_[._.....
    0030 - ab 40 26 4c 09 d7 13 eb-55 74 34 95 ee 32 54 91   .@&L....Ut4..2T.
    0040 - 03 66 69 60 30 81 d8 61-a1 4f 9e 46 6b aa 39 db   .fi0..a.O.Fk.9.
    0050 - d9 91 16 84 13 78 d1 2c-e5 b7 87 51 d4 a3 83 73   .....x.,...Q...s
    0060 - 80 7c 63 34 46 d4 34 68-bc 67 e6 4f c9 4f df 06   .|c4F.4h.g.O.O..
    0070 - eb ca 6d bc fe dc c4 3e-75 ae 65 7a 2c 52 20 b7   ..m....>u.ez,R .
    0080 - cc 94 e1 2a 23 0f bc ce-72 6d c7 5a 3a 51 b1 d6   ...*#...rm.Z:Q..
    0090 - 93 83 22 2f 2f cb 59 94-53 8f e7 cb e6 87 51 27   .."//.Y.S.....Q'
    00a0 - 1d 81 c4 c7 a7 01 1e fe-c0 26 2f ca c3 8f f1 e3   .........&/.....
    00b0 - 0d 76 5c a6 29 f8 bd c5-45 da 73 59 d7 da 25 58   .v\.)...E.sY..%X
    00c0 - 9e 3d 79 8a 0f 9f 1d bf-ef c4 f1 2a 07 48 7e 18   .=y........*.H~.
    00d0 - d0 0a b8 7b 29 91 db 45-b4 29 dd df 2e cb 5e 3e   ...{)..T.)....^>
    00e0 - 91 f9 25 78 a1 a7 ba 70-06 dc e6 35 e6 4f 5c 28   ..%x...p...5.O\(

    Start Time: 1728976446
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 16384
---
read R BLOCK
  1. Tried to load https://webmail.mymaildomain.com and issue the “tail -f /var/log/nginx/error.log” and this is what I got (nothing relevant):
# tail -f error.log
2024/10/14 22:57:09 [error] 43234#43234: OCSP responder sent invalid "Content-Type" header: "text/html" while requesting certificate status, responder: r11.o.lencr.org, peer: 2.22.144.149:80, certificate: "/home/mymaildomain/conf/web/mymaildomain.com/ssl/mymaildomain.com.pem"
2024/10/14 23:06:01 [notice] 51870#51870: using inherited sockets from "9;10;11;"
  1. File “/var/log/nginx/access.log” is completely empty (look at this list):
root@webpanel:/var/log/nginx# ll
total 16
drwxr-xr-x  3 root  root   4096 Oct 14 20:01 ./
drwxrwxr-x 19 root  syslog 4096 Oct 14 23:22 ../
-rw-r-----  1 nginx adm       0 Oct 14 19:59 access.log
drwxr-xr-x  2 root  root   4096 Oct 14 23:15 domains/
-rw-r-----  1 nginx adm     353 Oct 14 23:06 error.log
root@webpanel:/var/log/nginx# cat access.log
  1. Issued the “tail -f /var/log/apache2/error.log” command while replicating the issue (loading https://webmail.mymaildomain.com) and nothing got populated. If you have a look at this output, that contains entries about 4 hours ago:
root@webpanel:/var/log/apache2# tail -f error.log
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using webpanel.buffacloud.com. Set the 'ServerName' directive globally to suppress this message
[Tue Oct 15 03:21:01.755628 2024] [ssl:warn] [pid 56337:tid 56337] AH01909: 10.0.1.10:443:0 server certificate does NOT include an ID which matches the server name
[Tue Oct 15 03:21:01.756525 2024] [mpm_event:notice] [pid 56337:tid 56337] AH00489: Apache/2.4.62 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/3.0.2 configured -- resuming normal operations
[Tue Oct 15 03:21:01.756538 2024] [core:notice] [pid 56337:tid 56337] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 15 03:42:24.909579 2024] [proxy_fcgi:error] [pid 100892:tid 100929] [client 10.0.1.10:43514] AH01071: Got error 'Primary script unknown', referer: http://effevimotor.com/formmail.php
[Tue Oct 15 04:46:35.708115 2024] [proxy_fcgi:error] [pid 100892:tid 100927] [client 10.0.1.10:46522] AH01071: Got error 'Primary script unknown', referer: http://calisthenics.us/config.php
[Tue Oct 15 04:46:35.709838 2024] [proxy_fcgi:error] [pid 100892:tid 100942] [client 10.0.1.10:46528] AH01071: Got error 'Primary script unknown', referer: http://calisthenics.us/root.php
[Tue Oct 15 04:46:35.755035 2024] [proxy_fcgi:error] [pid 100892:tid 100944] [client 10.0.1.10:46542] AH01071: Got error 'Primary script unknown', referer: http://calisthenics.us/roots.php
[Tue Oct 15 04:57:36.788149 2024] [proxy_fcgi:error] [pid 100891:tid 100934] [client 10.0.1.10:56346] AH01071: Got error 'Primary script unknown', referer: http://effevimotor.com/formmail.php
[Tue Oct 15 05:29:08.197817 2024] [proxy_fcgi:error] [pid 100892:tid 100920] [client 10.0.1.10:37314] AH01071: Got error 'Primary script unknown'








  1. Also the “/var/log/access.log” file is empty. Look at this output:
root@webpanel:/var/log/apache2# ll
total 56
drwxr-x--x  3 root adm     4096 Oct 14 20:01 ./
drwxrwxr-x 19 root syslog  4096 Oct 14 23:22 ../
-rw-r-----  1 root adm        0 Oct 14 20:01 access.log
drwxr-x--x  2 root root    4096 Oct 14 23:15 domains/
-rw-r-----  1 root adm    37851 Oct 15 05:29 error.log
-rw-r-----  1 root adm        0 Oct 14 20:00 other_vhosts_access.log
  1. Issue this command and received the very same output in both servers (affected one (which is in production) and the staging one):
# curl -I https://127.0.0.1:8443
curl: (7) Failed to connect to 127.0.0.1 port 8443 after 0 ms: Connection refused
  1. Issue the command to check PHP-FPM:
root@webpanel:/var/log/apache2# systemctl status php8.1-fpm
● php8.1-fpm.service - The PHP 8.1 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php8.1-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-10-14 20:36:19 BST; 11h ago
       Docs: man:php-fpm8.1(8)
   Main PID: 791 (php-fpm8.1)
     Status: "Processes active: 0, idle: 0, Requests: 0, slow: 0, Traffic: 0req/sec"
      Tasks: 1 (limit: 28689)
     Memory: 30.1M
        CPU: 1.808s
     CGroup: /system.slice/php8.1-fpm.service
             └─791 "php-fpm: master process (/etc/php/8.1/fpm/php-fpm.conf)" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Oct 14 20:36:16 webpanel.dummycloud.com systemd[1]: Starting The PHP 8.1 FastCGI Process Manager...
Oct 14 20:36:19 webpanel.dummycloud.com systemd[1]: Started The PHP 8.1 FastCGI Process Manager.
Oct 14 21:03:21 webpanel.dummycloud.com systemd[1]: Reloading The PHP 8.1 FastCGI Process Manager...
Oct 14 21:03:21 webpanel.dummycloud.com systemd[1]: Reloaded The PHP 8.1 FastCGI Process Manager.
  1. Forgot to mention that the following ports are opened:
  • TCP 443, 995, 25, 143, 465, 20, 21, 110, 993, 587, 53
  • UDP 53 and 143
  1. TCP port 8443 seems to be not explicitly opened but I checked in the staging server and it’s exactly the same:
# iptables -L -n -v | grep 8443
root@webpanel:/var/lib/roundcube#
  1. nginx seems to be running just fine (the same result in the staging server):
root@webpanel:/etc/nginx/conf.d/domains# systemctl status nginx
● nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-10-14 20:36:18 BST; 12h ago
       Docs: https://nginx.org/en/docs/
   Main PID: 51870 (nginx)
      Tasks: 6 (limit: 28689)
     Memory: 15.6M
        CPU: 42.419s
     CGroup: /system.slice/nginx.service
             ├─ 51870 "nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"
             ├─101330 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─101331 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─101333 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─101335 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             └─101336 "nginx: cache manager process" "" "" "" "" "" "" "" "" "" "" "" ""

Oct 14 23:20:38 webpanel.dummycloud.com systemd[1]: Reloading nginx - high performance web server...
Oct 14 23:20:38 webpanel.dummycloud.com systemd[1]: Reloaded nginx - high performance web server.
Oct 14 23:21:16 webpanel.dummycloud.com systemd[1]: Reloading nginx - high performance web server...
Oct 14 23:21:16 webpanel.dummycloud.com systemd[1]: Reloaded nginx - high performance web server.
Oct 14 23:21:17 webpanel.dummycloud.com systemd[1]: Reloading nginx - high performance web server...
Oct 14 23:21:17 webpanel.dummycloud.com systemd[1]: Reloaded nginx - high performance web server.
Oct 14 23:21:22 webpanel.dummycloud.com systemd[1]: Reloading nginx - high performance web server...
Oct 14 23:21:22 webpanel.dummycloud.com systemd[1]: Reloaded nginx - high performance web server.
Oct 15 03:21:02 webpanel.dummycloud.com systemd[1]: Reloading nginx - high performance web server...
Oct 15 03:21:02 webpanel.dummycloud.com systemd[1]: Reloaded nginx - high performance web server.
  1. nginx configuration seems to be correct:
root@webpanel:/etc/nginx/conf.d/domains# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
  1. About binding, everything looks fine (the same output in the staging server):
root@webpanel:/etc/nginx/conf.d/domains# netstat -tuln | grep 8443
tcp        0      0 10.0.1.10:8443          0.0.0.0:*               LISTEN
  1. DNS resolution seems to be fine:
root@webpanel:/etc/php/8.1/fpm/pool.d# dig webmail.mymaildomain.com

; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> webmail.mymaildomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3952
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;webmail.mymaildomain.com.                IN      A

;; ANSWER SECTION:
webmail.mymaildomain.com. 300     IN      A       130.162.187.98

;; Query time: 592 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Oct 15 08:44:26 BST 2024
;; MSG SIZE  rcvd: 67
  1. Received this output while checking local connectivity (the same result in the staging server):
# curl -I https://localhost:8443
curl: (7) Failed to connect to localhost port 8443 after 0 ms: Connection refused
  1. Tested direct SSL/TLS connection (the very same result appeared in the staging server):
# curl -I https://localhost:8443
curl: (7) Failed to connect to localhost port 8443 after 0 ms: Connection refused
root@webpanel:/etc/php/8.1/fpm/pool.d# openssl s_client -connect localhost:8443
20F0913973F40000:error:8000006F:system library:BIO_connect:Connection refused:../crypto/bio/bio_sock2.c:125:calling connect()
20F0913973F40000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
connect:errno=111

SECOND PART

  1. Checked log /var/log/syslog and /var/log/messages files:
# tail -n 100 /var/log/syslog
Oct 15 07:52:33 webpanel systemd[1]: Created slice User Slice of UID 1001.
Oct 15 07:52:33 webpanel systemd[1]: Starting User Runtime Directory /run/user/1001...
Oct 15 07:52:33 webpanel systemd[1]: Finished User Runtime Directory /run/user/1001.
Oct 15 07:52:33 webpanel systemd[1]: Starting User Manager for UID 1001...
Oct 15 07:52:33 webpanel systemd[140947]: Queued start job for default target Main User Target.
Oct 15 07:52:33 webpanel systemd[140947]: Created slice User Application Slice.
Oct 15 07:52:33 webpanel systemd[140947]: Reached target Paths.
Oct 15 07:52:33 webpanel systemd[140947]: Reached target Timers.
Oct 15 07:52:33 webpanel systemd[140947]: Starting D-Bus User Message Bus Socket...
Oct 15 07:52:33 webpanel systemd[140947]: Listening on GnuPG network certificate management daemon.
Oct 15 07:52:33 webpanel systemd[140947]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Oct 15 07:52:33 webpanel systemd[140947]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Oct 15 07:52:33 webpanel systemd[140947]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Oct 15 07:52:33 webpanel systemd[140947]: Listening on GnuPG cryptographic agent and passphrase cache.
Oct 15 07:52:33 webpanel systemd[140947]: Listening on debconf communication socket.
Oct 15 07:52:33 webpanel systemd[140947]: Listening on REST API socket for snapd user session agent.
Oct 15 07:52:33 webpanel systemd[140947]: Listening on D-Bus User Message Bus Socket.
Oct 15 07:52:33 webpanel systemd[140947]: Reached target Sockets.
Oct 15 07:52:33 webpanel systemd[140947]: Reached target Basic System.
Oct 15 07:52:33 webpanel systemd[140947]: Reached target Main User Target.
Oct 15 07:52:33 webpanel systemd[140947]: Startup finished in 84ms.
Oct 15 07:52:33 webpanel systemd[1]: Started User Manager for UID 1001.
Oct 15 07:52:33 webpanel systemd[1]: Started Session 730 of User ubuntu.
Oct 15 07:54:01 webpanel CRON[141107]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 07:55:01 webpanel CRON[141167]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 07:55:01 webpanel CRON[141168]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 07:55:01 webpanel CRON[141169]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 07:56:01 webpanel CRON[141570]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 07:58:01 webpanel CRON[141663]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:00:01 webpanel CRON[141757]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:00:01 webpanel CRON[141758]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:00:01 webpanel CRON[141759]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:02:01 webpanel CRON[142214]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:04:01 webpanel CRON[142304]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:05:01 webpanel CRON[142373]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 08:05:01 webpanel CRON[142374]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:05:01 webpanel CRON[142375]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:06:01 webpanel CRON[142789]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:08:01 webpanel CRON[142884]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:09:01 webpanel CRON[142940]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 15 08:09:04 webpanel systemd[1]: Starting Clean php session files...
Oct 15 08:09:05 webpanel systemd[1]: phpsessionclean.service: Deactivated successfully.
Oct 15 08:09:05 webpanel systemd[1]: Finished Clean php session files.
Oct 15 08:09:05 webpanel systemd[1]: phpsessionclean.service: Consumed 1.003s CPU time.
Oct 15 08:10:01 webpanel CRON[143264]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:10:01 webpanel CRON[143265]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:10:01 webpanel CRON[143266]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:12:01 webpanel CRON[144411]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:14:01 webpanel CRON[144501]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:15:01 webpanel CRON[144569]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:15:01 webpanel CRON[144570]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 08:15:01 webpanel CRON[144571]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:16:01 webpanel CRON[144971]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:17:01 webpanel CRON[145041]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 15 08:18:01 webpanel CRON[145080]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:20:01 webpanel CRON[145201]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:20:01 webpanel CRON[145202]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:20:01 webpanel CRON[145203]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:22:01 webpanel CRON[145672]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:22:40 webpanel spamd[1832]: spamd: connection from 127.0.0.1 [127.0.0.1]:42800 to port 783, fd 6
Oct 15 08:22:40 webpanel spamd[1832]: spamd: setuid to debian-spamd succeeded
Oct 15 08:22:40 webpanel spamd[1832]: spamd: checking message <[email protected]> for debian-spamd:123
Oct 15 08:22:41 webpanel spamd[1832]: spamd: clean message (-1.1/5.0) for debian-spamd:123 in 0.7 seconds, 45445 bytes.
Oct 15 08:22:41 webpanel spamd[1832]: spamd: result: . -1 - DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,HTML_FONT_LOW_CONTRAST,HTML_MESSAGE,MIME_HTML_ONLY,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS scantime=0.7,size=45445,user=debian-spamd,uid=123,required_score=5.0,rhost=127.0.0.1,raddr=127.0.0.1,rport=42800,mid=<[email protected]>,autolearn=ham autolearn_force=no
Oct 15 08:22:41 webpanel spamd[1599]: prefork: child states: II
Oct 15 08:24:01 webpanel CRON[145777]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:25:01 webpanel CRON[145848]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 08:25:01 webpanel CRON[145849]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:25:01 webpanel CRON[145850]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:26:01 webpanel CRON[146253]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:28:01 webpanel CRON[146345]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:30:01 webpanel CRON[146440]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:30:01 webpanel CRON[146439]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:30:01 webpanel CRON[146441]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:32:01 webpanel CRON[148209]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:34:01 webpanel CRON[148308]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:35:01 webpanel CRON[148364]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 08:35:01 webpanel CRON[148365]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:35:01 webpanel CRON[148367]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:36:01 webpanel CRON[148769]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:36:17 webpanel freshclam[767]: Tue Oct 15 08:36:17 2024 -> Received signal: wake up
Oct 15 08:36:17 webpanel freshclam[767]: Tue Oct 15 08:36:17 2024 -> ClamAV update process started at Tue Oct 15 08:36:17 2024
Oct 15 08:36:17 webpanel freshclam[767]: Tue Oct 15 08:36:17 2024 -> daily.cld database is up-to-date (version: 27427, sigs: 2067246, f-level: 90, builder: raynman)
Oct 15 08:36:17 webpanel freshclam[767]: Tue Oct 15 08:36:17 2024 -> main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Oct 15 08:36:17 webpanel freshclam[767]: Tue Oct 15 08:36:17 2024 -> bytecode.cvd database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
Oct 15 08:38:01 webpanel CRON[148863]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:39:01 webpanel CRON[148923]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 15 08:39:04 webpanel systemd[1]: Starting Clean php session files...
Oct 15 08:39:05 webpanel systemd[1]: phpsessionclean.service: Deactivated successfully.
Oct 15 08:39:05 webpanel systemd[1]: Finished Clean php session files.
Oct 15 08:39:05 webpanel systemd[1]: phpsessionclean.service: Consumed 1.039s CPU time.
Oct 15 08:40:01 webpanel CRON[149259]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:40:01 webpanel CRON[149258]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:40:01 webpanel CRON[149260]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:42:01 webpanel CRON[149731]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:44:01 webpanel CRON[149836]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
Oct 15 08:45:01 webpanel CRON[149896]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 15 08:45:01 webpanel CRON[149897]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-rrd)
Oct 15 08:45:01 webpanel CRON[149898]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue backup)
Oct 15 08:46:01 webpanel CRON[150298]: (admin) CMD (sudo /usr/local/hestia/bin/v-update-sys-queue restart)
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d#
root@webpanel:/etc/php/8.1/fpm/pool.d# tail -n 100 /var/log/messages
tail: cannot open '/var/log/messages' for reading: No such file or directory
root@webpanel:/etc/php/8.1/fpm/pool.d#
  1. Telnet connection on port 8443 refused in both servers:
# telnet localhost 8443
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

Any help / assistance / hint is much appreciated!

Thanks.

Regards.

Keep in mind that Apache2 is not listening on all interfaces, only on 10.0.1.10 so try this command:

telnet 10.0.1.10 8443

Show also the Apache status:

systemctl status apache2 --no-pager -l

Hi @sahsanu ,

That was not the problem.

I eventually found it and it was between my chair and my desk.

I forgot that for testing purposes I entered an A record in my hosts file for this FQDN. :nauseated_face: :hot_face:

That happens when you work countless hours and you have sleep deprivation. After removing this entry, the issue was resolved.

Thanks for trying though.

Regards.

1 Like

I’m glad you solved it.

Always is DNS :smiley: