Intermittent TLS handshake failure — nginx signs with wrong key material after reload (random vhost, ~1–2 per reload)

Community Support Web
1

Running HestiaCP on Ubuntu 24.04, nginx + system OpenSSL 3.0.13, ~25 vhosts, all LE certs, 4096-bit RSA.

Every so often a random vhost stops serving TLS. Clients get:

error:02000086:rsa routines:RSA_verify_PKCS1_PSS_mgf1:last octet invalid
error:0A00007B:SSL routines:tls_process_cert_verify:bad signature

Browsers show ERR_SSL_PROTOCOL_ERROR. Any systemctl reload nginx clears it — but the reload re-rolls, and typically 1–2 different vhosts out of 25 come up broken instead.

What I’ve verified on affected vhosts, while broken:

  • On-disk pair is sound: pem/crt/key moduli match, openssl rsa -check OK, and a manual PSS + PKCS#1 v1.5 sign/verify against the served cert’s public key succeeds.
  • nginx serves the correct certificate — served leaf fingerprint identical to the disk leaf for that SNI.
  • No cross-vhost mispairing: I hashed the served cert’s modulus against every .key on the box; it matches only that vhost’s own key. Confirmed on two separate incidents.
  • Fails under both TLS 1.3 (RSA-PSS) and TLS 1.2 forced rsa_pkcs1_sha256 — two different signing paths, same result.
  • Chain validates fine (Verify return code: 0).
  • Deterministic once established, so it’s set at config load in the master, not a flaky worker. worker_processes auto (8). Error randomly disappears after about 1h, or after nginx reload
  • Files are not touched across the break/heal transition — inode, mtime and ctime all unchanged.
  • Single ssl_certificate pair per vhost, no duplicate server blocks, no ECDSA/RSA dual cert.

So: right cert, right key, they correspond, and nginx still produces a signature that doesn’t verify.

Ruled out: LE renewal timing, cert/key rewrite races during rebuild, Hestia auto-update, stale/deleted libssl mappings (lsof shows current mem-mapped libs), key size, duplicate keys, SNI misresolution.

Present since at least early July on a system built 18 May, across several libssl and Hestia updates — not tied to any particular upgrade. Was only visible on non-proxied sites until I pointed monitoring at the origin IP; it affects Cloudflare-proxied vhosts too, they just fail as origin 5xx.

Since HestiaCP ships the nginx build: does it link system libssl, or something bundled? Happy to run whatever diagnostics would help.

2

The word “random” is quite concerning here. That’s a Virtual Host or a Dedicated Server? Because those random issues could be caused by faulty RAM…

Hestia builds the Nginx serving the control panel but your sites don’t use it, they use the package provided directly by Nginx:

❯ cat /etc/apt/sources.list.d/nginx.list
deb [arch=amd64 signed-by=/usr/share/keyrings/nginx-keyring.gpg] https://nginx.org/packages/mainline/debian/ trixie nginx

You can also check the Nginx version, build options, etc.

nginx -V

And you will see that it doesn’t use a specific ssl version, it uses the one installed in your OS. Example:

❯ ldd /usr/sbin/nginx | grep ssl
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f0ac5cf2000)
3

Thanks. Yes I thought about faulty ram, but this is a Hetzner VM - so I think they would notice. The server is stable apart from the SSL issue.

nginx version: nginx/1.31.3

Also running the latest OpenSSL shipped with the Ubuntu LTS

OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

I’m really puzzled. This has been happening for 2 months now. Before that I ran Cloudpanel, never had issues with that stack although it is quite similar. I do run quite an amount of sites (25) but I’m sure many users run many more of them…

4

Could you please show the output of this command? It checks the number of files used by each Nginx/Apache process and the file descriptor limit for each one.

curl -fsSL https://7j.gg/chknof | sudo bash -s --
5

Currently no ssl errors (since yesterday morning) and everything looks ok. I’ll check again when an SSL error occurs:

Checking services nginx apache2
The open files limit threshold has been set at 80%

Process 841149 :: nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
Current open files:   141
Limit for open files: 1024

Process 1409779 :: nginx: worker process
Current open files:   143
Limit for open files: 65535

Process 1409780 :: nginx: worker process
Current open files:   143
Limit for open files: 65535

Process 1409781 :: nginx: worker process
Current open files:   147
Limit for open files: 65535

Process 1409782 :: nginx: worker process
Current open files:   143
Limit for open files: 65535

Process 1409783 :: nginx: cache manager process
Current open files:   138
Limit for open files: 65535
6

I’m glad you don’t get more errors but the limit of 1024 for the main nginx process is too low and you should increase it.

mkdir -p /etc/systemd/system/nginx.service.d/
echo -e '[Service]\nLimitNOFILE=65535' > etc/systemd/system/nginx.service.d/override.conf
systemctl daemon-reload
systemctl restart nginx

Once done, check that the process has the new limit.

curl -fsSL https://7j.gg/chknof | sudo bash -s --
7

Yes the errors are quite intermittent. But your theory does make sense. It often happens when HestiaCP does resource-heavy things like updating, backing up etc.

Sometimes SSL handshake errors don’t appear for days - sometimes I get multiple a day.

I changed the limits, let’s see how that goes in the following days.

Greatly appreciate your help!

8

After a few days, the issue came back last night on one site. I had disabled my little ‘ghettofix’ script that checks the handshake and reloads nginx a few days ago.

Last time it occured was 6/8 at 2:14am, this time is happened at 11/8 at 2:14am, precisely the same time. I’ll check the logs but I’m pretty sure it was a HestiaCP update script or backup that triggered a Nginx restart or reload…