Getting err_too_many_redirects after upgrading to 1.8.x

1

Hello!

Last night, around 3 am JST, a server that is using Hestia was automatically upgraded to version 1.8.0 and started presenting err_too_many_redirects errors for websites pointing to it. Before the upgrade, it was working normally.

The server runs on Ubuntu 20.04 LTS.

It hosts a SaaS application that hosts several websites that points to it via CNAME. E.g:

CNAME example.com > saas.example.com

Both websites (example.com and saas.example.com) are using CloudFlare and have their SSL/TLS set to the Full mode, and Always use HTTPS set to ON.

Now, all the websites pointing to the SaaS platform via CNAME are presenting the err_too_many_redirects error.

Usually, Hestia Nginx config listen 8083 ssl is replaced with a custom port number in /usr/local/hestia/nginx/nginx.conf to enable accessing Hestia Panel by a domain name instead of the IP, and always there is an upgrade, Hestia seems to overwrite this file.

However, at this time, it seems that Hestia did not fully updated nginx configuration, and a notification started showing up in Hestia Panel … :point_down:

IMPORTANT: Manual Action Required


To enable the 'Enhanced and Optimized TLS' feature, we must update the NGINX configuration file at /etc/nginx/nginx.conf.

But for unknown reason or you edited it, may not be fully apply all the changes in this upgrade.

Please follow the default configuration file to sync it:
/usr/local/hestia/install/deb/nginx/nginx.conf

Backed up configuration file:
/root/hst_backups/120720230322/conf/nginx/nginx.conf

Visit PR #3555 on GitHub to learn more.

I looked into these mentioned files, and it looks like they have changed quite a lot. Some changes are hard to spot, becuase although the instructions are the same, it seems that they have been reordered inside the file along with additional changes …

Given the notification, I’m afraid the issue is related to “Enhanced and Optimized TLS” being disabled.

I’m not experienced with this, so I have no idea where to start!

Would you help me to fix this? :pray:

2

Cloudflare should be configured for Full (Strict). Always Use HTTPS can cause conflicts with Let’s Encrypt HTTP-01 challenges. It’s safer to carve out an exception for the .well-known/acme-challenge path ahead of your own HTTPS redirect rule.

Someone else will need to advise you on the config changes.I haven’t run across that issue yet.

3

Thank you, @linkp !

I haven’t tried going with full strict yet, but will give it a try soon and post back the results here.

Interesting, however, that it was working just fine before the upgrade. :thinking: So, my guess is that there is also something to be worked out on HestiaCP itself.

Edit:

Perhaps, the issue is related to changes made to Nginx, as reported in Hestia 1.8.1 change log:

Custom nginx templates require some changes due to deprecated http2 parameter for the listen directive by Nginx 1.25.1 (#3684, #3704) and 0-RRT Protection introduced in (#3692)

But, I have no idea what changes need to made.

4

Remove: https://github.com/hestiacp/hestiacp/blob/85464b4d345b9ca699fa18e959d8a78dfaf5dac7/install/deb/templates/web/nginx/php-fpm/default.stpl#L8

From this line

and

add:

At the same location as here in the linked file

1 Like
5

Thanks, @eris!

I checked the file you suggested, which I found at /usr/local/hestia/data/templates/web/nginx/php-fpm.stpl, and it looks like it already has the lines you suggested to be added. :thinking:

Should I try rebuild the domains?

6

Yes … You can try it…

7

I did, no luck! :disappointed_relieved:

In fact, I just realized that the domain in question is using a custom template for NodeJS. So, I updated the custom template as you recommended, and rebuilt users and domains. Didn’t work!

The custom template it’s using looks like this:

server {
    listen      %ip%:%proxy_ssl_port% ssl http2; 
    server_name %domain_idn% %alias_idn%;

    ssl_certificate      %ssl_pem%;
    ssl_certificate_key  %ssl_key%;
    ssl_stapling         on;
    ssl_stapling_verify  on;

    error_log  /var/log/%web_system%/domains/%domain%.error.log error;

    # TLS 1.3 0-RTT anti-replay
    if ($anti_replay = 307) { return 307 https://$host$request_uri; }
    if ($anti_replay = 425) { return 425; }

    gzip             on;
    gzip_min_length  1100;
    gzip_buffers     4 32k;
    gzip_types       image/svg+xml svg svgz text/plain application/x-javascript text/xml text/css;
    gzip_vary        on;

   # ... 
}

Let me add a bit more context …

Assuming there is a CNAME example.com > foo.bar.com :

  • Visiting foo.bar.com the SaaS application is running as expected
  • Visiting example.com, it gets ERR_TOO_MANY_REDIRECTS

The aforementioned SaaS application identifies the domain calling foo.bar.com (e.g example.com) and loads the necessary data from clients. There is no redirects in place.

The full-strict mode from CloudFlare cannot be actually used because when visiting example.com triggers an invalid certificate error, since the certificate issued to foo.bar.com does not cover example.com. At least, not for now. :thinking:

It was working like a charm before upgrading to Hestia 1.8.1 …

8

You will need to fix the SANs in your certificate. Are you using Cloudflare for SaaS? If not, it might be the right solution.

There must be redirection occurring or you wouldn’t receive that error message. You can test for redirection with curl. I find curl -Iiv http://example.com useful in that context.

Pausing Cloudflare temporarily during troubleshooting can sometimes be helpful.

I suggest breaking your overall issue into smaller components and work on fixing them from the server out to the edge.

9

Again, thank you for the help, @linkp.

You will need to fix the SANs in your certificate.

I guess you might be talking about Subject Alternate Name (I had to Google it to find it out! :face_with_peeking_eye:). I have no idea how that can be fixed. There are certificates issued by CloudFlare and also those by Hestia (using Let’s Encrypt) when a domain is added/modified.

Are you using Cloudflare for SaaS?

I was not. I enabled it for the SaaS domain, and added an existing demo website, which was also affected by this issue. The problem seems to persist.

There must be redirection occurring or you wouldn’t receive that error message.

You’re right! :man_bowing:

It looks like there is a 301 redirection happening somewhere! When a HTTPS request is received, it redirects to HTTP with 301. :thinking:

I tried the curl you suggested, and here is the result :point_down:

$ curl -Iiv https://demo1.kiwicart.io
* Trying 2606:...:443...
* Connected to [DOMAIN-NAME] (2606:...) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=[DOMAIN-NAME]
*  start date: Jul 12 13:47:51 2023 GMT
*  expire date: Oct 10 13:47:50 2023 GMT
*  subjectAltName: host "[DOMAIN-NAME]" matched cert's "[DOMAIN-NAME]"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: [DOMAIN-NAME]]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x138811400)
> HEAD / HTTP/2
> Host: [DOMAIN-NAME]
> user-agent: curl/7.84.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 301 
HTTP/2 301 
< date: Wed, 12 Jul 2023 15:19:38 GMT
date: Wed, 12 Jul 2023 15:19:38 GMT
< content-type: text/html
content-type: text/html
< location: [DOMAIN-NAME]
location: [DOMAIN-NAME]
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=lj7...%3D%3D"}],"group":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=lj7q...%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< strict-transport-security: max-age=0; includeSubDomains; preload
strict-transport-security: max-age=0; includeSubDomains; preload
< x-content-type-options: nosniff
x-content-type-options: nosniff
< server: cloudflare
server: cloudflare
< cf-ray: 7e5a...-NRT
cf-ray: 7e5a...-NRT

< 
* Connection #0 to host [DOMAIN-NAME] left intact

* Real domain name replaced with [DOMAIN-NAME] …

Pausing Cloudflare temporarily during troubleshooting can sometimes be helpful.

Tried this as well, but is doesn’t seems to have any effect!

10

Thanks for sharing the curl output. Redacting the domain is a minor hindrance. Removing the scheme and path makes the output nearly useless. I can’t see the broken redirect because you censored too much.

When redacting a domain, it’s best to substitute one of the reserved names like example.com and leave the rest of the URL intact. If you need to hide more than one domain, you can use example.net and example.org as required. Just remember to use them consistently so that each respective domain is always replaced with the same placeholder.

1 Like
11

Oh, my bad!
Here you go again, this time in full.

curl -Iiv https://demo1.kiwicart.site
* Trying 2606:4700:3035::ac43:b430:443...
* Connected to demo1.kiwicart.site (2606:4700:3035::ac43:b430) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=demo1.kiwicart.site
*  start date: Jul 12 14:52:30 2023 GMT
*  expire date: Oct 10 14:52:29 2023 GMT
*  subjectAltName: host "demo1.kiwicart.site" matched cert's "demo1.kiwicart.site"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: demo1.kiwicart.site]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15800bc00)
> HEAD / HTTP/2
> Host: demo1.kiwicart.site
> user-agent: curl/7.84.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 301 
HTTP/2 301 
< date: Wed, 12 Jul 2023 15:57:27 GMT
date: Wed, 12 Jul 2023 15:57:27 GMT
< content-type: text/html
content-type: text/html
< location: http://demo1.kiwicart.site/
location: http://demo1.kiwicart.site/
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=T3wN7Mx5FvbB331dyhos%2F6L%2FQyegzujufsnAuk1iOlfFTfMOmDKYSQbNpAIIT2lN%2BcxgcYaJXYlIZNCt9W540sc43eWCl4CgB%2B9i3WTpONqJEKi8wtekrucyJmTAUT5h%2FtbxUJJ%2BnDcZzv2CS3y1ZvtW"}],"group":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=T3wN7Mx5FvbB331dyhos%2F6L%2FQyegzujufsnAuk1iOlfFTfMOmDKYSQbNpAIIT2lN%2BcxgcYaJXYlIZNCt9W540sc43eWCl4CgB%2B9i3WTpONqJEKi8wtekrucyJmTAUT5h%2FtbxUJJ%2BnDcZzv2CS3y1ZvtW"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< strict-transport-security: max-age=0; includeSubDomains; preload
strict-transport-security: max-age=0; includeSubDomains; preload
< x-content-type-options: nosniff
x-content-type-options: nosniff
< server: cloudflare
server: cloudflare
< cf-ray: 7e5a7b66784af67d-NRT
cf-ray: 7e5a7b66784af67d-NRT

< 
* Connection #0 to host demo1.kiwicart.site left intact

Edit:

The domain above was included as a custom hostname for CloudFlare for SaaS. Calling it on browser ends up in a too many redirects error.

There is another domain, https://demo2.kiwicart.site which was not included as a custom hostname that when called in the browser, seems to reach the server with HestiaCP, however, instead of reach the right virtual host, it seems to be reaching the “global host” in the server, and displaying this :point_down:

Screenshot 2023-07-13 at 1.07.49
Screenshot 2023-07-13 at 1.07.491088Ă—846 13.8 KB

The kiwicart.site domain is behind CloudFlare too, and its SSL/TLS mode is set to Flexible. If set to Full or Full strict it triggers the too many redirection error.

12

@linkp,

I might be wrong, but I checked it with https://httpstatus.io/, and found something that makes me believe that the issue is, indeed, in Hestia configuration.

Screenshot 2023-07-13 at 1.26.26
Screenshot 2023-07-13 at 1.26.262790Ă—728 116 KB

Scrolling down a bit in the result page, there is a button to display the response body, which shows the following content :point_down:

Screenshot 2023-07-13 at 1.27.06
Screenshot 2023-07-13 at 1.27.061112Ă—812 43.8 KB

So, it looks like the Nginx is redirecting the requests … from https to http. :thinking:

Is my assumption correct?

13

Can you check your web application to see if it has the HTTP scheme entered somewhere?

Aa an example, you can enter a site URL in a WordPress site with HTTP instead of HTTPS and wind up with unwanted results.

14

Do you mean in the source code?

If so, the SaaS application has explicit https scheme being used as part of the base URLs. There is no redirections to a different scheme.

15

I am not someplace where I can review the Hestia site configuration interface and options presently. I would explore that next to see if that is where your redirect is being created.

16

Do you have any recommendation on what files could I look into to check?

17

Is it possible to downgrade Hestia to its previous version, perhaps, recovering the settings files from the backup it does before upgrading?

18

Hey, @linkp! :wave:

I guess I found something …

At /etc/nginx/conf.d there are some nginx config files and one of them is a file named with the server IP address, e.g 127.0.0.1.conf. Naturally, the IP was censored …

In that file, I found the following instruction :point_down:

server {
    listen       [SERVER-IP]:80 default;
    server_name  _;
    #access_log  /var/log/nginx/[SERVER-IP].log main;
    location / {
        proxy_pass  http://[SERVER-IP]:8080;
   }
}

server {
    listen      [SERVER-IP]:443 ssl;
    server_name _;
    ssl_certificate      /usr/local/hestia/ssl/certificate.crt;
    ssl_certificate_key  /usr/local/hestia/ssl/certificate.key;

    return 301 http://$host$request_uri;

    location / {
        root /var/www/document_errors/;
    }

    location /error/ {
        alias /var/www/document_errors/;
    }
}

As seen above, there is a return 301 http://$host$request_uri; instruction that, certainly is handling the request and redirecting visitors from https to http!

Naturally, this might be coming from a template somewhere … shouldn’t it be set to https instead?

1 Like
19

It might. Your find looks promising. Hopefully someone else can offer some additional guidance.

In the meantime, you might consider cross-referencing your templates for similarities and see if that leads you toward a solution.

20

Would you know where can I find the template that produces this file?