Web template confusion (can't turn off buffering)

1

I was poking around hestiacp and selected userName, then clicked domainName (to Edit Web Doman), then scrolled down and clicked Advanced Options to get to the web templates.

Under Web Template APACHE2

it says “default”, which blows my mind since I long ago had made and selected a slightly-modified template called “pw_prot” (which is still selectable) that enabled .htpasswd protection for a directory. For some reason, the .htpasswd still works, so I left that alone, more puzzled than ever.

Under Backend Template PHP-FPM

it says “default”

Under Proxy Template

it says “default”

Under Proxy Template there is a choice for “no_cache_CGIs” but that seems to be one of my rather old abortive attempts at disabling caching for some CGI perl scripts. (It fails pretty spectacularly…)

I searched here, and (again) found How to disable Fast CGI Cache for specific page? but I don’t see how to apply that solution. For one thing, it refers to /home/admin/conf/web/domain.com/nginx.fastcgi_cache.conf that I can’t find. It says to add “ set $no_cache 1;” as per wordpress.tpl but the only wordpress.tpl I can find is among 88 templates under /usr/local/hestia/data/templates/web/nginx/php-fpm/ – but none of them show up in any of the 3 drop-boxes, so even if I knew how to modify/rename one, I do not understand how that would help. :face_with_diagonal_mouth:

I’m sorry to be a pest like this, but with Apache and nginx and php-fpm I’m lost, and could really use a hand in solving this one long-remaining annoying issue. Between those 3, and fastcgi and gzip (and whatever unknown unknowns there may be) I’ve never found the right combination, though I’ve found many wrong ones with some interesting errors/failures…

Ideally I’d like to disable caching (so output would show up in a user’s browser as it becomes available) for just a handful of name1.cgi and diffname2.cgi scripts.

TIA for any/all assistance.

2

As you are using Nginx+Apache2:

Web Template APACHE2
Uses the templates located in /usr/local/hestia/data/templates/web/apache2/php-fpm/

Backend Template PHP-FPM
Uses the templates located in /usr/local/hestia/data/templates/web/php-fpm/

Proxy Template
Uses the templates located in /usr/local/hestia/data/templates/web/nginx/

So take a look to this post, it could be useful for your use case:

Note: In above post, newtemplate.{tpl,stpl} must be located in /usr/local/hestia/data/templates/web/nginx/

That’s because you are not using Nginx standalone so you don’t use fast_cgi cache provided by Nginx.

3

Thank you very much for your reply, but it’s not working for me.

I copied default.[s]tpl and then edited fresh /usr/local/hestia/data/templates/web/nginx/no_cache_CGIs.tpl and /usr/local/hestia/data/templates/web/nginx/no_cache_CGIs.stpl to include the location ~* .(htm|html|css|js)$ { section, which I simply changed to location ~* .cgi$ { (under the “404” section) before later fine-tuning the matching for individual scripts:

    location ~ /\.(?!well-known\/|file) {
            deny all;
            return 404;
    }

    location ~* \.cgi$ {
            root       %docroot%;
            access_log /var/log/%web_system%/domains/%domain%.log combined;
            access_log /var/log/%web_system%/domains/%domain%.bytes bytes;
            open_file_cache off;
            proxy_cache off;
            proxy_no_cache 1;
            proxy_cache_bypass 1;
            add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0";
            expires off;
            add_header Pragma "no-cache";
            etag off;
            fastcgi_hide_header "Set-Cookie";
    }

But my web browser returns a 404 error. Curiously, those failed requests for the script do NOT show up in the domain’s apache2 hits log nor the apache2 errs log, which are all I normally watch.

So I looked under /var/log/nginx/domains/ but only saw old (pre-apache?) logs for 2 domains other than the one I was using to test this, plus all the [sub]domains but for webmail, so I’m baffled how/why/where the https request got lost.

4

Paste the complete templates.

5

no_cache_CGIs.tpl:

#=========================================================================#

Default Web Domain Template

DO NOT MODIFY THIS FILE! CHANGES WILL BE LOST WHEN REBUILDING DOMAINS


#=========================================================================#

server {
listen      %ip%:%proxy_port%;
server_name %domain_idn% %alias_idn%;
error_log   /var/log/%web_system%/domains/%domain%.error.log error;

include %home%/%user%/conf/web/%domain%/nginx.forcessl.conf*;

location ~ /\.(?!well-known\/|file) {
	deny all;
	return 404;
}

location ~* \.cgi$ {
	root       %docroot%;
	access_log /var/log/%web_system%/domains/%domain%.log combined;
	access_log /var/log/%web_system%/domains/%domain%.bytes bytes;
	open_file_cache off;
	proxy_cache off;
	proxy_no_cache 1;
	proxy_cache_bypass 1;
	add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0";
	expires off;
	add_header Pragma "no-cache";
	etag off;
	fastcgi_hide_header "Set-Cookie";
}


location / {
	proxy_pass http://%ip%:%web_port%;

	location ~* ^.+\.(%proxy_extensions%)$ {
		try_files  $uri @fallback;

		root       %docroot%;
		access_log /var/log/%web_system%/domains/%domain%.log combined;
		access_log /var/log/%web_system%/domains/%domain%.bytes bytes;

		expires    max;
	}
}

location @fallback {
	proxy_pass http://%ip%:%web_port%;
}

location /error/ {
	alias %home%/%user%/web/%domain%/document_errors/;
}

include %home%/%user%/conf/web/%domain%/nginx.conf_*;

}

no_cache_CGIs.stpl:



#=========================================================================#

Default Web Domain Template

DO NOT MODIFY THIS FILE! CHANGES WILL BE LOST WHEN REBUILDING DOMAINS






#=========================================================================#

server {
listen      %ip%:%proxy_ssl_port% ssl;
server_name %domain_idn% %alias_idn%;
error_log   /var/log/%web_system%/domains/%domain%.error.log error;

ssl_certificate     %ssl_pem%;
ssl_certificate_key %ssl_key%;
ssl_stapling        on;
ssl_stapling_verify on;

# TLS 1.3 0-RTT anti-replay
if ($anti_replay = 307) { return 307 https://$host$request_uri; }
if ($anti_replay = 425) { return 425; }

include %home%/%user%/conf/web/%domain%/nginx.hsts.conf*;

location ~ /\.(?!well-known\/|file) {
	deny all;
	return 404;
}

location ~* \.cgi$ {
	root       %docroot%;
	access_log /var/log/%web_system%/domains/%domain%.log combined;
	access_log /var/log/%web_system%/domains/%domain%.bytes bytes;
	open_file_cache off;
	proxy_cache off;
	proxy_no_cache 1;
	proxy_cache_bypass 1;
	add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0";
	expires off;
	add_header Pragma "no-cache";
	etag off;
	fastcgi_hide_header "Set-Cookie";
}

location / {
	proxy_ssl_server_name on;
	proxy_ssl_name $host;
	proxy_pass https://%ip%:%web_ssl_port%;

	location ~* ^.+\.(%proxy_extensions%)$ {
		try_files  $uri @fallback;

		root       %sdocroot%;
		access_log /var/log/%web_system%/domains/%domain%.log combined;
		access_log /var/log/%web_system%/domains/%domain%.bytes bytes;

		expires    max;
	}
}

location @fallback {
	proxy_ssl_server_name on;
	proxy_ssl_name $host;
	proxy_pass https://%ip%:%web_ssl_port%;
}

location /error/ {
	alias %home%/%user%/web/%domain%/document_errors/;
}

proxy_hide_header Upgrade;

include %home%/%user%/conf/web/%domain%/nginx.ssl.conf_*;

}
6

Who is executing CGIs? Apache?

7

:thinking:

Sorry but I do not have a clue; I’d already confessed that I’m lost. Seems possible…

I cut my baby teeth on apache decades ago, got used to .htpasswd and .htaccess, never really learned much about nginx (though I think that old pw_prot template was a modified template needed for nginx to pw protect directories, left over from an old pre-apache install)… When I started VPSing I tried going withOUT apache to save RAM, but missed it, so added it for this install. But I’ve never wrapped my mind around the added complexity; I do not have a decent picture of how apache nginx and “backend” PHP-FPM get along together.

It all just works fine, automagically, except for controlling this CGI buffering thing.

8

Try these templates. Remember to rebuild the web domain after modifying the templates.

tpl template:

#=========================================================================#
#                       Default Web Domain Template                       #
#                                                                         #
#                 DO NOT MODIFY THIS FILE! CHANGES WILL BE               #
#                    LOST WHEN REBUILDING DOMAINS                         #
#                                                                         #
#=========================================================================#

server {
    listen      %ip%:%proxy_port%;
    server_name %domain_idn% %alias_idn%;
    error_log   /var/log/%web_system%/domains/%domain%.error.log error;

    location ~ /\.(?!well-known\/|file) {
        deny all;
        return 404;
    }

    location ~* \.cgi$ {
        proxy_pass http://%ip%:%web_port%;

        proxy_no_cache     1;
        proxy_cache_bypass 1;

        add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0" always;
        add_header Pragma  "no-cache" always;
        add_header ETag    "" always;
        expires -1;
    }

    location / {
        proxy_pass http://%ip%:%web_port%;

        location ~* ^.+\.(%proxy_extensions%)$ {
            try_files  $uri @fallback;

            root       %sdocroot%;
            access_log /var/log/%web_system%/domains/%domain%.log combined;
            access_log /var/log/%web_system%/domains/%domain%.bytes bytes;

            expires    max;
        }
    }

    location @fallback {
        proxy_pass http://%ip%:%web_port%;
    }

    location /error/ {
        alias %home%/%user%/web/%domain%/document_errors/;
    }

    proxy_hide_header Upgrade;

    include %home%/%user%/conf/web/%domain%/nginx.ssl.conf_*;

}

stpl template:

#=========================================================================#
#                       Default Web Domain Template                       #
#                                                                         #
#                 DO NOT MODIFY THIS FILE! CHANGES WILL BE               #
#                    LOST WHEN REBUILDING DOMAINS                         #
#                                                                         #
#=========================================================================#

server {
    listen      %ip%:%proxy_ssl_port% ssl;
    server_name %domain_idn% %alias_idn%;
    error_log   /var/log/%web_system%/domains/%domain%.error.log error;

    ssl_certificate     %ssl_pem%;
    ssl_certificate_key %ssl_key%;

    # TLS 1.3 0-RTT anti-replay
    if ($anti_replay = 307) { return 307 https://$host$request_uri; }
    if ($anti_replay = 425) { return 425; }

    include %home%/%user%/conf/web/%domain%/nginx.hsts.conf*;

    location ~ /\.(?!well-known\/|file) {
        deny all;
        return 404;
    }

    location ~* \.cgi$ {
        proxy_ssl_server_name on;
        proxy_ssl_name $host;
        proxy_pass https://%ip%:%web_ssl_port%;

        proxy_no_cache     1;
        proxy_cache_bypass 1;

        add_header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0" always;
        add_header Pragma  "no-cache" always;
        add_header ETag    "" always;
        expires -1;
    }

    location / {
        proxy_ssl_server_name on;
        proxy_ssl_name $host;
        proxy_pass https://%ip%:%web_ssl_port%;

        location ~* ^.+\.(%proxy_extensions%)$ {
            try_files  $uri @fallback;

            root       %sdocroot%;
            access_log /var/log/%web_system%/domains/%domain%.log combined;
            access_log /var/log/%web_system%/domains/%domain%.bytes bytes;

            expires    max;
        }
    }

    location @fallback {
        proxy_ssl_server_name on;
        proxy_ssl_name $host;
        proxy_pass https://%ip%:%web_ssl_port%;
    }

    location /error/ {
        alias %home%/%user%/web/%domain%/document_errors/;
    }

    proxy_hide_header Upgrade;

    include %home%/%user%/conf/web/%domain%/nginx.ssl.conf_*;

}
9

I’ve modified the tpl template right now, it didn’t copied as expected.

10

Thank you. I’ll have to get back to that later. But, for now, enough time for a question:

I am not at all sure that I know what you mean by “Remember to rebuild the web domain after modifying the templates.”. I think maybe I’ve missed a step? (I did verify that “cgi” was not among the Proxy Extensions as per Note 2)

I hadn’t even bothered restarting any web servers; I guess I assumed that that was what hestiacp was doing while I was waiting for it to say “Changes have been saved.”. fwiw, It looks like apache gets restarted but nginx and php8.2-fpm do not.

11

From command line:

v-rebuild-web-domain YourUser YourDomain yes

Or from the Web UI, select the web domain and, from the dropdown list Apply to selected, choose Rebuild, then click the right arrow icon.

12

Thank you for the CLI reminder; I’d never before used the GUI for that, and completely forgotten about this step, likely in part because I never understood what that did. (the modified/renamed templates show up just fine w/o doing that…)

But now I’ve uploaded the 12:09 pm templates, reloaded the web domain, selected and saved the latest no_cache_CGIs, and reloaded the web domain.

The script is still buffering/caching (holding back) data until the page is complete. :frowning:

fwiw, as an aside, as best I can reckon, the pw-prot apache template mentioned in my original post seems to be what was needed to .htpasswd protect a directory under cgi-bin/ – and that it is use on the domain being used for these attempts. (when I tried running without it in this domain’s cgi-bin subdirectory, 500 err; the test mentioned in original post was for a different domain, and it did not involve any cgi-bin, and there the .htpasswd worked OK with the default template)

13

I don’t know what server/module is executing the perl script, but the output sent to the web browser says “Server type: nginx” in the http header. (Content type: text/html)

It is being served with

Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0

in the header now. :slight_smile:

14

Still no success; after lots of attempts, the script’s output is still being buffered by the server. proxy_buffering off does not help… It’s as if the buffering is either happening or being called for by something upstream of nginx?

Apache has something called fcgid “(enabled by site administrator)” … could that be the culprit? If so, can it be selectively disabled or its buffer size set to zero by a location {or <directory> section in an apache template?s

When I tried an a2dismod deflate I got a stern warning (and chickened out).

I tried a

<Directory %home%/%user%/web/%domain%/cgi-bin>
    FcgidOutputBufferSize 0
</Directory>

in a template under /usr/local/hestia/data/templates/web/apache2/php-fpm/ but hestia rejected it (saving me from myself) when I hit Save. I know what I want to try, but don’t know where/how to say it, or even if it would help; I’m still stuck/lost…

15

Currently, in the template(s) under /usr/local/hestia/data/templates/web/apache2/php-fpm/ I have:


<Directory %home%/%user%/web/%domain%/cgi-bin>
    AllowOverride All
    SetEnv no-gzip 1
    SetEnv no-brotli 1
    FcgidOutputBufferSize 0
#   SetOutputFilter slowdown;slowdown;slowdown
</Directory>

(…very unsure about that OutputFilter but it didn’t prevent the accursed buffering)

…it sure seems like some other server process is buffering the script’s output; I am of the belief that setting apache environmental variable no-gzip disables the compression/buffering of mod_deflate. (I think) I’m disabling mod_brotli. What else is there?

I’ve followed so many leads that at one point I had so many browser tabs open that the browser consumed all of memory… One lead (solution) I’ve yet to try following (in part because I don’t know how) talks long ago about switching from fcgid to FastCGI.

I’m amazed at all the MIME types already defined, but disappointed that I can’t find one to simply tell the browser how to handle the output and the server to not buffer the output.

Also, could there be something about CGIWrap that interferes with the apache AllowOverride All (or [all] other) directive(s)? I’m being told (in the apache err log) that I can’t use SetEnvIf or <FilesMatch> in .htaccess in the cgi-bin directory.