In the [Service].
As I said in my post, the real change that made it work for me was to raise root’s limit specifically, and test it with ulimit -n as the root user.
1 Like
I’ve got 395 domains configured. With the changes I documented in this thread
1030 named
1139 php-fpm8.
2110 mariadbd
7398 nginx
114563 apache2
1 Like
All limits are raised and verified same as yours.
This is getting frustrating, also for our users…
Did you put the nginx.conf setting back to 65536?
# Server globals
user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
1 Like
yes, same exact lines
I manage 2 servers with 3000 websites on 1 user (not a very good choice on my opinion, but the client want it that way).
I have change the config to log everything on 1 access.log file for nginx and 1 for apache.
Removed the / domain log, remove the byte log format…
This helped a lot…
Issue returned, I am getting clueless…
@Sich can you share your configuration so I can try if it resolves the issue?
Hi,
The main change that I have done is to log everything at 1 place.
First NGinx :
In the template file (/usr/local/hestia/data/templates/web/nginx), comment the access_log that are setup and add the new one :
# access_log /var/log/%web_system%/domains/%domain%.log combined;
# access_log /var/log/%web_system%/domains/%domain%.bytes bytes;
access_log /var/log/nginx/access.log host;
To be able to see wich vhost in concerned by each log you have to create a logtype. This is in /etc/nginx/nginx.conf :
Add this line :
log_format host '$host $remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent"';
After that on Apache, depend on how you have setup your server about the template file. Same here, disable the default log files and add yours :
# CustomLog /var/log/%web_system%/domains/%domain%.bytes bytes
# CustomLog /var/log/%web_system%/domains/%domain%.log combined
# ErrorLog /var/log/%web_system%/domains/%domain%.error.log
CustomLog /var/log/apache2/access.log vhost_combined env=!dontlog
The env=!dontlog is to avoid logging all static ressources, usefull when you have a lof of activity. If you do want to log access to static ressources, remove env=!dontlog
Now you have to edit /etc/apache2/apache2.conf
Change the line with vhost_combined to be like that one :
LogFormat "%v %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
mostly change %v:p to %v and %h to %a (if I remember correctly).
Add a new error log format to get the vhost name in the log :
ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] %7F: %E: [client %a] %M% ,\ referer\ %{Referer}i %v"
And add the env to not log static file :
setenvIf Request_URI ".(jpg|jpeg|png|css|gif|ico|js|webp)$" dontlog
Once you have done all of this, creating new template, making change to configuration files, then you have to apply the new template to all your website. Some scripts with the cli will help for that.
Make sure to test on few site before applying to everything.
Note : this will break your awstats if you use some of them.
One master process has a soft limit of 1024 I just saw my log in the first post, maybe that i the root issue?
I have not been able to alter that in any way, despite plowing through posts like linux - How to set nginx max open files? - Stack Overflow
Apache or NGinx ?
nginx, see above Too many open files despite config adjustments
latest message from /var/log/nginx/error.log: 2023/09/27 22:01:29 [emerg] 183427#183427: open() "/var/log/nginx/domains/webmail.CENSORED.com.log" failed (24: Too many open files)
hm that script to regenerate templates would be handy
and awstats is not required, but would be handy to have working
did you check the worker_rlimit_nofile in nginx.conf ?
Did you set a LimitNOFILE in systemd for nginx ?
How did you setup the limits in security/limits.conf ?
Did you try to increase the limit far more than 65k ?
Did you restart your server after all those change ?
After reading the whole thread it seem that you have done those change.
But you should check the limit for apache2 too, because nginx and apache run with the same users.
For apache check the /etc/apache2/envvars file and this line :
APACHE_ULIMIT_MAX_FILES='ulimit -n 65536'
And about awstats, you need to configure it yourself with custom logtype if you want to use a unique log file. Or you will need to keep 1 log file / domain.
[EDIT]
What give you thoses command :
sysctl fs.file-max
sysctl fs.aio-max-nr
[/EDIT]
Yep, I have made countless raises and reboots. This issue already sucked up hours that might have been better invested splitting up our instance or something…
$ sysctl fs.file-max fs.aio-max-nr
fs.file-max = 70000
fs.aio-max-nr = 65536
Now trying your configuration hacks with rebuild via Force Reload From Template,
though these configurations should probably go into respective subfolders so they can easily be toggled.
Why is this 1024 limit still there after raising everything?
janek@main:~ :) $ sudo ps ux | grep nginx
root 759 0.0 0.0 21084 1384 ? Ss 12:20 0:00 nginx: master process /usr/local/hestia/nginx/sbin/hestia-nginx
root 1684 0.0 0.7 188236 131536 ? Ss 12:20 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
janek@main:~ :) $ sudo cat /proc/759/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 71810 71810 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 71810 71810 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Now this is odd:
janek@main:conf :) $ sudo ps ux | grep nginx
root 43483 0.0 0.0 21148 1372 ? Ss 12:32 0:00 nginx: master process /usr/local/hestia/nginx/sbin/hestia-nginx
root 49767 0.0 0.3 126792 66048 ? Ss 12:43 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
janek@main:conf :) $ sudo lsof -p 43483 | wc -l
22
janek@main:conf :) $ sudo lsof -p 49767 | wc -l
1252
janek@main:conf 1 $ sudo lsof | awk '{print $1}' | sort | uniq -c | sort -h | tail -5
2084 php-fpm7.
7328 named
15039 nginx
30265 mariadbd
66235 apache2
which nginx process is opening all those files?
Something we cant really say without access to the server…
ah I missed the workers, I think I might not be too far from the solution:
$ sudo ps aux | grep nginx
root 43483 0.0 0.0 21148 1372 ? Ss 12:32 0:00 nginx: master process /usr/local/hestia/nginx/sbin/hestia-nginx
admin 43484 0.0 0.0 22312 8088 ? S 12:32 0:00 nginx: worker process
root 49767 0.0 0.3 126792 66048 ? Ss 12:43 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
www-data 49768 2.6 0.3 127032 72588 ? S 12:43 0:09 nginx: worker process
www-data 49769 0.0 0.3 127032 71864 ? S 12:43 0:00 nginx: worker process
www-data 49770 0.0 0.3 127032 71076 ? S 12:43 0:00 nginx: worker process
www-data 49772 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49773 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49774 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49775 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49776 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49777 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49778 0.0 0.3 127032 68008 ? S 12:43 0:00 nginx: worker process
www-data 49779 0.0 0.3 127032 67788 ? S 12:43 0:00 nginx: cache manager process
please, can anyone help here?
sysctl limits definitely seem to be irrelevant:
$ sudo sysctl fs.file-max fs.file-nr fs.aio-max-nr fs.aio-nr
fs.file-max = 9223372036854775807
fs.file-nr = 7168 0 9223372036854775807
fs.aio-max-nr = 1048576
fs.aio-nr = 352
``
https://www.kernel.org/doc/html/latest/admin-guide/sysctl/fs.html#file-max-file-nrthis I get when the restart fails:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] open() "/var/log/nginx/domains/webmail.DOMAIN.error.log" failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed
but initiating v-restart-proxy from the CLI works just fine:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
so how can I reproduce this?