Too many open files despite config adjustments

Issue

$ sudo tail -2 /var/log/nginx/error.log
2023/02/20 11:56:01 [emerg] 21114#21114: socket() 212.224.115.156:80 failed (24: Too many open files)
2023/02/20 11:56:01 [emerg] 21122#21122: socket() 212.224.115.156:80 failed (24: Too many open files)

On a server with hundreds of domains, Even after adjusting all kinds of limits everywhere, nginx keeps crashing every 2 minutest due to Too many open files (see [Feature] Discuss raising LimitNOFILE for Nginx · Issue #2431 · hestiacp/hestiacp · GitHub and Server configuration | Hestia Control Panel)

Hestia Control Panel Version

v1.6.14

Operating system

Debian 11.6 (x86_64)

Details

[email protected]:~ :) $ cat /etc/systemd/system/nginx.service.d/override.conf
[Service]
TimeoutStart=5m
Restart=on-failure
RestartSec=60
StartLimitBurst=10
StartLimitInterval=1h
LimitNOFILE=65536
[email protected]:~ :) $ sudo ps ux | grep nginx
root         862  0.0  0.0  20168  1288 ?        Ss   10:54   0:00 nginx: master process /usr/local/hestia/nginx/sbin/hestia-nginx
root       19315  0.0  0.0  89992 40088 ?        Ss   11:50   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
[email protected]:~ :) $ sudo cat /proc/862/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             257124               257124               processes 
Max open files            1024                 524288               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       257124               257124               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
[email protected]:~ :) $ sudo cat /proc/19315/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             257124               257124               processes 
Max open files            65536                65536                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       257124               257124               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
[email protected]:~ :) $ cat /etc/default/nginx
# Defaults for nginx initscript
# sourced by /etc/init.d/nginx

# Additional options that are passed to nginx
DAEMON_ARGS=""
ULIMIT="-n 15000"
[email protected]:~ :) $ sudo nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] socket() 212.224.115.156:80 failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed
[email protected]:~ :) $ head -20 /etc/nginx/nginx.conf
# Server globals
user                    www-data;
worker_processes        auto;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

# Worker config
events {
        worker_connections  10240;
        use                 epoll;
        multi_accept        on;
}

http {
    # Main settings
    sendfile                        on;
    tcp_nopush                      on;
[email protected]:~ :) $ head -20 /etc/nginx/nginx.conf
# Server globals
user                    www-data;
worker_processes        auto;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

# Worker config
events {
        worker_connections  10240;
        use                 epoll;
        multi_accept        on;
}

http {
    # Main settings
    sendfile                        on;
    tcp_nopush                      on;

Further Infos

Very insightful post, but even putting connection limits did not fix it: 500 error - nginx 500 (24: too many open files) - Server Fault

The whole issue is weird, because I do not think we get thousands of clients connecting at a time considering our typical customers. And the crash always happens at exactly HH:MM:01 on every even minute…

See [Bug] · Issue #3286 · hestiacp/hestiacp · GitHub

https://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/

I went through this exact post and tried all suggestions over there, as you can see in my terminal log :confused:

No still missing:

ulimit -Hn
ulimit -Sn

I know some VPS software can impose some limits it as well…

1 Like