Issue
$ sudo tail -2 /var/log/nginx/error.log
2023/02/20 11:56:01 [emerg] 21114#21114: socket() 212.224.115.156:80 failed (24: Too many open files)
2023/02/20 11:56:01 [emerg] 21122#21122: socket() 212.224.115.156:80 failed (24: Too many open files)
On a server with hundreds of domains, Even after adjusting all kinds of limits everywhere, nginx keeps crashing every 2 minutest due to Too many open files (see [Feature] Discuss raising LimitNOFILE for Nginx · Issue #2431 · hestiacp/hestiacp · GitHub and Server configuration | Hestia Control Panel)
Hestia Control Panel Version
v1.6.14
Operating system
Debian 11.6 (x86_64)
Details
[email protected]:~ :) $ cat /etc/systemd/system/nginx.service.d/override.conf
[Service]
TimeoutStart=5m
Restart=on-failure
RestartSec=60
StartLimitBurst=10
StartLimitInterval=1h
LimitNOFILE=65536
[email protected]:~ :) $ sudo ps ux | grep nginx
root 862 0.0 0.0 20168 1288 ? Ss 10:54 0:00 nginx: master process /usr/local/hestia/nginx/sbin/hestia-nginx
root 19315 0.0 0.0 89992 40088 ? Ss 11:50 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
[email protected]:~ :) $ sudo cat /proc/862/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 257124 257124 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 257124 257124 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[email protected]:~ :) $ sudo cat /proc/19315/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 257124 257124 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 257124 257124 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[email protected]:~ :) $ cat /etc/default/nginx
# Defaults for nginx initscript
# sourced by /etc/init.d/nginx
# Additional options that are passed to nginx
DAEMON_ARGS=""
ULIMIT="-n 15000"
[email protected]:~ :) $ sudo nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: [emerg] socket() 212.224.115.156:80 failed (24: Too many open files)
nginx: configuration file /etc/nginx/nginx.conf test failed
[email protected]:~ :) $ head -20 /etc/nginx/nginx.conf
# Server globals
user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
# Worker config
events {
worker_connections 10240;
use epoll;
multi_accept on;
}
http {
# Main settings
sendfile on;
tcp_nopush on;
[email protected]:~ :) $ head -20 /etc/nginx/nginx.conf
# Server globals
user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
# Worker config
events {
worker_connections 10240;
use epoll;
multi_accept on;
}
http {
# Main settings
sendfile on;
tcp_nopush on;
Further Infos
Very insightful post, but even putting connection limits did not fix it: 500 error - nginx 500 (24: too many open files) - Server Fault
The whole issue is weird, because I do not think we get thousands of clients connecting at a time considering our typical customers. And the crash always happens at exactly HH:MM:01 on every even minute…
