Hello,
I confirm that 4096 is sufficient for my use case but it needs to be added to the systemd unit of nginx, otherwise it does not work.
worker_rlimit_nofile directive in nginx config is not enough (can be lower than in the systemd unit but not greater than).
By default on Debian Buster the systemwide limit is set to 1024.