/var/cache/nginx/fastcgi_temp/xxxxxx" failed (13: Permission denied) and Error 500 on any file upload or download over 2mb (sometimes less)

Eveything has been going smoothly for months but the last week, I have been getting users getting errors uploading images over 2mb and today, I couldn’t upload or download large CSV files from multiple sites I host.

See an error like this in the error.log each time. Below is an example but different websites and different plugins do the same thing. No changes to my php or nginx configs. Everything is well above 2-3mb limit.

2022/04/15 11:58:17 [crit] 1068#1068: *1722073 open() “/var/cache/nginx/fastcgi_temp/7/12/0000000127” failed (13: Permission denied) while reading upstream, client: 184.161.112.84, server: siteandinsight.com, request: “GET /wp-admin/tools.php?page=acui&tab=export&nonce=96b5cb73cb&action=download_user_csv&filename=user-export.csv HTTP/1.1”, upstream: “fastcgi://unix:/run/php/php7.4-fpm-siteandinsight.com.sock:”, host: “www.siteandinsight.com”, referrer: “Log In ‹ Site & Insight — WordPress

On the sites themselves, most of the time I get
"Internal Server Error
Oops! Something went wrong.
The server encountered an internal error or misconfiguration and was unable to complete your request.
Error Code: 500

Update: Just tried with a file that 500kb and it had the issue.

Any help would be great!

Are you using any custom setup for cache?

/var/cache/nginx/fastcgi_temp is normally not writeable by the user that would explain the error…

Nope. Everything should be pretty much default and nothing has changed in months (maybe a year now).

Is there a way to reset things to make sure this is doesn’t happen?