Woke up to crashed server. Hestia 1.9.3

1

I woke up today with my client complaining of “No Disk Space left on device” error on their websites. I logged into server, and the server storage was really full, I checked root folder, there was hst_backups and hst_install_backups taking loads of storage. I cleaned some storage by deleting some logs files, I tried to open hestia login page. Below is the screenshot what is coming when I open https://server_hostname:8083/login page.

image
image2138×814 34.8 KB

I guessed, A new update might have arrived and as usual it doesn’t asks any permission before updating and auto updates. I was right, Hestia 1.9.3 arrived …

I appreciate for the new version. But plz also check this issue as it might have hampered many user’s not having enough disk space before auto updating.

2

Sorry but no disk has nothing to do with the update…

And because the no disk availble probally the update failed

Login via ssh … make some space free try clearing tmp folders / backup folders

And reinstall hestia-nginx / hestia-php and hestia again

Then add some monitoring service

3

to 100%, server monitoring is out of scope of hestia. Please ensure that your server isnt running out of disk due to the common monitoring software (like icinga2, nagios, etc.).

4

No Raphael, but what I meant is, I saw in many other software, there is some preventive measure taken by automatic update scripts before updating the system. Like they calculate the risks, like if the space left in device is < 20GB which might be required in updation, the script will not run and show the warning to users to free the space.

I was talking about such preventive measures.

Thanks

5

hestia update doesnt take more than 100mb storage which means your server was already full or 99.9%… I dont think there is a need for such a prevention.

6

Then what is this hst_backups folder taking 10gb of space almost in my server.

image
image1010×836 118 KB

Thanks

7

backup fail, remove the files, run v-backup-users and check the output.

8

Please, show the output of this command:

find /root/hst_backups -size +1M
1 Like
9 
root@server2:/home/galler# find /root/hst_backups -size +1M
/root/hst_backups/120820230525/conf/nginx/on
/root/hst_backups/280320230525/conf/nginx/on
/root/hst_backups/120720230525/conf/nginx/on
/root/hst_backups/210920230525/conf/nginx/on
/root/hst_backups/060520230525/conf/nginx/on
/root/hst_backups/070420230525/conf/nginx/on
/root/hst_backups/251020230525/conf/nginx/on
/root/hst_backups/050220250525/conf/nginx/on
/root/hst_backups/250820230525/conf/nginx/on
/root/hst_backups/200720230525/conf/nginx/on
/root/hst_backups/281120230525/conf/nginx/on
/root/hst_backups/270820240526/conf/nginx/on
/root/hst_backups/060220250526/conf/nginx/on
/root/hst_backups/010720230525/conf/nginx/on
/root/hst_backups/180520230525/conf/nginx/on
/root/hst_backups/230320230525/conf/nginx/on
/root/hst_backups/010820230525/conf/nginx/on
/root/hst_backups/200420230525/conf/nginx/on
/root/hst_backups/050520230525/conf/nginx/on
/root/hst_backups/280220250525/conf/nginx/on
/root/hst_backups/250720230525/conf/nginx/on
root@server2:/home/galler#
10

What’s that on file?

ls -lh /root/hst_backups/120820230525/conf/nginx/on
file /root/hst_backups/120820230525/conf/nginx/on
11 
root@server2:/home/galler# ls -lh /root/hst_backups/120820230525/conf/nginx/on
file /root/hst_backups/120820230525/conf/nginx/on
-rw-r--r-- 1 root root 97M Aug 12  2023 /root/hst_backups/120820230525/conf/nginx/on
/root/hst_backups/120820230525/conf/nginx/on: ASCII text, with very long lines (451)
root@server2:/home/galler#
12

Did you check that file? Because that file shouldn’t be there.

head -n20 /root/hst_backups/120820230525/conf/nginx/on
tail -n20 /root/hst_backups/120820230525/conf/nginx/on
13

Yes, that file exist, and it contains tons of Web access logs.

14

The default configuration doesn’t leave any log in that file or in the directory.

So, if that is a recent log file, I would check the Nginx configuration to determine which directive is generating that log. If it is an old file, I would remove it.

15

Running below commands in order solved the issue

sudo su
apt reinstall hestia-nginx
apt reinstall hestia-php
/usr/local/hestia/bin/v-add-sys-dependencies