Server going on read-only mode after reboot

Community Support
1

After I reboot my system (2 productions server) I always get this issue and I’ve been having hard time to troubleshooting the problem and i don’t know how to fix this

// All PHP-FPM are down!

root@s1:~# systemctl status php7.3-fpm
● php7.3-fpm.service - The PHP 7.3 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.3-fpm.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-02-25 00:41:42 IST; 37s ago
Docs: man:php-fpm7.3(8)
Process: 814 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.3/fpm/pool.d/www.conf 73 (code=exited, status=0/SUCCESS)
Process: 662 ExecStart=/usr/sbin/php-fpm7.3 --nodaemonize --fpm-config /etc/php/7.3/fpm/php-fpm.conf (code=exited, status=254)
Main PID: 662 (code=exited, status=254)

Feb 25 00:41:37 s1.b2btech.in systemd[1]: Starting The PHP 7.3 FastCGI Process Manager…
Feb 25 00:41:42 s1.b2btech.in php-fpm7.3[662]: Tue Feb 25 00:41:42 2020 (662): Fatal Error Unable to create lock file: Bad file descriptor (9)
Feb 25 00:41:42 s1.b2btech.in systemd[1]: php7.3-fpm.service: Main process exited, code=exited, status=254/n/a
Feb 25 00:41:42 s1.b2btech.in systemd[1]: php7.3-fpm.service: Failed with result ‘exit-code’.
Feb 25 00:41:42 s1.b2btech.in systemd[1]: Failed to start The PHP 7.3 FastCGI Process Manager.
root@s1:~# systemctl status mysql.service
● mariadb.service - MariaDB 10.4.12 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Tue 2020-02-25 00:41:41 IST; 1min 1s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 792 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 679 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 669 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 659 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 792 (code=exited, status=1/FAILURE)
Status: “MariaDB server is down”

Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] InnoDB: The innodb_system data file ‘ibdata1’ must be writable
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] Plugin ‘InnoDB’ init function returned error.
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [Note] Plugin ‘FEEDBACK’ is disabled.
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] Failed to initialize plugins.
Feb 25 00:41:41 s1.b2btech.in mysqld[792]: 2020-02-25 0:41:41 0 [ERROR] Aborting
Feb 25 00:41:41 s1.b2btech.in systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 00:41:41 s1.b2btech.in systemd[1]: mariadb.service: Failed with result ‘exit-code’.
Feb 25 00:41:41 s1.b2btech.in systemd[1]: Failed to start MariaDB 10.4.12 database server.

root@s1:~# systemctl status dovecot
● dovecot.service - Dovecot IMAP/POP3 email server
Loaded: loaded (/lib/systemd/system/dovecot.service; enabled; vendor preset: enabled)
Active: failed (Result: resources)
Docs: man:dovecot(1)
http://wiki2.dovecot.org/

Feb 25 00:41:39 s1.b2btech.in systemd[1]: dovecot.service: Failed to run ‘start’ task: Read-only file system
Feb 25 00:41:39 s1.b2btech.in systemd[1]: dovecot.service: Failed with result ‘resources’.
Feb 25 00:41:39 s1.b2btech.in systemd[1]: Failed to start Dovecot IMAP/POP3 email server.
root@s1:~#

Does anyone know how to fix this!

here’s my installation setup.

bash hst-install.sh --apache yes --nginx yes --phpfpm no --multiphp yes --vsftpd no --proftpd yes --named yes --mysql yes --postgresql no --exim yes --dovecot yes --clamav no --spamassassin no --iptables no --fail2ban no --quota yes --api yes --lang en --hostname lb.b2btech.in --email [email protected] --password **** --force

2

Hi @alber

You already have opened a thread on our github site: https://github.com/hestiacp/hestiacp/issues/665

The current state is still the same, as we wrote on the closed issue.

3

moucho seems to have pointed you to a proper way for a solution: https://github.com/hestiacp/hestiacp/issues/665#issuecomment-579332391

4

Just hoping this issue get solved after Version 1.1.0 :frowning:

5

It will not, because it is a issue with your system, probaly it has something to do with quotes as muchos wrote!

6

the problem is fixed by restoring the MySQL config file on one server but still having the same issue on another one!

/etc/mysql/my.cnf // restored file.

7

image
image1277×765 57.2 KB

another stuff i notice the server which is not working after reboot shows 2 php.ini file on

https://server:8083/edit/server/php/

i hope in new version this problem should be fixed!

8

of course it shows two php.ini files, because you obviously have php7.4 and php7.3 on your system.
showing the filepaths there is intended to show you, that edits in the field below will affect settings for both versions. nothing to fix for the new version either and seems totally unrelated to the initial issue though… :thinking:

as others already pointed out the whole thing you reported seems pretty much related to something you do/add/modify on your system outside the scope of hestia and which we don’t know about…

could be as simple as running into the disk quota after the restore for whatever user, just because you activated it during your install. with a hard quota set your disk might show available space but the affected users still would not be able to write to it - which again could lead to services refusing to start…
just a wild guess though, could be plenty of other things too.

did you at least check the quotas and usage?

however, as @Raphael already mentioned please understand, that we cannot manage your servers or solve your issues. and also Hestia is not a product to do that for you.
it’s rather a tool to make tasks easier which you are expected to be able to do manually anyway… sorry, but probably we can’t help you with that :man_shrugging: :grin:

1 Like
9

If we can’t reproduce this issue or anybody else, we can’t really fix it :wink: we wouldn’t know were to begin.

10

server is having 320gb Nvme! :frowning: :expressionless:

ummm anyway thanks boss ! I’m going to reinstall the OS once the new version is released

11

image

12

image

i tried to run v-delete-sys-quota and got this error

13

As @falzo has brought it to the point, I’ll going to close this thread now…

15

one last reply to this topic:

that’s not what a quota is about and what I asked, but clearly shows that you installed quotas in the first place without knowing what it does or what it’s for :thinking: :man_shrugging:

to not be the smartarse here, quotas are per user limitations of disk space. as mentioned earlier your overall disk usage via df does not show, if you’re running into a hard limit on a quota.
repquota is more likely what you would want to look at…

reinstalling might be a good idea, I suggest you don’t enable/use quotas during hestia install next time then :wink:

2 Likes