SystemD auto restart services (Debian, Nginx only)

I had both my php or my nginx service stop a couple times during the year. So to get them to auto restart, I edited this file:
usr/lib/systemd/system/php7.4-fpm.service
In the [Service] section, after the last ExecReload line, I added the Restart=always:
ExecReload=/bin/kill -USR2 $MAINPID
Restart=always

Then I ran:
systemctl daemon-reload
and also restarted the php service.
Then I did the same for nginx.service (after the ExecStop line).
So far, so good. But I wonder, did I act correctly? Placing my line at the very end after the ExecReload or ExecStop lines?

And will these get overwritten in an update? Perhaps I need to copy these 2 edited files and put them in:
/etc/systemd/system/

And if these services ever stop again, and are auto restarted, is that all logged? In which file? Would it be possible to create a Cron job to parse that log, and then notify of a stop and restart? And any way to figure out what caused them to stop? (They stopped separately, in different months).
Thanks for ideas.

If you want those services to restart even when they terminate with an exit code of 0, then yes.

The order doesn’t matter.

Usually, yes.

No, you want to create an override.conf file. Just execute systemctl edit yourservice and create the directives you want to add or override, and save the file. systemd will create the override file for that service and won’t be removed, changed, etc. when the service updates.

Example:

systemctl edit php7.4-fpm.service 

And now just create the directives you want to add or override in the space assigned for this task:

### Editing /etc/systemd/system/php7.4-fpm.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
Restart=always

### Lines below this comment will be discarded

### /lib/systemd/system/php7.4-fpm.service
# [Unit]
# Description=The PHP 7.4 FastCGI Process Manager
# Documentation=man:php-fpm7.4(8)
# After=network.target
#
# [Service]
# Type=notify
# ExecStart=/usr/sbin/php-fpm7.4 --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf
# ExecStartPost=-/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74
# ExecStopPost=-/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74
# ExecReload=/bin/kill -USR2 $MAINPID
# Restart=on-failure
#
# [Install]
# WantedBy=multi-user.target

Once saved, it will create this file:
/etc/systemd/system/php7.4-fpm.service.d/override.conf

In /var/log/syslog and in systemd journal.

Example of how to look for automatic restarts of the php7.4-fpm service:

grep 'php7.4-fpm.*Scheduled restart job' /var/log/syslog
journalctl -u php7.4-fpm.service | grep 'Scheduled restart job'

Yes, you must create the script but of course, it’s possible.

Their logs should contain that info.

2 Likes

Wow! Thanks, Sahsanu, for all that detailed info!!! Especially on how to do the override, as I really was clueless on that. I should be able to write the cron job myself.

1 Like

I’ve never been able to use Nano; I’m partially blind and its not accesible at all.
So a much easier way for the override: I simply used WinSCP to remotely create and edit a text file(using an accessilbe text editor on my laptop), & put it here:
/etc/systemd/system/php7.4-fpm.service.d/override.conf
And I simply put my changes in it; I did not bother to add the comments, as it won’t serve for anything?:
[Unit]
StartLimitIntervalSec=0

[Service]
Restart=always
RestartSec=1s

Then I ran:
systemctl daemon-reload
This will work to override any updates?
I did the same for:
/etc/systemd/system/nginx.service.d/override.conf

2 Likes

personally, I don’t like the idea of ‘randomly restarting services’ in the middle of the night.

that sounds like stuff we had to do on Windows NT 4.0 days.

What I DO find really useful?
I have a ‘MariaDB Watchdog Script’ that checks the service (once a minute) and if it doesn’t see that MariaDB is already running then it starts the service.

This has saved me ~20 hours of work since I wrote this 8 weeks ago.

I can share if you want. I’m not the best with Cron. It took me about 3-4 hours to get the script perfected.

1 Like

Perfect, you can also do it that way.

The comments are only useful when using systemctl edit, if you created the file manually you don’t need to add them.

Note: Keep in mind that your approach is a bit aggressive, and if the service encounters any problems, you could end up with an endless restart loop.

Yes.

1 Like

@aaronkempf: From what I read or understood, MariaDB already has a built in restart service, which is why I didn’t need to edit its systemD service. (I edited for Nginx and php services).
Nevertheless, if you have a working script, it could be useful for people to see (and could be edited for other services). I’m learning about cron scripts now. So yes, go ahead and post it and I’ll read it. Thanks.

Here is my code. I kept finding that my service would abend. I have this running every minute now, and it will

  • check the service
  • start if it’s not running
  • send me an email if it’s not running

#!/bin/bash

# Variables
LOG_FILE="/var/log/mariadb_service_monitor.log"
SERVICE_NAME="mariadb" # Change this to "mariadb" if that's your service name

# Log function
log_message() {
    local MESSAGE="$1"
    echo "$(date +'%Y-%m-%d %H:%M:%S') - $MESSAGE" | tee -a "$LOG_FILE"
}

# Check service status and log output
if systemctl is-active --quiet mariadb; then
    echo "$(date +'%Y-%m-%d %H:%M:%S') - Service 'mariadb' is running" >> /var/log/mariadb_service_monitor.log
else
    log_message "Service '$SERVICE_NAME' is not running. Attempting to start it."
    if systemctl start "$SERVICE_NAME"; then
        log_message "Successfully started the '$SERVICE_NAME' service."
    else
        log_message "Failed to start the '$SERVICE_NAME' service. Please check manually."
    fi
fi

Off topic: Abend… That reminds me of the time I worked as a mainframe operator :wink:

1 Like

this has saved me countless hours in the last 2 months. It works perfectly.

Easy, proactive monitoring of processes, programs, files, directories, filesystems and hosts | Monit This is working very well for me. Give it a try.

1 Like

Do you have any recommendations on ‘how to install’?
i’d love to consider ‘monit’.

I just don’t know what it does.

My mariadb watchdog script does exactly what I want.