Monit is a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system. It is often used to restart daemons on failures (e.g., a daemon crashed) or if an abnormal situation occurs (e.g., a daemon consumes too much CPU or RAM). Sometimes it is also used as a “poor man’s IDS” to detect changes into critical files and / or daemon binaries.
However, this functionality does not play well with package managers.
Consider a situation:
monit is configured to watch MySQL and restart when it crashes:
check process mysqld with pidfile '/var/run/mysqld.pid' start program = "/usr/bin/systemctl start mysql" stop program = "/usr/bin/systemctl stop mysql" if failed host 127.0.0.1 port 3306 protocol mysql with timeout 10 seconds then restart if 5 restarts within 5 cycles then timeout depends on mysql_bin check file mysql_bin with path /usr/sbin/mysqld if failed checksum then unmonitor if failed permission 755 then unmonitor if failed uid root then unmonitor if failed gid root then unmonitor
Then, when you run
yum update, and there is an update for
yum will stop
mysql daemon, replace its binary, and then start
What can go wrong here?
monitactivates in the middle of the process, detects that mysql is no longer running, and tries to start it.
/usr/sbin/mysqld‘s checksum is now different, and “unmonitors” MySQL (which means it will not be automatically restarted on crashes).
The solution is obvious: stop
yum is run and start it afterwards. If you are OK with stopping and starting
monit manually, then everything is great (but note that
yum can be called by some external software such as a server control panel).
apt, the solution is pretty easy: we can use
DPkg::Post-Invoke hooks to stop and start monit. Unfortunately, I am not aware of similar configuration options for
yum supports plugins, and we can write a custom plugin to stop and start monit automatically.
import os import yum from yum.plugins import TYPE_CORE requires_api_version = '2.1' plugin_type = (TYPE_CORE,) def pretrans_hook(conduit): conduit.info(2, 'Stopping monit') command = '/usr/bin/systemctl stop monit.service' os.system(command) def posttrans_hook(conduit): conduit.info(2, 'Starting monit') command = '/usr/bin/systemctl start monit.service' os.system(command)
This plugin (
/usr/lib/yum-plugins/monit.py) uses two hooks:
pretrans (called before yum begins the RPM update transaction), and
posttrans (called just after yum has finished the RPM update transaction). The first one is used to stop monit, the second one starts it, it is that simple.
The only things that is left is to enable this plugin. To do that, we create a file called
/etc/yum/pluginconf.d/monit.conf with this content:
I have also created a GitHub repository with all the code.