Monit is a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system. It is often used to restart daemons on failures (e.g., a daemon crashed) or if an abnormal situation occurs (e.g., a daemon consumes too much CPU or RAM). Sometimes it is also used as a “poor man’s IDS” to detect changes into critical files and / or daemon binaries.

However, this functionality does not play well with package managers.

Consider a situation: monit is configured to watch MySQL and restart when it crashes:

Then, when you run yum update, and there is an update for mysql available, yum will stop mysql daemon, replace its binary, and then start mysql again.

What can go wrong here?

  1. monit activates in the middle of the process, detects that mysql is no longer running, and tries to start it.
  2. monit detects that /usr/sbin/mysqld‘s checksum is now different, and “unmonitors” MySQL (which means it will not be automatically restarted on crashes).

The solution is obvious: stop monit before yum is run and start it afterwards. If you are OK with stopping and starting monit manually, then everything is great (but note that yum can be called by some external software such as a server control panel).

With apt, the solution is pretty easy: we can use DPkg::Pre-Invoke and DPkg::Post-Invoke hooks to stop and start monit. Unfortunately, I am not aware of similar configuration options for yum. However, yum supports plugins, and we can write a custom plugin to stop and start monit automatically.

This plugin (/usr/lib/yum-plugins/monit.py) uses two hooks: pretrans (called before yum begins the RPM update transaction), and posttrans (called just after yum has finished the RPM update transaction). The first one is used to stop monit, the second one starts it, it is that simple.

The only things that is left is to enable this plugin. To do that, we create a file called /etc/yum/pluginconf.d/monit.conf with this content:

Done!

I have also created a GitHub repository with all the code.

How to Integrate monit with yum
Tagged on:         

Leave a Reply

Your email address will not be published. Required fields are marked *