Photos:

China pictures processed

 
0%
 
2008.04.25 @18:57 

Python Daemon Watchdog WDog

Python Daemon Watchdog WDog

WDog or WatchDog is a daemon supervisor and can even turns simple programs into a daemon. The basic functionalities looks like /etc/init.d scripts and apachectl combined.

It simplifies the execution, stop, notify, tail of log... of these daemons and take care to restart them automatically if anything has gone wrong. It can also check the space left on disks. If anything's wrong it will contact you either by email and/or on msn (usually both).

Usage :
$ wdog.py -guard    <-- manually or by crontab, in this case will trigger alerting
 or
$ wdog.py name_of_daemon [start|stop|restart|status|notify|tail [-INTEGER]]

I had first the idea of this software several years ago in the testing company where I was working, I implemented a version at that time partly during the job, partly during leisure time for a more general use for my own servers. Since this time I removed the features in connection with the company and completely rewrote and enhanced the rest of the code to be able to release it.

How it works ?

The principle is that you configure the WDog what to check and you make it check periodically by launching it every X minutes in the crontab. The WDog itself is not a daemon to avoid it being stuck or crashing. Thereafter everytime you want to act on a daemon you have configured you simply add "wdog.py" before the command.

What you can do

Here is an example of config file. At the top you can configure the two alerting mechanisms currently supported. Either send an email (all variables starting by smtp*) and MSN. To make it work you only need one msn account for the server and of course one msn account for yourself/your sysadmin. You also need your server to allow outbound connection to the outside world on port 1863.

In this example, 2 partitions are configured to be checked + 2 daemons. Partitions are quite self explanatory.

You can use as many definitions as you want AS long as they start with "0" and are in a row. (So for example 0, 1, 2, 5 will consider the [Daemon5] to be commented). " ; " are comments as well.

Daemon configuration

For the daemons you have one example with a "real" daemon that you've made (or an existing one) : [Daemon0]

This is why "daemonize=0" since daemonize defaults to 1.

A real daemon means that the software itself takes care to follow the daemon guidelines.

If you dont know or dont want to take care in each of your program this faculty, wdog can do it for you. This means in particular that it will redirect stdout and stderr to files so you can check how your program runs. See [Daemon1] to see how it works.

The notifiable option enables your daemon to be notified using WDog for a specific action such as "reload". This is done by using signal and is completely asynchronous to the current run of your process. This technic is often used for daemons such as apache. Your program needs to catch this kind of signal to proceed the request (whatever this is). The option in the WDog just means that your program handles this and so you can use it when running WDog commands.

logwatch is an option to make your WDog looks for changes in your log files. This is useful to detect activity as sometimes your program can still be in memory but completely blocked (ie, no action logged to the logfile). In this case, the lastWrite option is the number of seconds minimum before WDog will decide to restart the daemon by itself. While this may not be necessary for daemons such as apache if you dont have many users on your site, this is especially useful for schedulers that regurlaly logs their activity to the file even if it is "nothing to do, sleep".

WDog itself will log everything to its own logfile (in the same directory where it is installed) so you can keep traces of all the prior checks and all the command you've typed. It will never fill the disk since it uses rotating logs for itself.

Automate the checks

Once you're happy with the configuration, run your checks with : wdog.py -guard and then put it in the crontab such as :

*/15    *       *       *       *       /var/local/wdog/wdog.py -guard >/dev/null 2>&1

This software is considered version 0.7 and in BETA. Please test heavily your configuration and the different options on a TEST server before going to PROD environment.

You can access directly the svn at this url : http://svn.arluison.com/wdog/trunk without any password.

Let me know if you use it and if you find it useful or you have any question. 

2 Comments
  1. abrakadabra said:

    Wednesday, July 7, 2010 at 17:07

    Thanks for this. This is exactly what I was looking for. I will contact you if I can come up with some suggestions during the course of my work.

  2. Archange said:

    Wednesday, July 7, 2010 at 17:40

    You're welcome. I'm quite busy at the moment (and looking for developers actually since too much work) but you can ask, I'll try to answer asap.

Post a Comment
Comments have been deactivated thanx to screwers.