proc_check tool for High Availability

Introduction:

The proc_check program is for supplementing heartbeat(http://linux-ha.org/) by handling local restarts of a process (if it is possible). It watches local resources that are supposed to be running as defined by the heartbeat resources file (/etc/ha.d/haresources ). If the application is discovered to have gone down then proc_check attempts to restart it. If it cannot be restarted, then a failover is forced by shutting down the local heartbeat.

The program requires that all applications are controlled by SysV style init files in /etc/ha.d/resource.d or in /etc/rc.d/init.d. These files must process a 'status' arg, returning 'not running' or 'stopped' in the string if the service is not running. It must also process a 'restart' arg.

If any of the applications in the 'haresources' file are to be ignored, then that entry should be put into the hardwired 'ignore' list. This is useful to prevent proc_check from trying to manipulate such things as ipchains which are not really server-type processes.

Installation:

• 1. Check the source file proc_check.c and adjust the handful of hardwired configuration settings appropriately:
  
  

  /* ========================================================================================== */
  /* HARDWIRED CONFIGURATION SETTINGS */
  
  // syslog facility setting
  #define FACILITY LOG_DAEMON
  // syslog severity setting
  #define SEVERITY LOG_NOTICE
  
  // how long to wait between checks when in daemon mode (seconds)
  #define DREAM_TIME 30
  
  // how many times to locally restart the process before giving up and failing over
  // (only applies when in daemon mode)
  #define RESET_THRESHOLD 4
  
  // the network interface that is up when serving as the HA master
  #define HA_INTERFACE "eth0:0"
  
  
  /* list whatever resources are to be ignored here, NULL terminate the list */
  char *ignore[] =
  {
  "ipchains",
  NULL
  };
  
  
  /* ========================================================================================== */
  

• 2. Just typing make should be sufficient to build proc_check. Then just copy it to someplace reasonable (say /usr/local/sbin ).
  The program can be run in either of two ways:
  • -- single run. This will cause proc_check to go through the applications listed in 'haresources' one time and then exit when it is done. (any restarts that were required will be attempted, and a failover is forced if it cannot be done). This mode is suitable for use from cron. It is the default mode of proc_check
  • -- daemon mode. This will cause proc_check to become a daemon and to go into an infinite loop, periodically testing all the applications on its own. In this mode, proc_check keeps track of how many times an application has been reset, after RESET_THRESHOLD times, it will no longer attempt a local restart and will force a failover. This mode will also listen for a SIG_HUP signal, which will cause it to re-read the 'haresources' file when it is received. The SIG_HUP will also zero out the reset counts (the reset count is also set to zero after forcing the failover). This mode is invoked by the command: proc_check -d

  In either mode, running proc_check will have no effect on the HA standby system, it only influences the processes on the system that is in control of the HA resources at the time. All the actions of proc_check are logged to syslog with the facility and severity controlled by FACILITY and SEVERITY.