Automating Process Restarts with CopperEgg Webhooks

Overview

Several of our customers have asked for the ability to take action automatically under certain conditions. CopperEgg supports this capability using webhooks, as described in detail in CopperEgg Webhooks Part 1 and Part 2. The most requested use case is described as follows:
  • For a given set of monitored servers (cloud, virtual and/or physical),
  • Detect when a specific process is not running, and
  • Without user intervention, restart that process.
What follows is a walk-through on setting up a specific example: restarting nfsd if / when it stops running.

Setting up the Copperegg Process Alert and Webhook Notification

Here are the steps to follow:
  • Create a tag called ‘nfsd’
    • Servers tab -> Tags (left nav panel) -> New Tag
  • Apply this tag to all servers running nfsd
    • Servers tab -> Servers (left nav panel) -> Details -> click + next to Tags
  • Create an Alert Definition, set to trigger on Process List does not contain nfsd, and enable WebHook notification
    • Alerts tab -> Configure Alerts (left nav panel) -> New Alert
    • Description : Process nfsd not running
    • Match : Tags
      • Tags field : nfsd
    • Comparison : Process List
        • Comparator : !~ (does not contain)
        • Value : nfsd
      (note, this is case sensitive)
    • HTTP / WebHook : type in the URL of your webhooks handler
    • Click Save at the bottom of the form
When any of the tagged servers is discovered to have no active nfsd process, a webhook post will be sent to the URL you entered.

Setting up the Copperegg Webhooks Handler

CopperEgg has written a sample webhooks handler, intended to get you started on implementing your own custom webhooks. Our hope is that once you see how easy it is to use webhooks, you will see how flexible and powerful the interface is, and start expeimenting with new ways to use it that we haven’t thought of yet! You will find the sample code at copperegg_webhooks_handler. We covered how to set up the sample code on your local test machine in (link to Part 2), if you need a quick refresher.

Tying it all together

The last piece of the puzzle is to edit your webhooks handler to add code to execute the automatic response, once loss of the nfsd process is detected.
  • cd into your cloned copperegg_webhooks_handler directory
  • open the file myapp.rb with a text editor
  • on line 227, you will find the Lost Proc Handler named handle_lost_proc() . Notice that this method is called from only one method, handle_process_alert() (line 134), and is only called when the comparator is “does not contain”.
  • On line 133 in handle_lost_proc(), you will see code that checks whether the process missing is “nfsd” … if it is, our ‘automated process restart’ code executes!
And that is how its done! Note that the ‘restart nfsd’ code is commented out, and specifies a server URL of 127.0.01. To make this code work on your site, you need to
  • Change the IP address to that of the alerting server, and enter the username and password of a user with sufficient privelages to restart the nfsd service.
  • Verify that the nfsd restart command is correct for your system.
  • Uncomment the code.

Taking it to the next level

The copperegg_webhooks_handler is NOT production code; it provides a skeletal framework for parsing webhooks, providing a simple way to insert your own automated response methods. Tweet and share any exciting new ways to use CopperEgg webhooks via your Github account.

Related Posts