•  I want to monitor service “x” because it sometimes stops responding (process not running).
• This particular service requires some manual intervention to get it running again (don’t get me started...).
• There is no monitoring agent installed on the system. However, I can SSH to it.
• If I run a “ps -ef <process name>”, I can see if the process is running.
• If the process state changes, I would like to notify my team via Slack with a single notification.
• I want to automate this, so I don’t have to manually check the status.

The approach:

• I will determine status by using SSH to access the system, and running a "ps -ef <process name> | grep -v grep"
  • If process is found, then status is “running”
  • If process is not found, then status is “not running”
• I want to represent the status of the process persistently, so I can check whether it has changed since I last checked to avoid sending duplicate notifications to my team.
  • Use a file to hold status value
    • 0 = running
    • 1 = not running
  • If current status matches status value in file, then don’t notify via Slack
  • If current status does not match status value in file, then notify via Slack and update the status file with current status value
• I want to check the status of my process every 15 minutes

Designing the Workflow in Concert Workflows

We start out our workflow by adding some variables that we will use to our workflow's "Start" block:

• We define two sets of Authentication SSH credentials
  • One for our target system, allowing us to run a “ps -ef” command
  • Another for our CW system so we can store the result of the “ps -ef” command persistently
• We define the process name we are checking, along with the status file name to write the result of the “ps -ef” command

Once we run the command, we need to handle what comes next – if the process is found, we take one path.  If it is not found, we take another.

We determine if we need to notify the operations team by comparing the current return value with the value that has been written in the $statusFile file.  If the current value is different than the status file value, we need to send a notification to the Slack channel.  If it is the same, then we can simply do nothing, since we have already sent a prior notification about this state.

Assuming the state has changed, we send the notification via Slack.  Then, we update the status file with the current status value so we don’t send another notification unless the status changes.

To notify via Slack, we are using a HTTP Request action block.  You will need to request a URL and bearer token from your Slack administrator for the Slack channel that you want to write a message into.  You will also need to refer to the Slack documentation to define your body.

Once the notification is sent, we need to update the status file with the current state value so that it is ready for the next workflow run comparison.  This will make sure that we don't spam our Slack channel with duplicate state messages.  We use a SSH block and simply echo the state value to the state file using the greater than (>) symbol so that it overwrites the current value.

Scheduling the Workflow

Concert Workflows has a job scheduler that we will use to run our automation workflow every 15 minutes.

The Result

So, we have designed our workflow and tested it.  The example Slack messages are shown below.  If the service check result is normal, then a NORMAL message is sent with the output of the “ps -ef” command.  If it is not normal, then a CRITICAL message is sent.

#TechnicalBlog
#Highlights-home