AIOps

 View Only

Nested Runbooks

By Sebastian Wegmann posted Tue December 20, 2022 06:47 AM

  

Introduction

With the latest releases of Cloud Pak for Watson AIOps 3.6 and Netcool Operations Insight 1.6.7 the included Runbook and Automation component introduces a major new capability which we are calling "Nested Runbooks".

Nested Runbooks are a somwhat strange feature: It is most likely the largest feature we have delivered for the component in the last years and still you can explain it in one sentence. There are some amazing scenarios and use cases of many clients that become easily achievable and solvable with it, but it's not obvious that you can do this. So this article will give you both: A short introduction with the cold, hard facts and an outlook of some scenarios that become possible or really easy with it.

Nested Runbooks: In a (tiny) nutshell

Here are two explanations of what Nested Runbook is about:
  1. The Nested Runbook feature provides inheritance to Runbooks.
  2. Nested Runbooks allows you to define a whole runbook as a step of another runbook.
Is that all? Yes, no, … let's explain: While the two one-liners provide a terse explanation from which you can deduce the actual impact and functionalities, these are by no means obvious. Let's explore the possibilities a little bit more.

Nested Runbooks extends the number of step types to three. You are already familiar with manual steps and automated steps. The new step type is a nested runbook step. Similar to an automated step it does not define new content by itself, but instead references a different asset. Upon adding such a step you will map all the parameters of the referenced object. This parameter mapping works just like it does for automation actions and offers the same capabilities.

Runbook editor with the new nested runbook step type highlighted


When running a runbook referring to other runbooks via the new feature, all the content of those runbooks is pulled it, the parameter mappings are resolved and you can run the runbook just like any other runbook. This means when running a runbook you have to learn absolutely nothing as everything will work just like before.

When authoring runbooks though the available possibilities just dramatically increased. You just received every runbook in your runbook library as a reusable asset to plug into every runbook you create or change. You can refer to the same runbook from any number of other runbooks which means you can efficiently split up complex runbooks into several reusable runbooks. When you update your runbook, all the runbooks referencing to it are also updated saving your potential hours of refactoring in a bigger runbook library.

Displays how a step looks like after a runbook has been chosen

This leads us to a list of key use cases:
  • Using inheritance like concepts to "write once and reuse".
  • Remove duplicating runbook steps, reuse them instead.
  • Sharing runbook authoring work across multiple people.
  • Leverage the parameter mapping functionality for runbooks.

Cold, hard facts (and witty remarks)

Here are some quick facts that apply to the nested runbook feature. They might come in handy for reference:
  • You can only reference a published runbook. Draft runbooks have to stay outside (sorry!).
  • When nesting runbooks the following limits apply:
    • Depth: The maximum depth is ten. This means you can put a runbook in a runbook in a runbook in a runbook in a runbook in a runbook in a runbook in a runbook in a runbook in a runbook. But no more!
    • Total: After resolving all runbook references a single runbook may not contain more than one hundred references in total. So for example, you can reference ten runbooks, which in turn reference ten other runbooks.
    • You don't have to tiptoe around though. If you accidentally step across the boundary we'll let you know and prevent saving such a runbook.
  • A runbook reference points to the latest version of a runbook, updating a runbook with a new version automatically changes the runbooks referring to it.
    • Make sure you update runbooks depending on the runbook you just changed. Adding or removing parameters might make the parameter mappings not work anymore.
    • Be extra careful when deleting runbooks. We warn you, if you are about to delete a runbook used somewhere else, but there is a "force" option available. However we only check latest versions for this. Previous versions of runbooks are not checked.
    • API connoisseurs may notice they can define the version of reference for nested runbooks. If you want do want to reference a fixed version instead of "floating" with latest version, you can do that as well.
  • If you are afraid of endless loops you can relax. We prevent you from ever creating a circular reference that would create such a thing.
  • A runbook containing another runbook cannot run automatically. The rule that in order to do that all steps of a runbook must be automated steps still applies.
  • Running a runbook will present you with adjusted step numbers, e.g. a runbook with three steps referenced as step 2 will show step numbers 2.1, 2.2, and 2.3 when you run the runbook. We call this representation of a runbook the "flat" representation.
  • API users can define whether they want to see the default representation or the flat representation of a runbook. The flat representation resolves all references and looks like one big runbook. The default representation will show the steps as the type they actually are and contain the information where the reference points to (Runbook ID and version).
  • Parameter mappings include
    • using a runbook parameter
    • define a fixed value
    • use the default from the referenced runbook
    • use the output of a previous automated step
    • use the logged in user

Use Cases, Scenarios, and beyond

So much information input, I might be feeling a little lightheaded. Maybe continue with some examples and an outlook? Good!


Scenario #1 - Common procedures

Let's take a look at our fictional company "ComPro". They have created a lot of runbooks and are running them as responses to their events coming in. A typical runbook in ComPro looks like this:

  1. Log in to the target system following the company security guideline, which contains of:
    1. Enter name and reason for change into a company security audit log
    2. Request a security one-time token for server access from a security service
    3. Perform the login using the security token from previous step
  2. A set of actions depending on the incident and the goal of the resolution procedure.
  3. A conditional step: In case a problem is not solved by the step(s) before, open a ticket in the ticketing system used by ComPro.
Up until now, ComPro had a lot of duplication in their runbooks. The procedures in #1 and #3 were basically the same and should remain the same throughout many runbooks. Any update needed touching a lot of runbook assets to remain up-to-date. A common problem seen among companies is that in such a case documentation becomes outdated and unreliable quickly. ComPro now can adapt in a smart way:

  1. Create one runbook to perform the log in procedure flow.
  2. Create one runbook to perform the conditional opening of a ticket.
  3. Change the existing runbooks and replace their #1 and #3 steps with the newly created runbooks.
In case the login procedure changes, you only have to update one(!) runbook. In case the ticketing system changes: You only have to update one(!) runbook. In case you automate the ticketing procedure: You only have to update one(!) runbook. You get the gist!

Scenario #2 - One runbook, many looks

In our next example we are looking at our second fictional company "Onrumalo". Here the runbook author is facing a dilemma. There is a runbook with a cool procedure available. It guides an operator to log onto a system and perform one of ten actions indicated by an integer number. He would like to make it globally available, but the procedures with number greater than five are actually disruptive and should only be done by senior engineers. Additionally the systems people should be logging on shall be limited for operators.
Luckily for Onrumalo with nested runbooks this is easy to do. Create one runbook containing everything you want with the parameters hostname and procedure as string and integer parameter. Now create different "facades" for this runbook suitable for different user groups. Each can contain a single step referring to our core runbook, but the parameter mappings will be different. Here are some options:

  • For hostname you can map this string parameter to a string parameter with an enumeration. Restricting execution to a whitelist of systems you handpicked. You can even repeat that for different user groups.
  • For procedure you can map the integer paramaeter to another integer parameter that has more restricting minimum and maximum values. In this case we reduce the value range from 1-10 to 1-5.
With these easy steps Onrumalo has now a set of runbooks which will always do the right thing and only need one update to keep their procedure up-to-date, while retaining a tight control over who can enter which values for accessing systems and which procedure is allowed to be called.

Nested Runbooks… and beyond

As you see the nested runbook feature unfolds the more you do with it and the more runbooks and automation actions you have. In this article we only touched the surface and have not yet combined the feature with automation actions at all. Watch this space for further more elaborate examples! Until then have fun nesting your own runbooks!
0 comments
14 views

Permalink