AIOps: Monitoring and Observability - Group home

The OMEGAMON for Storage Tivoli Enterprise Portal (TEP) interface  now enables the user to execute Storage Toolkit actions across multiple systems with a single situation.

Before this feature, storage administrators who used the Storage Toolkit in situations and wanted to issue actions on multiple managed systems had to manually perform a two-step process:

1. Create a unique situation with the same formula for each system.
2. Create the same Storage Toolkit actions for each system.

Now by simply distributing existing situations for new systems, you no longer need to do these manual tasks to support Storage Toolkit actions.

Example of the new feature

Let’s review how distributed situation toolkit requests can be used with a simple example: send an e-mail notification from various systems when the value of some attribute is greater than the threshold value. For example, to receive an e-mail notification if the free space percent of any SMS Storage Group is less than 5% from several SYSPLEXES.

Create a new situation named STOR_GROUP_SPACE_CRITICAL_LOW using the “S3 Volume Group Summary” Attribute Group with Free Space Percent as the Attribute Item as shown below:

Set the formula to be triggered if Free Space Percent <= 5.0 and change the sampling interval to be the same as the RMF interval as shown below (your RMF interval may be different):

On the ‘Distribution’, tab, shown below, assign all systems that should send e-mail notifications (formula will be validated in these systems):

On the ‘Action’, tab, shown below, change the radio buttons to, ‘Storage Action’, ‘Take action on each item’, (to receive an e-mail for each Storage Group) and, ‘Take action in each interval’. The Request Location table now contains a Status column which will show status information about each Storage Toolkit action for each system. It’s empty here because I have not defined an action for any node.  Press the ‘Storage Toolkit’, button outlined in red, and proceed with Storage Toolkit action creation:

Because this is a new situation without any associated Storage Toolkit actions, I can just create a new Batch Job as shown below that will send the e-mail from selected Nodes.  I have chosen the ‘Group Type’ ‘SMS_GROUP’ since I want to use SMS Storage groups.

To prepare the JCL used to send e-mail notifications for each Storage Group that has triggered the situation, I can use the same dataset name on all SYSPLEX’ as shown below, or I can edit actions later to use unique datasets for each SYSPLEX. You can add as many variables as there are attributes, in this case, I am only interested in: ‘Free Space Percent’,  ‘Free Space MB’, ‘Group name’ and ‘Managed System’.

Optionally, the ‘Action name’ can be changed in the, ‘General’, Tab and JCL Job card in the ‘JCL’, tab.

After pressing the ‘OK’ button you will see that for each selected system, the Storage Toolkit action has been created and the status is OK. If the action is deleted using the Storage Toolkit workspace or if the TEMS is offline you will see the corresponding status.

After pressing the ‘Apply’ button the situation will be refreshed.  The situation will be triggered and if there are storage groups that fit the formula, an email e-mail notification will be sent as shown below:

I have just created a situation that will send e-mail notifications if any storage group from any of 3 systems has less than 5% free space. If I want to review storage groups from another system, I just add the new system in the Distribution tab for the situation and use the Storage Toolkit button on the ‘Action’ tab again. Use, ‘Edit Existing Node’, radio button and specify the new system in the Node table:

On the ‘Create Batch Job’ under, ‘Command’, all variables and fields are filled in from the previously defined action. Check that everything is acceptable for the new system: dataset containing the batch JCL, JCL Job card etc. and if everything looks good, press the ‘OK’ button. The new system will be added to the table. Press, ‘Apply’ to save changes.

To remove the system association with the situation, select it in the table, press the, ‘Clear’, button and remove them from the assigned table in the ‘Distribution’ tab.

Summary

In the example above an e-mail was sent by the Batch job option. However, you can use manually defined and predefined commands as well. You can also prepare more complex JCL which could activate volumes for the storage groups with critically low free space or create other actions to generate more space for the affected storage groups. The storage group example is just one example, you can use any set of interested attributes items from any attribute group.

Now you can easily add new managed systems to existing situations.  This enables you to:

• Easily distribute Storage Toolkit actions associated with a situation across multiple systems.
• Simplify the administration of situations.
• Avoid creating and managing multiple situations which do the same filtering for multiple systems.
• Quickly extend the functionality of existing situations which already use Storage Toolkit actions.
• Easily change Storage Toolkit actions for a situation which is distributed across multiple systems.

Distributed Storage Toolkit actions is available via the following APARs (PTFs):

OA60192 (UJ04974) – for OMEGAMON for Storage on z/OS 5.5.0

OA60753 (UJ04891) – for OMEGAMON for Storage on z/OS 5.4.0

#OMEGAMON #IBMZ #IBMZOS #AIOps #Storage