In this first blog article, I want to share some insight about ISVA (IBM SECURITY VERIFY ACCESS) appliance monitoring, DevOps and how they can sometime affect (negatively) each other, and how you can apply some mitigation to ensure you do not have to sacrifice one over the other.
A good way to keep a close eye on your ISVA appliance infrastructure and services health is to configure them to externalize system events they generate to centralized external consumer objects such as a SNMP Manager (in the form of SNMP traps), an Rsyslog server or even both.
This capability has been around with ISVA appliance as long as I can remember and is a key requirement in your monitoring strategy whether you aim at achieving basic monitoring and alerting user cases or a more comprehensive continuous improvement strategy mapping to core DevOps practices (Continuous Integration, Continuous Deployment and Continuous Feedback).
When configuring your ISVA appliance fleet for SNMP trap externalization (for instance), which trap should you send to your external SNMP Manager? Looking for an answer in ITIL event management processes, the answer is probably that you should not cherry pick which events should to be externalized out of your appliance fleet.
ITIL event management is about managing system events throughout their lifecycle, and that includes event detection, making sense of them, and determining the appropriate action.
Cherry picking can prove to be efficient when you know what you are looking for but problematic when troubleshooting incidents for which you don't know the root cause; the excessive filtering at the source can impact incident analysis, miss important state changes, and slow down determining quick resolution (should there be any required). Moreover, it impedes the principle of visibility in the Continuous Feedback practice of DevOps.
Surprisingly, however, applying no filtering at all on system events generated by your ISVA appliance fleet can sometime flood the SNMP Manager infrastructure if the generated system event rate becomes too high; this can happen even under normal operating conditions if you practice some form of continuous deployment. In some IBM Security Access Verify deployments, unattended Ansible automations are run daily in a Continuous Integration/Continuous Deployment fashion and in some instance, even just to perform Continuous Compliance verification (with no intended changes). All of this is pretty cool and exciting, but it was observed that the ISVA appliance tends to generate somewhat noisy (audit) system events. In particular, every time an ISVA appliance RESTAPI endpoint is invoked, the following system event is logged and relayed to any external targets configured (such as SNMP Manager):
GLGSY9025I Component component_name has successfully entered FIPS mode.
With all the DevOps CI/CD/CF practices at work, I have seen this system event summing up easily to a few million entries on a monthly basis and clog the SNMP Manager inbound queues unnecessarily, even triggering flooding alerts.
Keeping your SNMP Manager IT partners happy, by not flooding their system with countless repetitive system events such as the one above (which does not bring values in the area of Event Management), is a good thing.
Luckily, an RFE was raised in the spring of 2020 and an implementation of system alert muting is now available with the ISVA 10.0.1.0 release (December 2020). Before that release, there was no other way of selectively muting system event at the source beside unconfiguring altogether the external SNMP Manager object from the ISVA appliance fleet, with the undesired effect of losing visibility over the entire fleet’s health and behavior.
The following instructions guide you on how to selectively mute system event generated by the ISVA appliance using the appliance Local Management Interface.
In this particular example, I’ve decided that I wanted to stop relaying the GLGSY9025I system (audit) event to my external SNMP Manager in order to improve the signal to noise ratio:
- Connect to the ISVA appliance’s Local Management Interface
- Go to the Advanced tuning Parameters page (System->System Settings->Advanced Tuning Parameters)
- With the “New” button, add a new parameter using the following:
-
- Key: "events.response.snmp.ignored"
- Value: "GLGSY9025I"
- Comment: "Reduce noise sent to external SNMP Manager"
- Hit “Save Configuration”, “Review Pending Changes” and “Deploy”.
Alternatively, and assuming that you are using IBM Security’s open-source Python module along with the isam.ibm Ansible Collection, the following Ansible playbook fragment applies the above advanced tuning parameter but in a more manageable fashion in a large appliance fleet:
- name: setup system settings on all nodes
hosts: all
vars:
username: "admin@local"
password: "passw0rd"
roles:
- role: ibm.isam.set_advanced_tuning_parameter
ansible_connection: "ibm.isam.isam"
advanced_tuning_parameters:
- advanced_tuning_parameter_key: "events.response.snmp.ignored"
advanced_tuning_parameter_value: "GLGSY9025I"
advanced_tuning_parameter_comment: "Reduce noise sent to external SNMP Manager"
There are other ways and combination to mute generated system events, and across a variety of external configured objects (email, Rsyslog and Event Log), not just SNMP. The documentation note IBM Security Access Verify - Muting selected system alert events that you can find in the reference section covers in more depth all the possible combinations.
But the important thing to retain from this blog is that there exist ways to maintain a harmony between your desire to adopt as much as possible DevOps practices for managing your ISVA appliance fleet (and at your own pace) while addressing pain points showing up occasionally, and without sacrificing on monitoring and visibility.
For more details on the ISVA appliance monitoring, see the following references:
#ibmchampions-highlights-home#ibmchampions-highlights