There are several reasons why you may want to be able to accurately reproduce a stream of SNMP trap packets, independently of the source host that generated them. These include; - for use in reproducing a particular scenario to debug some code or integration - for developing new Netcool/OMNIbus probe rules or Netcool/Impact policy code - to do pre-rollout regression testing of new code to ensure compatibility - any scenario wherein the traps need to be ingested into a system other than the production instance
There are several options to accomplish this replay function within the Netcool/OMNIbus probe environment, including: - using the inbuilt Raw Capture feature to convert incoming SNMP packets into tokens. These can then be replayed using the Standard Input (Stdin) probe, whereby the tokens can be directly used to test the probe rules logic, i.e.
$ cat capturefile.cap | nco_p_stdin -server NCOMS -rulesfile snmptrap.rules - capturing the traffic via tcpdump and viewing in a tool like Wireshark. This could then be manually converted at that point into an equivalent Raw Capture format for use in the above replay method.
So why would we try to go any further than these solutions? It may help to describe what kind of output we will attain from the methods just described, and why these may not give adequate replay solutions for all situations.
Firstly, let's consider the case where we can enable the
Raw Capture feature of the Netcool SNMP probe while the engineering team can replicate the conditions which trigger the SNMP alerts. These are then converted into tokens by the probe and logged in the probe capture file for replay. While we now have the capture file ready for replay, there is a considerable effort involved in its generation, possibly including interacting with Change Management to get this feature enabled.
Let's now consider the second scenario, where we may not have direct access to the production instance of the SNMP probe in order to enable the Raw Capture feature. In this case, we may be able to get the engineering team to record the tcpdump file for us. We now have to manually recreate the capture file via copy & paste or direct input for each varbind within the SNMP packet.
Obviously each scenario described above presents some challenges for the OSS/Operations and engineering teams in terms of coordination and/or manual effort. The resulting capture file itself and the Stdin input method have some drawbacks in addition to this, namely; - the standard input probe will replay events at the rate at which they are received, which may give a false representation of the rate at which the alarms occur, e.g. that all alarms occurred within a second or two. This occurs as a result of using cat to replay the file to the Stdin probe. This may be counteracted by adding the following line to end of the rules file, which will result in the LastOccurrence field of the Netcool/OMNIbus event being set to the value at which the SNMP alert was captured in the capture file.
@LastOccurrence = $ReceivedTime - even with the correct values set in @LastOccurrence, some logic in the system may depend on events appearing after a particular time period, regardless of whether they have suitable deltas in their @LastOccurrence field. This could be for example a suppression policy, which may set a flag in certain events at the beginning of the stream, and clear this flag as a result of events occurring later in the stream. If the packets/alerts were received at the correct cadence, then this policy would be triggered. In the case where all events arrive essentially simultaneously at the ObjectServer, this policy may not fire at all, and the suppression behaviour may not materialise.
So what's the alternative? Clearly it would be nice to be able to replay this stream to our alternative SNMP probe (Dev/Staging/etc.) at the original recorded pace, and have this be a repeatable and a light touch procedure.
- Let's imagine our capture file is from a production network, with SNMP packets flowing from 10.1.1.1 to our SNMP probe on 10.1.1.5.
- We also have a development system which has a SNMP probe running on 172.16.3.5.
- We need to be able to capture this stream intended for 10.1.1.5 and replay it to 172.16.3.5.
Let's work step by step: 1. Capture (or have engineering capture) the initial stream via tcpdump and write it to a file: tcpdump -i eth0 port 162 -w /tmp/snmp-prod.pcap 2. Utilise the tcprewrite feature of the
tcpreplay utility to rewrite the destination IP address and ethernet address on the capture file. Let’s check what we need to rewrite the MAC and IP address to. Let’s run ifconfig on our staging server: [root@staging-server ~]# ifconfig eth0 eth0: flags=4163 mtu 1500 inet 172.16.3.5 netmask 255.255.255.0 broadcast 172.16.3.255 ether 08:00:27:6b:45:bb txqueuelen 1000 (Ethernet) 3. We can see that our new IP is 172.16.3.5 and the new MAC is 08:00:27:6b:45:bb. Let’s now rewrite the capture file to reflect that. tcprewrite -i snmp-prod.pcap -o snmp-staging.pcap --dlt=enet --dstipmap=0.0.0.0/0:172.16.3.5/32 --enetdmac=08:00:27:6b:45:bb 4. Now we have the correct capture file for replay on our staging environment, we need a tool to replay this feed. Thankfully,
udpreplay by Erik Rigtorp will do the job nicely. We transfer our
new staging capture file snmp-staging.pcap to this server and execute via: udpreplay -i eth0 snmpstaging.pcap 5. We now have the original SNMP stream replayed in our staging environment whenever we need it.
Finbarr Hennessy
IBM Champion - Cloud