Security Global Forum

This article is the third in my series of using QRadar with ISAM WebSEAL. Previously I have written about:

• Introduction to QRadar Log Management for WebSEAL Administrators
• QRadar Event Correlation Rules

We the left story at the end of the last article with a working correlation rule, however a problem existed in that the rule response created a new event, and this new event did not contain the detailed information about a user and their email address that I wanted in my reports. The new event only contains standard QRadar event attributes, not any of the custom attributes that were in the original WebSEAL request log event. Further, there was some data (a login username) which needed to be looked up from an external data source for reporting purposes. This article will show how you can hook in an external process to receive events triggered in a rule response, and then build a completely new event to be sent back to QRadar. These new, enriched-data events, can then be used for reporting purposes.

The process diagram for this entire exercise looks like this:

Changing the Rule Response to a Forwarding Destination

First step to this process is to go modify the otp_not_submitted rule to have a different rule response. Rather than “Create a new Event”, set a forwarding destination, as shown:

I used the Manage Destinations link to create that destination, which is essentially UDP on “127.0.0.1:1234”. Of course nothing actually listens there yet – that’s what we will look at next.

Creating a simple event listener using socat

What I’ve done on my QRadar system is add a special script that listens on port 1234 for UDP events. I don’t recommend you do this precisely the same way – instead I recommend you run such a listener on another machine – external to the QRadar virtual appliance, and specify the IP address of that system in your forwarding destination rather than localhost. Really you should not install your own software on the QRadar VM as it could affect support and migration/upgrade compatibility.

I also know there are other approaches besides this one to setting up your own destination listener for event forwarding. For example one of my colleages, David Druker, has created a Security Directory Integrator (SDI) assembly line for this same purpose. I was trying to write a minimal listener, using basic Linux tools, and that’s what lead to the approach I have taken.

On the Redhat Linux machine where the listener runs (in my case the QRadar box itself), I set up a script that runs on boot (from /etc/rc.local) to listen for forwarded events:

nohup socat -u udp4-listen:1234,fork,reuseaddr,bind=127.0.0.1 SYSTEM:/root/process_line.sh > /root/mylog.out &

Note the use of the socat utility – it’s the swiss army knife of network tools, and an optional package for Linux that I thoroughly recommend. This command starts a UDP listener on port 127.0.0.1:1234 and each time there is data to process it will call the process_line.sh script, passing the received data as stdin. Any logging information echo’d from process_line.sh ends up in the mylog.out file. If you were running this on a machine external to QRadar, you would want to bind the socat listener to the external IP of that machine rather than the localhost/loopback IP of 127.0.0.1.

Next let’s take a look at what process_line.sh does. It contains some rather unobvious tricks, which took me quite a while to figure out and are well worth exploring.

#!/bin/bash

#
# e.g. of input (sometimes these arrive non-newline terminated and may have more than one event concatenated together into one big long string)
# <114>1 2014-04-01T20:15:37-05:00 web-2 webhttp 19733 – – U:http://www.ibm.com/2700011TPA AL:2 SI:3aa3ce64-ba04-11e3-b394-000c2969c3bd T:11 M:GET E:sweeden@au1.ibm.com R:www.ibm.com ST:200 B:1207 URL:/fimotp/sps/xauth?Target=…
#

#
# This little trick uses non-blocking IO to deal with stdin not being newline terminated
#
myline=`dd iflag=nonblock bs=1 count=1 2>/dev/null`
while [ “$myline” != “” ]
do
LINES+=$myline
myline=`dd iflag=nonblock bs=1 count=1 2>/dev/null`
done

# debug
echo initial processing: $LINES

#
# Sometimes multiple events come in and are read at once, so use this pattern to split them
#
echo $LINES | sed -e “s/3<114>1 /3n<114>1 /g” |
while read LINE
do
# debug
echo processing record: $LINE

#
# get the ibm unique id, email address, session index and transaction number from the line
#
IBMUID=`echo $LINE | grep -o ” U:[^ ]*” | sed -e “s/ U:http://www.ibm.com///”`
IBMEMAIL=`echo $LINE | grep -o ” E:[^ ]*” | sed -e “s/ E://”`
SESSIONINDEX=`echo $LINE | grep -o ” SI:[^ ]*” | sed -e “s/ SI://”`
TRANSACTIONID=`echo $LINE | grep -o ” T:[^ ]*” | sed -e s”/ T://”`
DEVTIME=`echo $LINE | grep -o “^<114>1 [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]” | sed -e “s/<114>1 //”`

#
# Was this even a valid event line? Check for presence of non-null IBMUID
#
if [ “$IBMUID” != “” ]
then
#
# Use an external process (however you want) to do any attribute mapping.
# For this demo I’ve hard-coded something simple straight into the script
#
IBMUSERNAME=”unknown”
if [ “$IBMUID” = “2700011TPA” ]
then
IBMUSERNAME=”ShaneWeeden”
fi

#
# Create our final enriched event. Note the 2000001 is a qidmap entry returned from qidmap_cli.sh, and I manually mapped that to the QID after sending through one event.
#
FINALEVENT=`date “+%b %d %H:%M:%S”`” otpnotsubmitted LEEF:1.0|IBM|QRadarDemo|1.0|2000001|sev=8tdevTime=$DEVTIMEtdevTimeFormat=yyyy-MM-dd’T’HH:mm:sstusrName=$IBMUSERNAMEtotpEmail=$IBMEMAILtibmUniqueid=$IBMUIDtsessionIndex=$SESSIONINDEXttransactionId=$TRANSACTIONID”

# debug
echo -e FINALEVENT: $FINALEVENT

# send back to QRadar (change the IP if your listener is running on a box outside of QRadar)
echo -e $FINALEVENT | socat – udp-sendto:127.0.0.1:514
fi
done

The script is fairly well commented. The really tricky part was understanding that QRadar can send blocks of events together, and the dd utility with non-blocking io was needed to grab what was available and process it. The rest of the example breaks apart the original event, extracting desired attributes using grep and sed, and then performs a user mapping (this is where you could consult an external datasource for example). Finally a new LEEF event string is created and sent back to QRadar, again using socat to do this.

A LEEF event is just a string with a fixed header, and then a variable set of tab-separated name=value pairs. Some of the attribute names are standard QRadar event attributes, and others are custom to my event. You can read more about LEEF events in the QRadar LEEF Format Guide, which I found on the Intelligence and Analytics Developerworks Wiki.

The real process_line.sh script on my system uses curl to call to an external Web API to map the IBMUID to a real username, however as you can see I’ve removed that for obvious reasons and left an example in it’s place. There’s still a few jobs to do on QRadar though to get our new event recognized and parsed correctly.

Creating a new QRadar event using qidmap_cli.sh

On the QRadar system, logged in as root on the command line, I ran the following utility to create a new event type:

/opt/qradar/bin/qidmap_cli.sh -c –qname ‘OTPLoginFailure’ –qdescription ‘A one-time-password was issued that the user never submitted’ –severity 5 –lowlevelcategoryid 3005

The only non-obvious parameter in this is the lowlevelcategoryid of 3005. This represents “Misc Login Failed” and you can find all the available values via:

/opt/qradar/bin/qidmap_cli.sh -l

When creating the new event, the qidmap_cli utility returned the following text. Note the QID of 2000001, which is why the process_line.sh script contains that value in the LEEF header:

Created entry:
qid: 2000001
name: OTPLoginFailure
description: A one-time-password was issued that the user never submitted
severity: 5
low level category id: 3005
ratethreshold: 0
catpipename: Echo
rateshortwindow: 0
ratelongwindow: 0
reverseip: false
rateinterval: 0

Invoking operation: forceNotification ( )
Result: true

Creating a Log Source for the LEEF Events in QRadar

In the same way that we manually created log sources for WebSEAL in the QRadar admin console, the same needs to be done for receiving and parsing LEEF events that are sent from the process_line.sh script.

On the QRadar admin console, navigate to the Admin tab, open the Log Sources, and add a new Log Source, as shown:

Note that the Log Source Identifier (otpnotsubmitted) matches a field in the LEEF event string generated in process_line.sh

After creating the log source, QRadar will require a “Deploy Changes”.

Sending test events

In this section I will show you how to use socat to send test events to each of:

• Directly to QRadar to exercise the new LEEF event source, to test out LEEF events and perform a one-time mapping to tell QRadar what the event is.
• The socat listener (event forwarding destination) so that we can test the process_line.sh script

Sending a LEEF Event to QRadar and Performing the One-Time Map Event

If you looked thoroughly at the process_line.sh script above, you’d already know how to send a LEEF event to QRadar. It’s contained within that script. As my listener (and socat) is on the QRadar machine itself, I can send a LEEF event to QRadar with the following example run from a command prompt on the QRadar VM:

echo -e `date “+%b %d %H:%M:%S”`” otpnotsubmitted LEEF:1.0|IBM|QRadarDemo|1.0|2000001|sev=8tdevTime=2014-08-27T11:48:00tdevTimeFormat=yyyy-MM-dd’T’HH:mm:sstusrName=testusertotpEmail=testuser@ibm.comtibmUniqueid=0987654321tsessionIndex=12345ttransactionId=1″ | socat – udp-sendto:127.0.0.1:514

Watching all raw events in realtime on the Log Activity tab shows me the new event arriving:

Pause the live stream, and double-click the new event to open it:

Whilst the Username and Log Source Time attributes are correct, the Event Name, Low Level Category and Event Description are all shown as unknown. Use the Map Event button, and search for your new QID (2000001) to map this event to the correct type. You will see a message indicating all future events of this type will be mapped automatically.

Now go back to the live stream, send another test event with the socat utility, pause the live stream, and open the new event and you will see it correctly mapped, as shown:

It also makes a lot of sense at this point to create new event property regexp extractors for the other custom name/value pairs that we included in the LEEF event payload. This is something that we already did for custom attributes in the WebSEAL request log events in the first article in this series. Here’s a table showing the attribute names and regexp values that I used for my new OTPLoginFailure LEEF events:

Having set up those extracted properties, they should now be shown with the event:

Having the extracted custom properties will allow us to perform a search (and report) on them.

Sending a request log event to the forwarding desitnation

Another use of socat is to send a test event to the event listener we have configured as the forwarding destination for the correlation rule that detects an OTP that has not been presented by a user. The actual event that gets sent by QRadar to the forwarding destination is the request log entry that was originally identified as an issued OTP, but did not have a matching successful OTP submitted. I’ve captured one of these from the exercise in the previous article, and can send it to the socat listener as shown:

echo -e “<114>1 2014-08-26T02:37:09-04:00 web-dev webhttp 28398 – – U:http://www.ibm.com/2700011TPA AL:2 SI:498551aa-2ceb-11e4-ad10-000c29ec17e4 T:8 M:GET E:sweeden@au1.ibm.com R:www.ibm.com ST:200 B:1208 F:001313416 URL:/fimotp/sps/xauth?Target=https://idaasdev.stage1.ng.bluemix.net:443/sps/oauth20sp/oauth20/authorize?client_id=1C2ZW1MDNtsSHBZNaift&response_type=code&scope=profile&redirect_uri=https://idaasdevtestclient.stage1.mybluemix.net/index.jsp&state=09f7d8d3-7724-4d12-bf8a-1f14038f2dfa&requestedAuthnPolicy=http://www.ibm.com/idaas/authnpolicy/otp&AuthenticationLevel=3” | socat – udp-sendto:127.0.0.1:1234

Whilst displaying realtime OTPLoginFailure events in the log activity view, and tailing the mylog.out file that the process_line.sh dumps tracing to, we can see that events sent to the forwarding endpoint are processed correctly:

Creating the Final Search and Report

Now that failed OTP’s are being detected, enriched with extra user information, and sent back to QRadar, I was able to create a custom search on the new OTPLoginFailure events, grouped by username, as shown:

Creating a report that utilized this search gave me nice PDF documents that were auto-emailed out to business owners on a daily basis, such as this one (PII redacted):

What was the REAL business problem?

You may recall that at the start of my previous article I described a business problem. There were lots of complaints of users failing to receive emails containing their one-time-passwords, and that this was the main reason we were observing OTP failures. Well, why was that?? It turns out that from the reports I was able to generate, we could see that almost all failed OTP’s were for IBM employees – those users with an email address ending in “.ibm.com”. That was very obvious once you looked at the QRadar reports. This quickly led us to figure out that the problem was not with sending the emails, but receiving them. A couple of phone calls later and we had a root cause – SPAM filters on inbound IBM email servers were quarantining the one-time-password emails. There was a quick fix for this, and the problem went away.

Conclusion

This series of articles has presented three key topics related to the integration of WebSEAL and QRadar:

• Configuration of WebSEAL and QRadar for custom request log management
• Using correlated queries in rules within QRadar to answer business questions
• Using a forwarding destination to enrich event data and create new events for business reporting

I hope you can take away some of the patterns I have shared in this series of blog posts on integrating QRadar and WebSEAL request logs, and using correlation queries and patterns for event data enrichment to produce detailed business-level reports for your deployment. I certainly learned a lot when building this solution, and now feel empowered to extract all kinds of use statistical information from searches on the request log entries, with very minimal effort.