This article is a follow-up to my Introduction to QRadar Log Management for WebSEAL Administrators. In this article we will start with a business question that I was asked to answer, and show how I used QRadar event correlation rules to figure it out.
The Business Problem
The WebSEAL-protected application environment that I have been running in support of the IBM Bluemix platform includes a service which allows an application to perform single sign on. This service can also be asked (as part of a consuming application’s SSO request) to verify a user’s email address, and it does so using the one-time-password product capability of Tivoli Federated Identity Manager (TFIM).
I started receiving a lot of complaints that browser-users were being prompted for a one-time-password (OTP), but never receiving it in their inbox. I was occassionally seeing this behaivour myself, however not always, and the same mail server was successfully being used for other email operations without a problem. We didn’t know how big the problem really was and needed more information to try to figure out what was going on. Then one of the business owners asked a simple question:
Can you tell me how many people are being sent these one-time-passwords and then not submitting them, and who they are?
Decomposing the problem
What seemed like such a simple question was actually not all that easy to answer. I started looking at my QRadar-managed request log entries (which initially didn’t contain as much information as they do today) to see what I could figure out. I found there were interesting patterns of records that allowed me to determine:
- When an OTP was sent out (issued)
- When an OTP was submitted, and whether or not that was successful
Here’s a sequence of events from WebSEAL, as seen in QRadar, for a successful OTP flow. Notice they are ordered descending sequence number (most recent at the top), and are all for the same session index:
Let’s look at a request log entry for an issued OTP. That is sequence number 8. Here’s the raw record:
<114>1 2014-08-26T02:06:29-04:00 web-dev webhttp 28398 – – U:http://www.ibm.com/2700011TPA AL:2 SI:037d7a6a-2ce7-11e4-ad10-000c29ec17e4 T:8 M:GET E:sweeden@au1.ibm.com R:www.ibm.com ST:200 B:1208 F:000808195 URL:/fimotp/sps/xauth?Target=https://idaasdev.stage1.ng.bluemix.net:443/sps/oauth20sp/oauth20/authorize?client_id=1C2ZW1MDNtsSHBZNaift&response_type=code&scope=profile&redirect_uri=https://idaasdevtestclient.stage1.mybluemix.net/index.jsp&state=38870bc7-24ee-4c6a-bcb9-64d977939e19&requestedAuthnPolicy=http://www.ibm.com/idaas/authnpolicy/otp&AuthenticationLevel=3
And here’s how it looks in QRadar after being parsed:
Similarly, lets look at the request log entry for a successfully submitted OTP. That is sequence number 10. Here’s the raw record:
<114>1 2014-08-26T02:06:49-04:00 web-dev webhttp 28398 – – U:http://www.ibm.com/2700011TPA AL:3 SI:037d7a6a-2ce7-11e4-ad10-000c29ec17e4 T:10 M:POST E:sweeden@au1.ibm.com R:www.ibm.com ST:302 B:1764 F:000155717 URL:/fimotp/sps/authservice/authentication?action=phaseThree&CSRFToken=epu6h73i
And here’s how it looks in QRadar after being parsed:
Notice that in the case of a successful OTP the isam_http_authlevel changes to 3 (it was 2) – this is how you can determine that it was a successful submission rather than a bad OTP.
The other interesting thing to note about the use of the TFIM OTP verification process is that it must be done on the same HTTP session as the OTP was issued – TFIM stores the issued OTP in a distributed memory cache keyed from the user’s session. The lifetime of an issued OTP is controlled by configuration, and the lifetime in this environment was 30 minutes.
Picking a correlation rule
Given that we can detect when an OTP is issued and also when one is successfully submitted, the questioned remained, how can I determine those cases where an OTP was issued but NOT successfully submitted, given that I will have no record for that. Enter QRadar correlation rules. First let me state, in English, what the correlation rule is that I want to implement:
I want to detect when an OTP is issued and then there is NOT a valid OTP submitted on the same HTTP session within 30 minutes.
I dived into the QRadar Offences tab (where Rules live) and clicked Actions -> New Event Rule to take a look at what was available. Following the wizard, I chose to create a rule based on events, and came across a large list of rule templates. It took a while to understand all the variants, but finally I found one that sounded promising:
What is particularly key about the above rule template is the “none” piece and the “with the same event properties” piece. The “none” directive allows me to base the rule evaluation on something I *can* detect (a validly-submitted OTP). The “with the same event properties” piece allows me to do correlation between the events matched by the first and second rules – in particular correlation of the same isam_http_session value. This rule template is however based on “other rules”, which I did not have yet. In QRadar you can use special building-block rules for this purpose. Building-block rules are like other rules except they don’t need to have a specific response action since they are only used in conditions of other rules.
Creating the building block rules
In order to use my chosen correlation rule, I needed to create building block rules for:
- Identifying an event when an OTP is issued
- Identifying an event when an OTP is correctly submitted
The correlation rule I found earlier will handle the “NOT” part for me, based on the same http session too. Great – I know how to find these events, because I have identified URL patterns and other mathcing logic which determine them. Let’s look at each in turn.
Building block rule for OTP Issued
Detecting when an OTP is issued is quite simple, look for a URL starting with the path:
/fimotp/sps/xauth?
I used the rules wizard to create a building-block rule based on events, with the key matching criteria shown here:
Building block rule for OTP Submitted
Detecting when an OTP is submitted is a little more involved. I need to check for a matching URL, *and* a particular value of the isam_http_authlevel.
Again I used the rules wizard to create a building-block rule based on events, with the key matching criteria shown here:
Creating the correlation rule
Going back to the correlation rule pattern I picked earlier, I can now populate the rule as shown:
This rule has a response, which is to create a new event called OTP Not Issued, as shown:
Testing the correlation rule
Rules cannot be run on historical data – they only work on realtime data. To test this rule I initiated a one-time-password flow, but didn’t enter the value when prompted, and waited 30 minutes with baited breath, watching the Log Activity view of the QRadar console with a filter set to show only my new event type. Well, not quite. Really in my test environment I changed the 30 minutes to 1 minute to avoid impatience, and exercised the test case. The OTP Not Submitted event did appear, as shown:
Pausing the log activity view, and opening the event, I can see the following detail:
Even the username was filled in – that was great, almost.
Searching and Reporting
Next I wanted to be able to send daily reports of all these events, grouped by username so that I only showed one row for each user no matter how many times they tried and failed. I was able to do this by creating a custom search in the Log Activity view, and the creating a Report in the Reports view that used the custom search.
The filtering and column grouping for the custom search looked like this:
Executing this search showed rows such as this:
I was able to configure a report that looked very much like this and have it automatically generated on a daily basis and emailed out to stakeholders. I was almost a superhero.
Whilst fairly impressive at this stage, the reporting wasn’t exactly what I was after because the username used by WebSEAL and sent in request log entries was not a human-recognizable field. Instead it was an internal ID that mapped to a human-recognizable value, and that mapping was stored in an external data source. What I really wanted was a QRadar report that was built after that ID->Username mapping was done. I couldn’t use the email address I already had in the original otp_issued event because when you use a rule response to create a new event, only standard QRadar event properties are contained in the new event (not custom event properties). I also didn’t want to compromise my Username attribute field in QRadar by populating it with the non-unique email address. Instead I wanted a report that contained several attributes for the user – their Useranme as seen by WebSEAL, their human-readable username, and the email address they used.
It turns out that to do this I needed to set my otp_not_submitted rule response to “Send to Forwarding Destinations”, have an external process listen for that and do the lookup of the human-readable username, and then send new event data back to QRadar as a new custom event (using the LEEF event format). Reporting would instead be done on these new events which contained lots of rich information. Performing that out-and-back loop, configuring custom events, and using LEEF will be the subject of my next article.