The signal and the noise or the needle in the haystack. That has always been the challenge in security and Guardium is no exception. How do you monitor effectively when you are overseeing thousands of databases and each one produces an average of 1,000 errors per day for a potential of millions of errors per day?
The answer is analytics – a key focus area for Guardium Big Data Intelligence (GBDI). But what does “analytics” actually mean? The dictionary says: “the systematic computational analysis of data or statistics,” and that’s certainly a good description of what GBDI is all about.
There are 6 different pre-built analytical engines within GBDI, along with several analytical services, including a powerful noise-canceling engine that is a key weapon for filtering massive volumes of raw data.
The noise-canceling engine uses various clustering and de-duplication algorithms to reduce the size of a large data set by various orders of magnitude. Obvious targets include errors/exceptions, a query data set or even session profiles, but the engine can be applied to virtually any data set in order to transform raw data into a more efficient representation. An especially valuable use case is lowering the cost and impact of large data sets on downstream systems. For example, if events are sent to a SIEM such as QRadar or Splunk, rather than sending billions of raw events, tens of thousands of “summary events” make the SIEM just as effective from a correlation or “big picture” perspective but incur far less cost. Another use is to leverage noise cancellation to help an analyst understand Guardium data better. Rather than looking at millions of individual log lines the system clusters them together by understanding what is common and what is different and clearly showing this to the user via the GUI.
As an example of a simple signal-within-the-noise application, a large healthcare organization quickly discovered that by using GBDI they could ignore a large chunk of their database errors from a security perspective, but not ignore it from a change management perspective. Using noise-canceling analytics, GBDI showed that if the client looked at the entire aggregate of data there was no clear pattern, but when they clustered and analyzed the data over a 180 day window, a clear pattern quickly emerged. When they looked at all the errors they could not see the pattern because of the volume, but analytics “dissected” the data and showed them the patterns (conceptually shown below). When they saw the pattern they immediately understood that it involved a faulty change management process in which passwords were changed without necessary changes to a number of tools. And while this was not necessarily a direct security concern, it caused the security analysts to ignore all the errors – including those that might imply a security event. Using GBDI they could immediately filter out this “explained root cause case” and let the security analysts deal with a tiny fraction of the generated errors (and at the same time fix their change management process).
Figure: Analytics Can Identify Security-Related Errors While Reducing Noise from Periodic Errors
One of the core challenges of every database activity-monitoring program is the sheer volume of data that must be captured, managed and inspected. Guardium is the proven world leader in capturing this raw data, and adding Guardium Big Data Intelligence to the architecture paves the way to dramatic improvements in the ability to leverage automation and analytics to sift thru TB’s of raw data and isolate the key events needed to achieve even greater insights from your security and compliance efforts.