Introduction
In my article “#BeyondtheDSMGuide: How Real time is Real Time? Streaming data into QRadar vs. ingesting it via an Object Storage Bucket”, I discussed the concepts around where and why latency is introduced during data ingestion into QRadar. In this article, I’ll review some specific examples in AWS:
QRadar/Amazon Integration Overview
QRadar has several integrations with Amazon Services and we support ingestion in a variety of methods that include both Object Storage and Streaming.
QRadar has two protocols for ingesting data from AWS: (1) our Amazon Web Services Protocol which supports ingestion via Kinesis Data Streams, SQS, and CloudWatch Logs; and (2) our integration with AWS S3 which supports ingesting data from an AWS S3 bucket.
Amazon Kinesis Data Streams
Kinesis Data Streams (here) is Amazon’s “massively scalable and durable real-time data streaming service”. Many AWS services can publish their logs and findings directly to a Kinesis Data Stream; QRadar can ingest data directly from a Kinesis Data Stream. Ingesting data directly from an AWS Service into QRadar via Kinesis happens in real time; any latency would be introduced if there was a lag between an event happening and it being written to Kinesis. AWS Services can also stream data to an S3 via Kinesis Firehose and QRadar can pull the data from AWS at that point. We will dive deeper into S3 integrations a little bit later.
Amazon Simple Queue Service (SQS)
In 1H 2021 we added the ability to directly ingest data from Simple Queue Service (SQS) to QRadar (here). Queues with SQS are subscribed to by publishers (i.e. Amazon Simple Notification Service (SNS)) and then those queues are polled by receivers. In our example QRadar would poll SQS. QRadar can poll SQS rapidly for example at 10 second pollings. Therefore, data can be ingested from SQS almost as fast as it is available and is “near real time”.
Amazon Simple Storage Service (S3)
Many organizations choose to store their security relevant data in an S3 bucket for analytics. QRadar can pull data that is stored in an S3. It is made aware of the location of data in an S3 bucket either by polling continuously from a pre-defined directory pre-fix or from a location provided by an SQS notification.
In an integration between S3 and QRadar leveraging SQS, QRadar can subscribe to the SQS notifications at 10-second intervals. This means that the data can be polled from S3 into QRadar almost as soon as it is available in bucket. For an integration that leverages directory prefix, QRadar supports a 60-second polling interval. This means that it is close to, but not exactly, real time. However, in this instance we need to take into account how data is being written to S3.For example, AWS CloudTrail is on a 15-minute delay and only writes to S3 every 5 minutes. Similarly, AWS VPC flow logs are sent to an S3 bucket at 5-minute intervals.
Amazon Cloud Watch Logs
Another popular service in Amazon is Cloud Watch logs which enables users to monitor, store, and access files from Amazon EC2, AWS CloudTrail, Route 53, and other services. QRadar can then subscribe to a real time feed of events from Cloud Watch logs that is delivered via Kinesis Data Stream (here).
As QRadar can directly subscribe to feeds from Cloud Watch logs there is a short time lapse from the time data is written to Cloud Watch logs until it is ingested into QRadar. However, there may be latency in how fast the data is written to Cloud Watch logs. For example, Amazon CloudTrail is on about a 15 minute delay due to the SLAs in AWS (here). So although it’s streaming, it won’t be real-time due to the required delay incurred on collection to ensure things aren’t missed.
For reference, our architects suggest that data ingestion via CloudWatch logs should not exceed 5,000 EPS. For AWS ingestion that exceeds 5,000 EPS, organizations should leverage an S3 Bucket or Kinesis Data Streams.
Conclusion
Hope you found this helpful! Please reach out if you have any further questions! Big THANK YOU to Chris Collins for his contributions to this post!
Thanks,
Wendy