IBM QRadar

Hello,

we configured a ClodTrail log source to collect logs from our customers'AWS instance; we do not use SQS queues but we pull them directly from bucket.

Customer is collecting logs from several AWS region in a single bucket (the bucket belongs to eu-west-1 region), each region has a subdirectory inside the bucket..for example we have:

....../CloudTrail/eu-central-1/2022/02/....

....../CloudTrail/ap-east-1/2022/02/...

If we configure directory prefix ...../CloudTrail/ and .* as file pattern, are we sure we are going to collect any file inside any subdirectory and so we get all the events?

If not, is there a way to collect all the files (in json.gz format) from any sudirectory inside a single directory prefix?

Thanks

Davide

Hi,

Restriction: A log source using directory prefix can retrieve data from only one region and one account, so use a different log source for each region and account. Include the region folder name in the file path for the Directory Prefix value when you configure the log source.

If you have log sources in an S3 bucket from multiple regions or using multiple accounts, use the Amazon AWS S3 REST API protocol with an SQS queue instead of with a directory prefix.

Reference:

https://www.ibm.com/docs/en/dsm?topic=caaclsbuaasrap-configuring-amazon-aws-cloudtrail-log-source-that-uses-s3-bucket-directory-prefix

Due to the basic way S3 works, fetching a stream of new objects (logs) from it is inherently unreliable unless you use SQS notifications. There is no way to make the "directory prefix" method reliable.

Your best bet is to implement an SQS queue for any S3 bucket delivering logs. It is not hard.

If you have a very high volume of logs, you may encounter a performance problem resulting in QRadar ingestion falling behind log generation. If this happens, simply implement multiple log sources pulling from the same SQS queue. This works great.

Hi,

thanks to everybody..so the better option will be to change the collection method from Prefix specific to SQS Queue..I will ask our customer to follow these guidelines and then verify if all the logs are ingested.