IBM Security QRadar

 View Only
Expand all | Collapse all

LogDNA integration with Qradar

  • 1.  LogDNA integration with Qradar

    Posted Fri December 06, 2019 02:16 AM
    ​Hi All,

    We have received a requirement where we need to integrate "logDNA" with Qradar and this logdna resides in Azure cloud. So my question is How we start about this. What all things I would require for this integration, I am completely new for this kind of integration however I have some experience of integrating Bigfix, Qualys, Service now and few windows authentication servers with Qradar.

    Any answer would be of great help.

    Regards
    Asif Siddiqui



    ------------------------------
    asif siddiqui
    ------------------------------


  • 2.  RE: LogDNA integration with Qradar

    Posted Tue December 10, 2019 10:23 AM
    Hi @asif siddiqui,

    ​Quickly looked up Azure LogDNA. It implements machine learning to analyze and predict issues with server logs. The first you can do is challenge them on how will it provide any value(apart from generating offences to notify themselves) and the actual use case.

    But to integrate this with QRadar(with integrate, I guess getting logs from this Azure service into QRadar), you can use Azure event hubs which is supported by QRadar. You need to ask Azure or any team working with Azure to put the logs into an Azure event hub and then you can pull the logs to your QRadar instance. Another option is if you have log management system like Splunk or elastic, you can first pull into those systems via APIs and then forward to QRadar via syslog protocol.

    Let me know if this makes sense.

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 3.  RE: LogDNA integration with Qradar

    Posted Wed December 11, 2019 12:16 AM
    ​Hi Chinmay,

    Thanks a lot for this valuable information. Unfortunately we do not have Splunk or any other log management system implemented in our environment. All log sources like Bigfix, Qualys , service now, I have used apps provided by IBM exchange.
    The first option seems feasible , I will check with the concern team about Azure event hub, I have a little silly question here,
    When I click on add a log source, why don't I see that in the protocol list in admin tab while If Qradar supports Azure event hub.

    Regards
    Asif Siddiqui

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 4.  RE: LogDNA integration with Qradar

    Posted Wed December 11, 2019 03:35 AM
    What QRadar version are you using if I may ask? If I am right, it is available since QRadar 7.3.1 or something need to check.

    You can go to add a log source and try log source type to be Microsoft Azure or Microsoft Azure platform. Then you can select Microsoft Azure event hubs in the protocol configuration.

    If you are on or after this version and still cannot see the protocol, see if you get automatic DSM updates. If not, install it manually. Refer to manually installing RPMs

    I would like to describe Azure event hub a bit. It is just a place to store your account. When you have your logs configured to be in the Azure event hub, configure a log source with that event hub and appropriate credentials. QRadar will then pull the logs from that event hub and give you as events. Beware of any firewall changes you need to open communication between QRadar and event hub(Event hub and storage account need different TCP ports to be open). Also make sure that in this case i.e. pulling the logs, the target event collector option in the log source configuration will list the event collector which will actually pull the logs.

    Let me know if you have more questions.

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 5.  RE: LogDNA integration with Qradar

    Posted Fri December 13, 2019 12:54 AM
    Hi Chinmay,

    Thanks for this valuable information​ in simple language. Now I have something to discuss with clients about the requirements for this integration.
    We are currently running QROC (Qradar on cloud) with version v7.3.2.
    And yes I do see the "Microsoft Azure event hubs " under protocol configuration when I select "Microsoft Azure platform"
    So, manually installing RPMs is not required I guess.
    I will check with client which option they have in place (I mean 1) log management system or 2) Azure event hub)
    Based on their reply, I will proceed further.
    I am going through documentation regarding azure integration with Qradar.

    Kind Regards
    Asif Siddiqui

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 6.  RE: LogDNA integration with Qradar

    Posted Fri December 13, 2019 03:34 AM
    @asif siddiqui

    Perfect. Hope it helps.
    Good luck man.

    Needless to say, if you need any help, just let me know :)

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 7.  RE: LogDNA integration with Qradar

    Posted Fri December 20, 2019 06:29 AM
    Hi Chinamy,

    We are slowly proceeding towards our integration of logdna (which resides in azure cloud) and Qradar and we have selected the option of event hub (Its already created on azure portal in other side). I have asked them the details of the parameters we require  for this integration,​
    1. Event hub connection string: This includes a) Event hub name space name , b) Event hub name, c) SAS key name, d) SAS key.
    2. Consumer Group name : When an Azure Event Hub is created, a $Default consumer group is created. This allows an application using that consumer group to get started and begin reading events.
    3. Storage account connection string :This includes a) Storage account name, b) Storage account key.

    Now I need to understand the communication part. What all ports need to be open from both sides, I need little clarity on this,
    I might need to contact my network/communication team, I went through the document and it says below,

    "To collect events from Microsoft Azure Event Hubs, you need to create a Microsoft Azure Storage Account and an Event Hub entity under the Azure Event Hub Namespace. For every Namespace, port 5671 and port 5672 must be open.
    For every Storage Account, port 443 must be open.
    Note: The Microsoft Azure Event Hubs protocol can't connect by using a proxy server.

    So I can ask my network team to open ports 5671 and 5672 , do we have another requirements beside this for a successful communication.

    Regards
    Asif Siddiqui


    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 8.  RE: LogDNA integration with Qradar

    Posted Fri December 20, 2019 06:48 AM
    Hello Asif,

    ​That's all you should need to get the synchronous path in place.

    Sometimes perimeter security devices can be too clever and cause issues (i.e. the device is doing other traffic monitoring or shaping, not just permit/deny). So if you get stuck (as I did in the Summer), you can run up a lab environment that has direct access to test the end-to-end and monitor the permit/deny traffic at the perimeter to verify those ports into Azure.

    Just recognise that you may see latency as an Event Hub is "just a pipe" from the LogDNA source. There are issues with a lot of the Microsoft products that do not surface events in real-time (i.e. less than 15 minutes). If interested, can explain via another thread.

    Regards,

    Darren H.

    ------------------------------
    Darren H.
    ------------------------------

    Original Message


  • 9.  RE: LogDNA integration with Qradar

    Posted Mon December 23, 2019 06:03 AM
    ​Thanks Darren H.


    Regards
    Asif Siddiqui

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 10.  RE: LogDNA integration with Qradar

    Posted Fri December 20, 2019 12:27 PM
    Hi @asif siddiqui,

    There are some points to take into account and you're done:

    1) Choose an evet collector or processor which will connect to Azure. In my case, I have an event processor, as a virtual machine, in Azure itself so it is easy. You will need to configure this azure collector/processor as your target collector in the log source configuration.

    2) The Azure log source requires 2 connections. One connection to Azure Event Hub, and one to Azure Storage. Both the Event Hub and the Storage entity needs to be created under the same Event Hub namespace. This depends on your thinking of QRadar design.

    Putting event processor on Azure doesn't send all the events to console hence saving bandwidth. if events are not too many, you can even put an event collector in Azure bit totally depends on the scenario.

    Azure Event Hub connects to [Namespace Name].e-servicebus.windows.net over port 5671 and 5672, where the Storage Account connects to [Storage_Account_Name].blob.core.windows.net over port 443. 
    So connection to event hub on ports 5671 and 5672 and connection to storage account on port 443.

    Let me know if you have any questions.    ​

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 11.  RE: LogDNA integration with Qradar

    Posted Mon December 23, 2019 06:02 AM
    Hi Chinmay,


    ​1) yes We have 3 event processors or event collectors in our company environment and all 3 are on VM palotform.(We call          
        these 3 as data gateways)

    Question: So we can select any one out of these 3 event collectors right ?
    Question : What you mean by event collector in azure itself? do we need to setup another event collector  in Azure side as well?


    2) Yes, Both the Azure event hub and Azure storage are created under the same hub namespace.


    Could you please elaborate below sentence, I need to understand this so I can approach them with  clarity.

    "Putting event processor on Azure doesn't send all the events to console hence saving bandwidth. if events are not too many, you can even put an event collector in Azure bit totally depends on the scenario."



    Regards
    Asif Siddiqui


    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 12.  RE: LogDNA integration with Qradar

    Posted Thu January 23, 2020 01:32 AM
    ​Hi Chinmay,

    I received the required parameters (event hub connection string, consumer group name and storage connection string )to add the azure log source into Qradar and when I deployed changes I can see the "success" status against azure log sources  in log source list.(Success means connection is established and logs are coming in right?)
    However when I searched for events in log activity tab it shows no results returned , any idea why is it so?


    Regards
    Asif Siddiqui

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 13.  RE: LogDNA integration with Qradar
    Best Answer

    Posted Thu January 23, 2020 03:38 AM
    Hi @asif siddiqui,

    Apologies. Missed your last message.

    ​Question: So we can select any one out of these 3 event collectors right ?
    Answer : Yes. You can select any of the event collector out of those as far as that collector has access to specified Azure services.

    Question : What you mean by event collector in azure itself? do we need to setup another event collector  in Azure side as well?
    Answer: With this I mean that if you put an event processor in Azure, the event processor there will collect the data locally and then can save the logs itself. See the diagram for clarity.

    As per the information, you received all the details about the connection but you cannot see the details.
    1) Normally takes some time to retrieve events from event hub.
    2) If not, see in the log source configuration page and confirm that the target event collector is the one you chose to connect to the Azure services.
    3) If correct, even though there is succcess on the log source, there might be some issues in the log retrieval from Azure through the event hub protocol. Check qradar.log and qradar.error for any Microsoft Azure event hub logs/errors
    cat /var/log/qradar.log | grep -i Azure
    cat /var/log/qradar.error| grep -i Azure
    Also see if the traffic analysis for your deployment is on. Microsoft Azure creates logs automatically.
    3) Check the connection to the Azure services itself as per the below details:

    From the event collector you chose to get logs, run the below checks on the box:
    telnet [Storage_Account_Name].blob.core.windows.net 443
    telnet [Namespace Name].e-servicebus.windows.net 5671
    telnet [Namespace Name].e-servicebus.windows.net 5672

    See if the connection succeedes. If not, you still need to open some firewall ports somewhere.
    But hope this fixes the issue or at least gives a bit more information on the issue. :)


    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 14.  RE: LogDNA integration with Qradar

    Posted Mon January 27, 2020 01:32 AM
    Hi Chinamay

    Thanks for those troubleshooting steps, I am checking with QROC team nd azure team. However Just to provide more information, I checked and observed that Qradar sent few system notification regarding azure log source. The notification is as below,

    "A protocol source configuration may be stopping events from being collected."
    I have


    any idea about this notification.


    Regards
    Asif Siddiqui.




    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 15.  RE: LogDNA integration with Qradar

    Posted Fri January 31, 2020 12:03 AM
    ​Hi Chinmay,

    Qradar admin team ran  few connection commands  on their end  and it seems Event hub connection string is great and connecting properly, however they  are gathering errors for the storage account connection string.

    Ensure that the Storage Account Connection String is valid and that QRadar is able to connect to [t=https://xxxxxxxxxxxx.blob.core.windows.net/.blob.core.windows.net]

    Can you please confirm with azure that you have the right storage account connection string within Qradar and if it seems like it is correct then please gather a new storage account connection string?


    I need your input on this to understand the issue.



    Regards
    Asif Siddiqui




    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 16.  RE: LogDNA integration with Qradar

    Posted Fri January 31, 2020 03:22 AM
    Hi @asif siddiqui,

    It is good that you have progress.

    Now for "Ensure that the Storage Account Connection String is valid and that QRadar is able to connect to [t=https://xxxxxxxxxxxx.blob.core.windows.net/.blob.core.windows.net]", there might be two things.

    1) QRadar cannot connect to the storage account 
    Do "telnet [Your_Storage_Account_Name].blob.core.windows.net 443". This should give you connected or a telnet terminal. If it doesn't, the port on the firewall needs to be open by network team.

    2) The connection string is invalid. This is where you depend on your Azure team to give you the correct string. Maybe sit with them to see if they are giving you the string from the correct storage account.
    As an alternative, if you do not trust the connection string, you can also uncheck the "Use Storage account connection string" aand use the Storage account name and storage account key(which you still have to get from Azure team.) ​

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message


  • 17.  RE: LogDNA integration with Qradar

    Posted Mon February 03, 2020 10:41 PM
    Hi Chinmay,


    Understood. Will check both the options. Will keep you posted till success.

    Thanks Again.


    ​

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 18.  RE: LogDNA integration with Qradar

    Posted Wed February 12, 2020 12:33 AM
    ​Hi Chinmay,

    Finally we have successfully integrated Microsoft azure with Qradar. Few things I observed,
    1) First we received "A protocol source configuration may be stopping events from being collected"
        Here we replaced the log source type from "MS Azure platform" to "MS Azure Active directory"
    2) Protocol configuration was same ""MS Azure Event Hubs"

    Now we are receiving logs as below,(Sample logs I have exported from Qradar, Also I have replaced IP address with x.x.x.x)

    Event Name Log Soure Event Count Time Low Level Category Source IP Source Port Destination IP Destination Port Username Magnitude
    Microsoft Azure Active Directory Message Azure_AD 1 Feb 12, 2020, 3:02:38 AM Stored x.x.x.x 0 x.x.x.x 0 N/A 3
    Microsoft Azure Active Directory Message Azure_AD 1 Feb 12, 2020, 3:02:38 AM Stored x.x.x.x 0 x.x.x.x 0 N/A 3
    Microsoft Azure Active Directory Message Azure_AD 1 Feb 12, 2020, 3:02:36 AM Stored x.x.x.x 0 x.x.x.x 0 N/A 3
    Microsoft Azure Active Directory Message Azure_AD 1 Feb 12, 2020, 3:02:36 AM Stored x.x.x.x 0 x.x.x.x 0 N/A 3
    Microsoft Azure Active Directory Message Azure_AD 1 Feb 12, 2020, 3:02:36 AM Stored x.x.x.x 0 x.x.x.x 0 N/A 3
     
    Now I need to understand,
    1) Why source and destination IP address is same in this log.
    2) In Payload we can see the information but why its not in log (column itself)
    3) How start about rule creation so that offenses  will get generated.

    Note: Information and simple language helped me understand this integration easily and I was leading the call for this troubleshooting with IBM team and azure team. Would like to extent my gratitude and say thank you.





    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 19.  RE: LogDNA integration with Qradar
    Best Answer

    Posted Thu February 13, 2020 03:11 AM
    Generally, when you see low level category "Stored" it means that the logs are not recognized; that is also why the IP is of the device sending logs. You can check through DSM editor if the Event ID (and any other) field is parsed. If all the events are such, it is probably that the type that you set is wrong. In some cases it may occur when e.g. vendor introduced new events but the DSM is still not updated to recognize them.

    ------------------------------
    Dusan VIDOVIC
    ------------------------------

    Original Message


  • 20.  RE: LogDNA integration with Qradar

    Posted Mon February 17, 2020 04:41 AM
    Hi,

    All the events has low level event category as "Stored" which means Qradar  is receiving events from azure however its unable to parse those events so what are the solutions to the issue,
    Shall we open ticket to IBM for support or are there any steps we can perform manually before approaching IBM.

    Please guide

    ​

    ------------------------------
    asif siddiqui
    ------------------------------

    Original Message


  • 21.  RE: LogDNA integration with Qradar

    Posted Mon February 17, 2020 04:53 AM
    Hi @asif siddiqui,

    Apologies for overlooking this thread.
    I am guessing these logs are NOT active directory logs but generic Azure platform logs.
    To start with, it is weird and interesting to see that Azure active directory as the log sources type instead of Azure platform for the following reasons

    1) A system administrator can be easily confused if he/she sees the log source type of Azure active Directory
    2) Maybe that is the reason you are seeing Stored events in the catergory as these are not Active Directory logs but the DSM expects Active Directory logs and do not know what to do with those logs.

    Potential Solution:
    1) Change the log source type back to either Micorost Azure event hub or Microsoft Azure platform to let the system detect and use the correct DSM.
    2) If you want to use Acive Directory as the log source type, create your own custom DSM using the DSM editor and map the event(s) to correct QIDs

    ------------------------------
    Chinmay Kulkarni
    ------------------------------

    Original Message