QRadar XDR

Expand all | Collapse all

QRadar in Azure collecting Events through Azure Event Hub - FAILED

  • 1.  QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Fri March 12, 2021 08:17 AM
    Hi community,

    I went through this forum and checked older posts related to the QRadar-Azure integration and could not find an answer.

    My Setup:

    QRadar in Azure in Tenant #1
    Sign-In and Audit Logs from Tentant #1 are sent to an Event Hub in the same tentant.

    My Problem:
    Mar 12 12:50:40 ::ffff:IP [ecs-ec-ingress.ecs-ec-ingress] [Thread-6976] java.lang.NoSuchMethodError: com/q1labs/semsources/sources/utils/GatewayLogSourcePatternParser.setupSourceNameModifier(Ljava/util/List;)Ljava/util/List; (loaded from file:/opt/ibm/si/services/ecs-ec-ingress/eventgnosis/lib/q1labs/q1labs_semsources_protocol_common.jar by com.q1labs.core.util.classloader.ChildFirstClassLoader$ChildURLClassLoader@6035a74b) called from class com.q1labs.semsources.sources.microsoftazureeventhubs.MicrosoftAzureEventHubsProvider (loaded from file:/opt/ibm/si/services/ecs-ec-ingress/eventgnosis/lib/q1labs/q1labs_semsources_protocol_microsoftazureeventhubs.jar by com.q1labs.core.util.classloader.ChildFirstClassLoader$ChildURLClassLoader@6035a74b).


    Mar 12 12:17:51 ::ffff:IP [ecs-ec-ingress.ecs-ec-ingress] [Thread-4546] com.q1labs.semsources.sources.microsoftazureeventhubs.MicrosoftAzureEventHubsSource: [ERROR] [NOT:0070003100][IP- -] [-/- -]There appears to be a configuration issue with the provider connection 'class com.q1labs.semsources.sources.microsoftazureeventhubs.MicrosoftAzureEventHubsProvider7'.


    There are some similar threads here without solution:

    https://www.ibm.com/mysupport/s/question/0D50z00006PFbmfCAD/errors-connecting-to-azure-event-hub?language=de

    https://www.ibm.com/mysupport/s/question/0D50z00006PEGdKCAX/errors-connecting-to-azure-event-hub-protocol-error?language=de


    What I've done so far?

    - I've followed this guideline https://www.ibm.com/support/knowledgecenter/SS42VS_DSM/com.ibm.dsm.doc/c_logsource_Microsoft_Azure_Event_Hubs_protocol.html

    and repeated this step many times. (Created the log source manually and also looked for auto discovered log sourcesI also asked another colleague to do the same. We both had the same problem.

    . https://www.ibm.com/support/knowledgecenter/SS42VS_DSM/com.ibm.dsm.doc/c_dsm_guide_ms_azure_troubleshooting.html

    We also tried to do some troubleshooting, but couldn't find any solution.
    Telnet to Storage Account and Event Hub Namespace work. Ports are open.

    We also created Event Hubs Namespaces and Eventhubs and allowed mostly everything to be sure it was not any permission problem.
    The same with Storage Account.


    Have we slipped up somewhere?

    Thank you!

    Greetings,

    Bruno



    ------------------------------
    BrunoMarX
    ------------------------------


  • 2.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Mon March 15, 2021 09:52 AM
    Hi - We had a recent issue with the eventhub integration too, but all i needed to do was disable and reenable and it was back up. I think we had some similar issues. One trick with log sources is you need to wait at least one min between changes ie enable or disable.





  • 3.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Tue March 16, 2021 03:00 AM
    Hello Bruno,

    we have the same Issue since the last Auto Update "DSM-MicrosoftAzurePlatform-7.4-20210205160601.noarch.rpm"
    Installed on "Mar 7, 2021, 2:15:37 AM".

    If anyone have a solutions. please let us know.


    ------------------------------
    Steven Beck
    ------------------------------



  • 4.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Thu May 20, 2021 10:50 AM
    HI Steven,

    To resolve this issue you should try to deploy full configuration after that may be this issue will resolve .

    ------------------------------
    Daniyal Abdul Razzak
    ------------------------------



  • 5.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Tue March 16, 2021 10:24 AM
    Hi Bruno,

    Judging from that error, there's nothing wrong with your Azure setup. That's a classloading problem, which means one of the needed jar files is either missing completely or is out of date and missing an expected method. Basically there's missing code. You can ignore the second error, I know it says there was a configuration problem but that's just a catch-all for any exception being thrown, it's almost certainly a side effect of the first error.

    You may be able to fix this by doing a full deploy from the Admin tab but if that doesn't solve it you should put in a support case.

    Cheers
    Colin

    ------------------------------
    COLIN HAY
    IBM Security
    ------------------------------



  • 6.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Tue March 30, 2021 09:09 PM
    Thank you @COLIN HAY and others.​​

    The integration just doesn't work.
    I've tried many times to set it up.(new Log Source, Full Deploy, Enable, Disable...) --> Same Problem.
    Followed every step and am sure made no mistakes. Using connection strings should make things easier and I am sure that I made no mistakes copying and pasting them.

    I tried it with Qradar in Azure(7.4.2) and two other Qradar Systems on-prem(7.4.1 and 7.4.2).
    Different Azure Protocol RPMs. PROTOCOL-MicrosoftAzureEventHubs-7.4-20200701234158.noarch and PROTOCOL-MicrosoftAzureEventHubs-7.4-20191218165336.noarch
    Same problem.

    Had another Qradar Colleague do the same idependently ---> Same problem.

    Rebuild my Azure environment 2 or 3 times and had two Azure Colleagues assure the configuration was fine. --> Same Problem

    I opened a ticket at IBM Support and after several messages, they told me to talk to azure and don't know what is going on.
    Azure told me that as long as I can access the resources through the right ports and use the correct connection strings, it should be fine.

    If I google the messages I get, I find other people facing the same problem but without any solution.

    I don't know what to do next.

    ------------------------------
    BrunoMarX
    ------------------------------



  • 7.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Wed March 31, 2021 11:27 AM
    Quick update:
     it worked with one on-prem system. with others same problem. used the same connection strings and verified networking to be sure traffic is not being blocked.
    the only difference is the qradar version running on those systems. IBM support is informed.

    ------------------------------
    BrunoMarX
    ------------------------------



  • 8.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Wed March 31, 2021 12:08 PM
    I think I mentioned it before but we are on 7.4.2 and previously 7.4.1 and do all Azure log collection via event hub and it seems to be solid except for one time where I had to disable re-enable.  Presumably all DSMs and Protocols updated?

    Thanks,

    Ian





  • 9.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Wed March 31, 2021 02:33 PM
    Hi Bruno,

    Are you still seeing this error:

    Mar 12 12:50:40 ::ffff:IP [ecs-ec-ingress.ecs-ec-ingress] [Thread-6976] java.lang.NoSuchMethodError: com/q1labs/semsources/sources/utils/GatewayLogSourcePatternParser.setupSourceNameModifier(Ljava/util/List;)Ljava/util/List; (loaded from file:/opt/ibm/si/services/ecs-ec-ingress/eventgnosis/lib/q1labs/q1labs_semsources_protocol_common.jar by com.q1labs.core.util.classloader.ChildFirstClassLoader$ChildURLClassLoader@6035a74b) called from class com.q1labs.semsources.sources.microsoftazureeventhubs.MicrosoftAzureEventHubsProvider (loaded from file:/opt/ibm/si/services/ecs-ec-ingress/eventgnosis/lib/q1labs/q1labs_semsources_protocol_microsoftazureeventhubs.jar by com.q1labs.core.util.classloader.ChildFirstClassLoader$ChildURLClassLoader@6035a74b).

    As I previously noted, this error means that code is outright missing on the QRadar side. There is no way this can possibly be resolved by configuration changes, either QRadar side (in the log source config) or Azure side - the code simply isn't present. So if you are indeed still hitting this error, I'd suggest reopening your support case or creating a new one. The support rep should recognize that this error is indicative of a QRadar-side problem but if they again tell you that it's an Azure issue, you can mention my name and ask the support person to contact me so I can assist them. I'm the Chief Software Architect for QRadar so if the rep does not know me, their team lead will. Either way I should be able to sort them out.

    Cheers
    Colin


    ------------------------------
    COLIN HAY
    IBM Security
    ------------------------------



  • 10.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Wed March 31, 2021 05:17 PM
    Hi @COLIN HAY and others,

    thank you!

    Well...As I stated above, it worked in my on-prem environment. I then tried to reproduce the same configuration on my Qradar in the Azure Cloud. Same error. I then decided to create the virtual machine from scratch again and use the same credentials as before. It worked!
    ​The VM is one Qradar from Azure Marketplace that I deployed and then patched it up to versin 7.4.2 FP2. I deployed the vm only for this purpose because I am migrating an on-prem system to a cloud system and wanted to test it before migrating. Therefore, there was no previous configuration since it was a fresh new QRadar System.

    But as I wrote yesterday still on my on-prem system it didn't work at first. The reason was something weird. Don't know whether it is a casual relation or correlation, but here https://www.ibm.com/support/knowledgecenter/SS42VS_DSM/com.ibm.dsm.doc/c_dsm_guide_ms_azure_troubleshooting.html

    there is this item:

    • Ensure that the port 443 is open to the storage account host. The storage account host is usually <Storage_Account_Name>.<something>, where <something> usually refers to the endpoint suffix.

    also here Microsoft shows that port 443 should be open:

    https://docs.microsoft.com/en-us/azure/event-hubs/troubleshooting-guide

    I then tried to see whether network was fine

    f you can also run the commands below and upload the output:

    I could get the certificate and also use telnet to see whether the port was open. But then I was checking on the networking options inside my Azure Storage account and saw that my qradar system in fact was not allowed to access the StorageAccount. I then added the QRadar IP to the list of allowed addresses and it worked, but it doesn't make sense because otherwise I would not have been able to use openssl to connect to the storage account through 443.

    to sum up, I think that
    - This networking settings prevented me from connecting my on prem system to EventHub
    - Some kind of error during the deployment of QRadar in Azure occurred that led to that class error. This is gone since I installed QRadar one more following the same steps as before, but this time no error showed up.

    Thank you!

    Regards,
    Bruno








    ------------------------------
    BrunoMarX
    ------------------------------



  • 11.  RE: QRadar in Azure collecting Events through Azure Event Hub - FAILED

    Posted Tue July 20, 2021 11:33 PM
    Hi Bruno,

    I faced the same issue but when i did deploy full configuration, issue has been resolved. Please check in your environment whether it solves the issue or not.

    Regards,
    Sujana Y

    ------------------------------
    Sujana Y
    ------------------------------