AIOps

 View Only
Expand all | Collapse all

Message Bus Probe - configure to parse different JSON data structures for multiple webhook endpoints

  • 1.  Message Bus Probe - configure to parse different JSON data structures for multiple webhook endpoints

    Posted Fri February 03, 2023 11:27 AM
    Michael Troitzsch asked this same question 4 years ago: https://www.ibm.com/mysupport/s/question/0D50z00006LMQ6lCAH/messagebus-probe-listen-to-multiple-webhooks?language=en_US . Here's the text of his question in case the link dies:

    We are setting up the message_bus Probe v8 to receive events from multiple senders (Test case: Grafana, Prometheus and simple events via curl).

All works fine for each webhook; but I'd like the probe to be able to accept multiple webhookURIs for the different senders. Reason: The json format is different, and I need to configure the json parser seperately. The latter is no problem - but I haven't been able to make the probe accept data from different URIs AND parse the correct json.

I've tried to set validateRequestURI=OFF, then the probe doens't check the webhook, but also doens't associated the json correcty.
Is there a way to define mutiple webhookUIRs ? Or do I really need to run multiple instances of the probe on different ports to accomplish my goal ?​

    And here is support's reply:

    14 Nov 2018 (4 years ago)
Hi Michael,

The MessageBus Probe Version 8 only support receiving events from a single endpoint or Webhook URI. This is because the Webhook Transport uses the WebhookURI set in the webhookTransport.properties file as the endpoint when it receives a HTTP notification and adds that to the event metadata for the JSON parser. Toggling the validateRequestURI=OFF turns off URI path validation which reduces security so that the probe still accepts HTTP notifications through the webhook address Host:Port even if the URI does not match, but the endpoint is still WebhookURI. Hence, the probe parser is limited to only parse the message from a single 'endpoint'.

You may raise a Request For Enhancement (RFE) through the RFE portal : https://www.ibm.com/developerworks/rfe/execute?use_case=submitRfe for the probe to support the use of the incoming HTTP notification URI as the endpoint so that you can configure several JSON parser configurations to parse different JSON event sources.

Hope this helps.

Thanks,

Azman​

    While the doumentation insinuates that it is now possible (see Specifying multiple parser configurations to parse different JSON structures ), the same limitation described above seems to still exist in the latest version of the probe, version 1.21, released on 12/14/2022 (the limitation is that one ONE webhook endpoint URI can be defined).

    Specifically, I am trying to configure two Webhook endpoints for the Message Bus Probe because I want to run only ONE probe, but receive different JSON payloads from two different systems. The problem I'm encountering is exactly as described in Michael's issue above: it ALWAYS believes that requests are coming into the endpoint that is specified in the Transport.properties file in the webhookURI variable. Here's the part of the log that shows what it's doing:

    1. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Received HTTP Request. Method: GET, URI: /foo, Content length: 0
    2. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Processing request...
    3. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Method: GET,Content readable bytes: 0
    4. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Extracting query.
    5. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Extracting query from URI.
    6. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Received message with length of 0 from endpoint /probe/webhook/prometheus:
    7. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Start parsing JSON message
    8. 2023-02-03T11:04:07: Debug: D-JPR-000-000: No exact parser for endpoint /probe/webhook/prometheus found. Parser configuration for source type "ANY" will be used.
    9. 2023-02-03T11:04:07: Debug: D-JPR-000-000: Done parsing JSON message


    In my configuration, I have webhookURI set to /probe/webhook/prometheus. Notice in the above on line 1 that the probe recognizes that this request (a simple curl command for testing) is accessing the URI "/foo". However, on line 6 it states that the message was received from "endpoint /probe/webhook/prometheus ", since that is the value specified for webhookURI in the Transport properties file. I specifically did not define a parser for that endpoint (as the log states on line 8) for testing purposes. If I do define a parser for that endpoint, then that's the ONLY parser that's ever used. I also set validateURI=OFF in my Transport properties file, which is how I am able to successfully curl to anything other than the URI defined as the webhookURI.

    So while the documentation specifically states that it has the ability to specify multiple aprsers for different JSON formats, that's simply not true, unless someone can prove me wrong.

    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------


  • 2.  RE: Message Bus Probe - configure to parse different JSON data structures for multiple webhook endpoints

    Posted Fri February 03, 2023 12:52 PM
    It DOES WORK! I found the latest Support's Guide to the Message Bus Probe document, which shows and example in section 4.7 for how to configure it. 

    I think the problem I had was that I wasn't setting webhookURI to "/". That one change seems to have made everything work correctly. Whew!

    ------------------------------
    Frank Tate
    Gulfsoft Consulting
    https://www.gulfsoft.com
    AIOps Experts. Contact us for implementation help.
    ------------------------------

    Original Message