Instana U

 View Only
Expand all | Collapse all

Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

  • 1.  Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

    Posted Tue September 26, 2023 02:38 AM
    Edited by Karthick Subramaniam Wed September 27, 2023 04:55 AM

    Hello Instana Team,

    We have configured Smart Alerts for Erroneous Calls , HTTP Status Code 5XX and this works nicely and very useful as well. When there are multiple issues in the environment our internal devops receive multiple notifications (one notification for each issue), which works as expected. 

    We are also planning to alert customers through email / slack / in future integrating with salesforce for opening a case. And these alerts to customer team are only for critical incidents like synthetic check failures or a very high number of error calls. In this scenario we don't want to send multiple notifications to the customers when the issues in the environment still persists. would like to embed some logic on notifying external customers so that number of message going out is controlled. can you please let me know if this is possible ?

    Regards,

    Karthick S



    ------------------------------
    Karthick Subramaniam
    ------------------------------



  • 2.  RE: Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

    Posted Wed September 27, 2023 05:44 PM

    Hi Karthick

    that is a tricky request as you want to reduce the "noise" without loosing precision (or timelines, or accuracy). 

    Technically, you have two avenues as I see it:

    a) One way is to route the events via AIOps Event Manager https://www.ibm.com/products/cloud-pak-for-aiops and let AI help you condensing events.

    or

    b) Besides creating a dedicated "customer" alert, your best bet is to use "Time Threshold" https://www.ibm.com/docs/en/instana-observability/current?topic=ma-smart-alerts#time-threshold to increase the time until a alert is triggered. But this will also reduce potential alerts which are not as severe as your original ones.

    Let me know if this makes sense and/or how you would like to decide which even is good and goes to the customer, and which one is to noisy.

    Cheers

    Tom



    ------------------------------
    Tom Tammann
    ------------------------------

    Original Message


  • 3.  RE: Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

    Posted Wed September 27, 2023 09:25 PM

    Hey Karthick,

    I agree with Tom's second point above:  you can add more than 1 Smart Alert for an entity (like an application perspective), which allows you to create one for your dev team which can be noisy, and another one (using a different alert channel target) for customers which utilize a time configuration which reduces the noise.



    ------------------------------
    Arthur De Magalhaes
    STSM - Instana and AIOps Architect
    IBM
    Markham ON
    ------------------------------

    Original Message


  • 4.  RE: Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

    Posted Thu September 28, 2023 10:12 AM
    Edited by Tom Tammann Thu September 28, 2023 10:15 AM

    One analogy I want to add which I think is very good and which is used often by our PM @Sivasundar Natarajan 
    If a fire breaks out in your house, when do you want to know about it? One spark in an electrical switch? the gas stove turns on or a candle burns? Fire place? How much fire do you tolerate until you trigger an alarm? Smoke is a good signal but then you also cook (the delicious lamb racks my wife does under the broiler produce a lot of smoke ;-) ). Obviously, you dont want to wait until the house is engulfed in flames, this signal is easier to spot ...
    Saying, the signals that something is wrong are not as obvious as saying "Send an alert if Prod is down or slow". Or only send for relevant events. For how long, how often, all of prod etc? 

    For your customers you might want to send an alert when at least one room is fully on fire??

    Last, ALWAYS, when you start configuring an alert, KNOW the underlying time series (or event in case of SLIs - SLO alerts are coming...). Visualize the time series, look it up and study its historical behavior to get a sense, if it describes a spark, a little smoke or a full fire...


    This is not at all an Instana issue. Its a general observability issue and why we all have a job ;-) 



    ------------------------------
    Tom Tammann
    ------------------------------

    Original Message


  • 5.  RE: Smart Alert Notifications from Instana - any ways to limit the number of notifications ?

    Posted Mon October 02, 2023 02:53 AM
    Edited by Karthick Subramaniam Mon October 02, 2023 02:58 AM

    Thank you Tom and Arthur for your valuable feedback and inputs. @Tom Tammann very good analogy which i can relate to it. and as you said this is a general observability issue.

    For now, we can consider to use one configuration for our internal devops + another config with a different time configuration to alert customers & other stake holders to reduce noise. also as you said need to look at the past behavior to understand more. In future when we have plan to integrate Instana with Salesforce for case creation (esp P1S1) might have to do more analysis + a PoC for this integration.

    Thank you for your responses.



    ------------------------------
    Karthick Subramaniam
    ------------------------------

    Original Message