Watson Assistant

Expand all | Collapse all

Adjusting confidence levels in Watson Assistant?

Jump to Best Answer
  • 1.  Adjusting confidence levels in Watson Assistant?

    Posted Mon October 12, 2020 12:35 PM
    Posting this for a "friend"......

    Is there a methodology to instruct Watson to recognize specific user intents only once they reached a certain confidence level?

    ------------------------------
    Daniel Toczala
    Community Leader and Customer Success Manager - Watson
    dtoczala@us.ibm.com
    ------------------------------


  • 2.  RE: Adjusting confidence levels in Watson Assistant?
    Best Answer

    Posted Mon October 12, 2020 12:39 PM
    Okay, found a nice blog post that shows you how to handle multiple intents in a single utterance.  It is in Dealing With Compound User Intents in IBM Watson Assistant, and the article shows how you can set your own confidence level, and then check to see if your particular intents have reached that confidence level.  Nice article - quick read.

    ------------------------------
    Daniel Toczala
    Community Leader and Customer Success Manager - Watson
    dtoczala@us.ibm.com
    ------------------------------



  • 3.  RE: Adjusting confidence levels in Watson Assistant?

    Posted Wed October 14, 2020 07:56 AM
    I have built some tests using the multi-intent handling examples and they really are a bit of a pain in the neck to maintain with jumps here and there throughout as you try to juggle the intents and handle each off the stack.  
    I'd love to see WA have multi-intent handling built into the product in a more elegant manner.  As mentioned in the original article, users frequently don't realise that they are only supposed to say one thing at a time and therefore we struggle to handle longer dialog entries.  I'm sure it's common enough that many chatbot builders would appreciate a better way to handle them.  I'm pretty sure most times only the top intent is handled at the moment due to the complexity of trying to juggle multiple intents.

    ------------------------------
    Graham Walker
    ------------------------------



  • 4.  RE: Adjusting confidence levels in Watson Assistant?

    Posted Wed October 14, 2020 10:45 AM

    Hi Graham, I agree with you that its a pain. However, its something that imo is very difficult to accurately identify in general, and even more difficult to do globally from the service POV. meaning... is the user actually asking for two things? or are they just saying one thing that is confused between two.
    I have always historically relied on disambiguation to have them choose the one thing they mean, and then they naturally ask for the second thing they wanted (if there were truly two) when they finish the first, and there really haven't been any major negative impacts from this method. I get its not ideal, but there's a balance between simplicity of the dialog design vs. end user experience.



    ------------------------------
    Mitch Mason
    ------------------------------



  • 5.  RE: Adjusting confidence levels in Watson Assistant?

    Posted Thu October 15, 2020 08:18 PM
    I agree that disambiguation and relying on the user to ask the next question is one way to handle this.  in regard to handling the multiple intents in the product, I was sort of thinking of a way to have a system variable that holds the intent array for a longer until reset.  I'm just looking at my current code (based on the examples I have seen) which has to do the following using dialog nodes and "jump to's" which is a really awkward way to do this:
    1. determine that a new utterance was received
    1.1. if yes, create a variable with an intent array for intents > my nominated threshold
    1.2. if no, ensure that only the anything else node at the end of the dialog tree triggers
    2. check if the intent array variable has anything in it (ie. there were intents above the threshold) 
    2.1. if there are no intents jump to the anything else node at the end of the dialog tree
    2.2. if there are intents jump to the iterate intents node
    3. if there are intents in the variable pull the first out of the array into another variable used to store the current intent
    5. remove the intent from the original intent array variable
    6. jump to a "dummy" dialog node to start evaluating nodes agains the currentIntent
    All nodes below this need to evaluate the $currentIntent variable rather than the normal #intent.  Oh, and the above has to handle associated entities too.

    I don't mind using the $variable instead of the #intent in my dialog node evaluations, but the ability to set up the intent array and pop intents off the array without having to jump all over the place using dialog nodes would be really handy.

    I'm curious if anyone else has an alternative method for actively handling multiple intents?​

    Cheers, Graham

    Multi-Intent Handling


    ------------------------------
    Graham Walker
    ------------------------------



  • 6.  RE: Adjusting confidence levels in Watson Assistant?

    Posted Tue October 13, 2020 02:08 PM

    Here's another good blog I wrote about identifying the optimal threshold, it should also explain how to set the threshold for a given dialog node:

    https://medium.com/ibm-watson/how-to-set-the-optimal-confidence-threshold-for-your-assistant-bbb1be6f294f



    ------------------------------
    Mitch Mason
    ------------------------------