Hi Ramona,
The average token usage is based mainly on the size of the results from your retrieval engine. Most retrieval systems have a max chunk size of about 2048 characters or 512 tokens. If you are retrieving 5 results from there and sending it to the LLM for generating a response, then you will use about 2,500 tokens per question/answer.
The Conversational Search feature within watsonx Assistant gathers 5 responses from the search connection, then sends it to the fine-tuned Granite LLM in this manner. If you are using custom extensions, you can adjust the above formula for your use case.
Thanks,
Jason
------------------------------
Jason Leiby
Principal WW Data & AI Technical Sales Specialist
jason.leiby@ibm.comDillon, CO USA
------------------------------
Original Message:
Sent: Tue July 23, 2024 06:33 PM
From: Ramona Ortega
Subject: Need guidance on token quota reached
Thanks for reaching out Nick- seems like we needed to upgrade to the paid plan. We are getting our assistants + watsonX ready for deployment. Do you know the average token usage for a Q&A conversion?
Thanks,
Ramomna
Original Message:
Sent: 7/23/2024 6:23:00 PM
From: NICK PLOWDEN
Subject: RE: Need guidance on token quota reached
Hi Ramona,
I wanted to confirm your usecase, are you attempting to integrate watsonx.ai with watsonx Assistant? I do not see the screenshot, could you re-attach it please.
How many days are left on your trial? From the message it seems that you have hit the token limit (25,000 tokens for the month) for the trial. Your token usage is the sum of your input and output tokens.
Thanks,
Nick
------------------------------
Nick Plowden
AI Community Engagement
IBM
Original Message:
Sent: Tue July 23, 2024 02:49 PM
From: Ramona Ortega
Subject: Need guidance on token quota reached
I am having the same issue and we are paying for Watson assistant instance. Any help appreciated.
------------------------------
Ramona Ortega
Original Message:
Sent: Mon July 22, 2024 03:35 PM
From: NICK PLOWDEN
Subject: Need guidance on token quota reached
Hello Samra,
I am reaching out to see if you were able to resolve the issue with your trial version of watsonx.ai. If not, I can follow-up on your behalf.
Thanks,
Nick
------------------------------
Nick Plowden
AI Community Engagement
IBM
Original Message:
Sent: Tue July 09, 2024 11:32 AM
From: Samra Darakshan
Subject: Need guidance on token quota reached
I have created a free trial account on watsonx.ai but after some time I got this error:
{
"errors": [
{
"code": "token_quota_reached",
"message": "Request of 1 token(s) from quota was rejected",
"more_info": "https://cloud.ibm.com/apidocs/watsonx-ai"
}
],
"trace": "549973ded9bb7da5c92f68a47dd66174",
"status_code": 403
}
Please find the screen shot below and provide some suggestion/guidance to resolve this issue.
![]()
#watsonx.ai
------------------------------
Samra Darakshan
------------------------------