Hi all
We are trying to use Watsonx.ai to deploy foundation models to be consumed in LangChain. However, while the The IBM WML library has a built-in conversion of WML models to LangChain models, it seems that this is no longer compatible the current version of LC. It simply doesn't work, particularly since we are calling a deployment instance. We are writing a lot of boiler plate code to get around this, and we can make it worth in a hacky way. Can someone here shed some light on this issue?
For reference, this is our approach. Note that we have defined a prompt_variable {text} in the prompt we have promoted to a deployment space and is calling via the python SDK.
from ibm_watson_machine_learning.foundation_models import ModelInference
WatsonXAIEnvDep = OurConfigStructure(...)
model = ModelInference(
params=llm_parameters,
credentials={
"apikey": WatsonXAIEnvDep.WXAI_IAM_API_KEY,
"url": WatsonXAIEnvDep.WXAI_INSTANCE_URL,
},
space_id=WatsonXAIEnvDep.WXAI_SPACE_ID,
deployment_id=WatsonXAIEnvDep.WXAI_DEPLOYMENT_ID,
)
llm = model.to_langchain() # trying to use the cast to LangChain's WatsonxLLM
Then we run
llm.predict(text="Hello, how are you?") # <- This will fail
Output:
No "prompt_variables" provided.
Reason: Prompt template contains input variables but `prompt_variables` parameter not provided in `params`.
There's no way to pass prompt variables like this into the LLM, which is required to call the API. Similarly, calling the following fails
llm("hello") # <- This will fail
or, if we ignore the langchain model object and go back to calling the ModelInference model object directly:
model.generate(
prompt="hello", params={"prompt_variables": {"text": "some text here!"}}
) # <- This will fail too
with output
# output:
# Failure during generate. (POST <DEPLOYMENT_URL>)
# Status code: 400, body: {"errors":[
# {"code":"json_validation_error",
# "message":"Json document validation error: 'input' cannot be specified in the request body"}
# ],
# "trace":"1f319a5c59a46a9c59668c03474cbb6e",
# "status_code":400
# }
Now, if we do the following it works:
model.generate(
params={
"prompt_variables": {
"text": "<s>[INST] Jeg hedder Mikkel[/INST]</s>[INST]Hvad hedder du?[/INST]" # mixtral format
}
}
)
So, it is indeed possible to do, but from here we have to write our own LangChain integration. It would be great if the IBM Watson machine learning library was compatible with LangChain directly.
#watsonx.ai------------------------------
Nicolai Thomsen
------------------------------