View Only
Expand all | Collapse all

Watsonx.AI deployments via LangChain

  • 1.  Watsonx.AI deployments via LangChain

    IBM Champion
    Posted Tue February 27, 2024 10:51 AM

    Hi all

    We are trying to use Watsonx.ai to deploy foundation models to be consumed in LangChain. However, while the The IBM WML library has a built-in conversion of WML models to LangChain models, it seems that this is no longer compatible the current version of LC. It simply doesn't work, particularly since we are calling a deployment instance. We are writing a lot of boiler plate code to get around this, and we can make it worth in a hacky way. Can someone here shed some light on this issue?

    For reference, this is our approach. Note that we have defined a prompt_variable {text} in the prompt we have promoted to a deployment space and is calling via the python SDK.

    from ibm_watson_machine_learning.foundation_models import ModelInference

    WatsonXAIEnvDep = OurConfigStructure(...)

    model = ModelInference(
            "apikey": WatsonXAIEnvDep.WXAI_IAM_API_KEY,
            "url": WatsonXAIEnvDep.WXAI_INSTANCE_URL,
    llm = model.to_langchain()  # trying to use the cast to LangChain's WatsonxLLM
    Then we run
    llm.predict(text="Hello, how are you?")  # <- This will fail 
    No "prompt_variables" provided.
    Reason: Prompt template contains input variables but `prompt_variables` parameter not provided in `params`.

    There's no way to pass prompt variables like this into the LLM, which is required to call the API. Similarly, calling the following fails

    llm("hello")  # <- This will fail

    or, if we ignore the langchain model object and go back to calling the ModelInference model object directly:

        prompt="hello", params={"prompt_variables": {"text": "some text here!"}}
    )  # <- This will fail too
    with output
    # output:
    # Failure during generate. (POST <DEPLOYMENT_URL>)
    # Status code: 400, body: {"errors":[
    #      {"code":"json_validation_error",
    #       "message":"Json document validation error: 'input' cannot be specified in the request body"}
    #     ],
    #     "trace":"1f319a5c59a46a9c59668c03474cbb6e",
    #     "status_code":400
    # }
    Now, if we do the following it works:
            "prompt_variables": {
                "text": "<s>[INST] Jeg hedder Mikkel[/INST]</s>[INST]Hvad hedder du?[/INST]"  # mixtral format
    So, it is indeed possible to do, but from here we have to write our own LangChain integration. It would be great if the IBM Watson machine learning library was compatible with LangChain directly. 


    Nicolai Thomsen

  • 2.  RE: Watsonx.AI deployments via LangChain
    Best Answer

    Posted Wed February 28, 2024 01:48 PM

    Hi, Nicolai,

    Here is a sample notebook that shows you how to work with models in projects and deployed prompt templates with variables (last section of the notebook) using Langchain.

    Hope it helps!

    Catherine CAO