Deploying custom foundation models to watsonx.ai

Since its release, IBM watsonx.ai has been enabling businesses to train, validate, tune, and deploy AI models in a fraction of the time and with a fraction of the data. And starting with watsonx.ai 1.1.4 software release, you can do even more with the enterprise studio: upload and deploy your own custom foundation models.  

There are many reasons to import a custom foundation model, all driven by the unique needs of your organization. Ultimately, it boils down to a specific foundation model that is optimal for the task at hand but resides outside of watsonx.ai. For instance, you may need support for a language that is not currently available in the watsonx.ai foundation model library. Or, your organization may have invested resources to fine-tune a model to optimize it for your specific industry or business domain.  This “bring your own model” approach provides greater flexibility in how you select and utilize the right foundation model to meet your specific generative AI use case.  

In this article, we will describe how to install and deploy a custom foundation model in watsonx.ai (version 1.1.4 or later) using the example of a foundation model from Hugging Face. 

What kind of model can I deploy? 

• For example: 
  
  

  apiVersion: batch/v1
  kind: Job
  metadata:
    name: model-copy-job
    namespace: cpd-instance
  spec:
    template:
      spec:
        containers:
        - name: models-convertor
          image: cp.icr.io/cp/cpd/fmaas-runtime-wisdom-ansible@sha256:2cc673a6066cab686eed7b8e6998bc453a49be4ccb993674ab0ff81f099f807f
          env:
          - name: ${MODEL_PATH}
            value: "/model"
          - name: ${MODEL_NAME}
            value: "google/mt5-base"
          - name: TOKEN
            valueFrom:
              secretKeyRef:
                name: huggingface-tokken
                key: ${HF_TOKEN}
          command: ["/bin/sh", "-c"]
          args:
          - |
            huggingface-cli login --token ${HF_TOKEN}
            huggingface-cli download ${MODEL_NAME} --local-dir ${MODEL_PATH} --cache-dir ${MODEL_PATH}
            text-generation-server convert-to-safetensors ${MODEL_PATH}
            text-generation-server convert-to-fast-tokenizer ${MODEL_PATH}
          volumeMounts:
          - mountPath: /model
            name: byom-model
        restartPolicy: Never
        volumes:
        - name: byom-model
        persistentVolumeClaim:
          claimName: model-copy-pvc 

Step 3: Create the deployment for your custom model 

After the model is stored and registered, you can add the model asset to a space, and then create an online deployment.  

Note: The steps in this section describe how to create the deployment using API commands. To create the deployment from a deployment space, see Creating a deployment from space.  

To create a deployment programmatically, you must first get the model asset ID and then create the deployment.  

View available custom foundation models 

Begin by running a command to list the deployable foundation models you have in watsonx.ai. To view the list of available custom foundation models by using the watsonx API, run this code: 

curl --location 'https://<cluster_url>/ml/v4/custom_foundation_models' \
--header 'Authorization: Bearer $TOKEN'

Example output: 

{
    "first": {
        "href": "/ml/v4/custom_foundation_models?limit=100"
    },
    "limit": 100,
    "resources": [
        {
            "model_id": "example_model_13b",
            "parameters": [
                {
                    "default": "float16",
                    "display_name": "Data Type",
                    "name": "dtype",
                    "options": [
                        "float16",
                        "bfloat16"
                    ],
                    "type": "string"
                },
                {
                    "default": 256,
                    "display_name": "Max Batch Size",
                    "name": "max_batch_size",
                    "type": "number"
                },
                {
                    "default": 1024,
                    "display_name": "Max Concurrent Requests",
                    "name": "max_concurrent_requests",
                    "type": "number"
                },
                {
                    "default": 2048,
                    "display_name": "Max New Tokens",
                    "name": "max_new_tokens",
                    "type": "number"
                },
                {
                    "default": 2048,
                    "display_name": "Max Sequence Length",
                    "name": "max_sequence_length",
                    "type": "number"
                }
            ]
        },
        {
            "model_id": "example_model_70b",
            "parameters": [
                {
                    "default": "float16",
                    "display_name": "Data Type",
                    "name": "dtype",
                    "options": [
                        "float16",
                        "bfloat16"
                    ],
                    "type": "string"
                },
                {
                    "default": 256,
                    "display_name": "Max Batch Size",
                    "max": 512,
                    "min": 16,
                    "name": "max_batch_size",
                    "type": "number"
                },
                {
                    "default": 64,
                    "display_name": "Max Concurrent Requests",
                    "max": 128,
                    "min": 0,
                    "name": "max_concurrent_requests",
                    "type": "number"
                },
                {
                    "default": 2048,
                    "display_name": "Max New Tokens",
                    "max": 4096,
                    "min": 512,
                    "name": "max_new_tokens",
                    "type": "number"
                },
                {
                    "default": 2048,
                    "display_name": "Max Sequence Length",
                    "max": 8192,
                    "min": 256,
                    "name": "max_sequence_length",
                    "type": "number"
                }
            ],
            "tags": [
                "example_model",
                "70b"
            ]
        }
    ],
    "total_count": 2
}

Note: If you access the model list programmatically, you can access all the parameters that you can set for the selected model. For models deployed through the UI, the parameters are available at the online deployment creation phase. See the description of parameters for custom foundation models. 

The next step is to create the custom foundation model asset. You can create a model asset in two contexts: project context and space context.  

Note: The model type must be custom_foundation_model_1.0. The software specification name must be watsonx-cfm-caikit-1.0. You cannot customize the software specification. 

Create online deployment

When the custom foundation model asset has been created, you are ready to create the online deployment.  

For instance, here is an example code that shows a sample deployment with some of the parameters overridden. 

curl -X POST "https://<cluster_url>/ml/v4/deployments?version=2024-01-29" \
-H "Authorization: Bearer " \
-H "content-type: application/json" \
--data '{
  "asset":{
    "id":<your custom foundation model id>  // WML custom foundation model asset
  },
  "online":{
    "parameters":{
      "serving_name":"test_custom_fm",
      "foundation_model": {
           "max_sequence_length": 4096
      }
    }
  },
  "hardware_spec": {                        // Only one, of "id" or "name" must be set.
    "id": "<your custom hardware spec id>",
    "num_nodes": 1
  },
  "description": "Testing deployment using custom foundation model",
  "name":"custom_fm_deployment",
  "project_id":<your project id>  // Either "project_id" (or) "space_id". Only one is allowed
}'

View the status for a deployment 

You can view the status for an existing deployment by running this command: 

curl -X GET "https://<cluster_url>/ml/v4/deployments/<your deployment ID>?version=2024-01-29&project_id=<your project ID>" \
-H "Authorization: Bearer "

Note: The deployed_asset_type is returned as custom_foundation_model. 

Step 4: Run a prompt using your custom model  

Now that you have stored and deployed your custom foundation model, you can start using it. You can use the Prompt Lab to prompt the model and generate responses or create a prompt programmatically. 

To run a prompt with the custom model using the API, run this code:  

curl -X POST "https://<cluster_url>/ml/v1/deployments/<your deployment ID>/text/generation?version=2024-01-29" \
-H "Authorization: Bearer " \
-H "content-type: application/json" \
--data '{
 "input": "Hello, what is your name",
 "parameters": {
    "max_new_tokens": 200,
    "min_new_tokens": 20
 }
}'

 Congratulations, your model is now fully deployed and ready to use! 

Summary 

By deploying a custom foundation model to watsonx.ai, you are able to work with a model that best fits your project and business needs. A custom model can be any model that is built with an architecture supported by watsonx.ai, which greatly expands your options and flexibility in terms of the models that best fit your specific use case.   

In this article, we covered the steps necessary to install and deploy a custom foundation model, using the example of a foundation model from Hugging Face. To learn about other deployment options, see the Deploying foundation model documentation.