With the watsonx.ai model gateway, you can enable scalable and secure deployment of generative AI capabilities. The model gateway is a unified interface that you can use to interact with foundation models from various large language model (LLM) providers. The model gateway is available on IBM watsonx.ai as a Service.
Benefits
With model gateway, you can:
-
Access multiple third-party providers such as OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Cerebras, and NVIDIA NIM.
-
Enable faster development by removing the need to manage multiple model endpoints.
-
Experiment freely with different providers and models.
-
Build and deploy AI agents, RAG patterns, and more.
Key features
The model gateway offers the following capabilities:
-
Routes inference requests across multiple LLM providers
-
Secures your credentials and sensitive configuration data with IBM Cloud Secrets Manager.
-
Integrates access control by using IBM Cloud Identity and Access Management (IAM).
-
Distributes workloads across models with load balancing for optimized performance.
-
Configures custom endpoints so you can securely integrate custom models tailored to your needs.
Setting up the model gateway
To set up the model gateway on IBM Cloud as an Admin:
-
Create a Secrets Manager instance to securely store API keys, credentials, and LLM provider configuration details.
-
Assign access permissions to users for the Secrets Manager service instance and watsonx.ai runtime instance through IBM Cloud Identity and Access Management (IAM).
-
Configure LLM providers through the model gateway by using the secret reference.
-
Enable models with the associated LLM providers.
After the model gateway is set up, you can send inference requests to the models.
Try out the model gateway in preview now. For more information and setup details, see the model gateway documentation.
#watsonx.ai