Every team in your company suddenly wants access to an LLM. Marketing wants to summarise campaign feedback, support wants to draft replies, and someone in finance has a spreadsheet idea you'd rather not know about. The usual result is a pile of API keys spread across config files and repositories.
I work on a feature in IBM DataPower Interact Gateway (IDIG) called LLMProvider that exists to fix exactly this. In this post I'll explain what it is, why you'd use it, and then walk through the full flow with watsonx.ai, from registering a provider to watching token usage show up in Analytics. To explore these features, let me introduce you to the Summit Gear, an online retailer.
Meet Summit Gear
Summit Gear specialises in outdoor equipment. It finds itself in a situation that I'm sure sounds familiar to you. The social media team is buried in comments. Product reviews, complaints about late deliveries, praise for the new trail-running line. They want to run every incoming comment through an LLM to classify its sentiment, so support can triage the angry customers first and marketing can spot trends early. The application itself is simple. All they need is a model, in this case IBM watsonx.ai, to do the classification. And this is the point where things usually go wrong in an enterprise.
The path of least resistance looks like this. A developer signs up for watsonx.ai, generates an API key, injects it into their code from yet another config file that needs to be managed, and starts calling the provider directly. Do that across ten teams and you end up with:
- API keys scattered everywhere, each one a credential to an external service with real billing attached
- No central view of consumption. Who is burning through tokens? Which team caused this month's invoice spike?
- Inconsistent security and governance. One team rate-limits itself sensibly, another doesn't. One team logs prompts, another sends customer data to a model with no oversight at all.
- No easy way to rotate, revoke or swap. If a key leaks, or the organisation decides to change models, someone gets to hunt through every team's codebase.
This fragmentation is the problem IDIG was built to address. The IBM documentation describes it well: without a unified interface to govern these interactions, organisations end up with fragmented AI environments where AI services are consumed inconsistently, with limited governance and visibility.
So in our demo, Summit Gear's platform team doesn't hand the social media team a raw watsonx.ai API key. Instead they register watsonx.ai as an LLM Provider in IDIG, publish a gateway-managed endpoint, and let the social media team subscribe to it with their own client credentials through the Developer Portal.
The app team gets an endpoint. The platform team keeps control. Nobody pastes an API key into a chat channel.
What is IBM DataPower Interact Gateway?
IDIG extends your existing IBM API Connect Cloud Service (SaaS) subscription with AI Management capabilities, and is also available as a completely standalone product. It governs, secures and monitors how applications and AI agents interact with AI models, APIs and enterprise services. It's built on the lightweight, cloud-native IBM DataPower Nano Gateway, so you get AI governance without standing up a whole separate AI infrastructure.
IDIG supports two main AI interaction patterns:
- MCP Gateway. Exposes your existing enterprise APIs as Model Context Protocol (MCP) tools that AI agents can discover and invoke. You can also register external, third-party MCP servers
- LLM Gateway. Provides governed access to external LLM provider APIs. Applications and agents call a managed endpoint on the gateway rather than calling the provider directly.
This post is about the second pattern, the LLMProvider capability, which is what Summit Gear needs.
There's also a Bring Your Own Coding Assistant (BYOCA) capability that lets you automate IDIG tasks, including registering LLM providers, through natural language prompts from an MCP client. I'll stick to the guided UI flow here, but it's good to know the Assistant route exists!
What exactly is an LLM Provider?
An LLM Provider is a registered, governed connection to an external LLM platform, managed centrally through IBM API Studio. Rather than letting applications reach the provider directly, all requests are routed through IDIG, which applies consistent security, usage controls and observability.
When an application calls an LLM provider API through IDIG, the gateway does four things:
- Authenticates securely with the LLM provider on the application's behalf. The provider credentials live with the gateway, not with the consuming team
- Applies governance policies before forwarding the request: rate limits, security checks, whatever the platform team has configured.
- Tracks token usage and logs activity for every call
- Returns the response through the same controlled path, so responses are just as observable as requests.
Out of the box you can register the following providers: watsonx.ai, Google Gemini, OpenAI, Azure OpenAI, and a generic OpenAI-compatible option for anything else that speaks the OpenAI API shape. More on those at the end of this post.
Lifecycle of an AI Asset
|
|
|
|
|
|
Register the provider, configure auth, apply policies, publish
|
|
|
|
Runtime policy enforcement on live traffic
|
IBM DataPower Nano Gateway
|
|
|
Teams find the asset, subscribe, and get credentials
|
|
|
|
Track usage, tokens, performance and cost
|
|
Why route LLM traffic through a gateway at all?
Sentiment analysis is just one example. The same pattern applies whenever a team needs model access, and it buys you a few things that are hard to get any other way.
Credential isolation. The consuming team never sees the watsonx.ai API key. They authenticate to the gateway with client credentials issued through a subscription. Keys can be rotated, revoked or moved into a vault without any consuming team touching their code.
Central governance. The platform team applies policies once (rate limiting, security enforcement, usage controls) and they apply to everyone consuming the endpoint. Governance stops being a per-team honour system.
Cost visibility. Every request through the gateway is logged with token consumption. When finance asks why the billing figures for a Provider are more than expected, Analytics has the answer, broken down by API and consumer.
Consistent onboarding. A new team wanting LLM access goes to the Developer Portal, finds the published LLM API, subscribes to a plan and gets credentials. It's the same self-service flow they already use for ordinary APIs, and approval workflows keep a human in the loop where you want one.
Provider flexibility. Consumers call a gateway-managed endpoint, so the platform team retains the freedom to change what sits behind it. Today it's watsonx.ai; the app's integration point is the gateway, not the provider.
Observability by default. Usage trends, performance, error rates and operational health are captured automatically, shown in dashboards, and can be offloaded to an external analytics stack if you already have one.
In the Summit Gear story, the sentiment feature ships quickly, the platform team sleeps at night, and when three more teams ask for LLM access next quarter the pattern is already there. Swap in your own organisation and use case; the flow is the same.
Summit Gear's Use Case: End to End
Step 1: Register the watsonx.ai provider in IBM API Studio
This part belongs to the platform team and happens in IBM API Studio. API Studio is project-based. MCP tools and LLM provider configurations are organised into projects, and projects integrate with version-controlled repositories or folders, so the configuration itself is managed like code. Summit Gear's platform team creates a project called summit-gear-ai to hold their AI assets.
Within the project, they open the AI view and use the Form View to register a new LLM provider, picking Watsonx.ai from the supported provider list. Registration means filling in:
- Connection details: the watsonx.ai endpoint the gateway will forward to
- Authentication: how the gateway itself authenticates to watsonx.ai. This is where the API key comes in, referenced from a Kubernetes secret either stored in an external Secret Manager when using IBM API Connect on SaaS or through a secret created by an administrator on IDIG standalone cluster
- Provider-specific details, such as the watsonx.ai project and the model configuration the endpoint will front.
The important mental shift here: this credential is configured once, by the platform team. No consuming team will ever handle it.
Step 2: Applying Gateway Policies for Governance and Security
Registering the provider generates a policy sequence with some required policies, and the Form View Canvas lets the platform team customise it. This is where "managed and governed" stops being a slogan and becomes actual configuration. You can apply any of the numerous different IBM DataPower Nano Gateway policies at this point.
For the sentiment use case, the platform team thinks about things like:
- Rate limiting. The social pipeline processes comments in batches, and a cap on requests per minute protects both the watsonx.ai spend and the gateway
- Security policies, so only authenticated, subscribed applications can invoke the endpoint
- Usage Controls, keeping the consumption within what the platform team has budgeted for.
Because policies are applied at the gateway, they're enforced on every request at runtime regardless of who's calling.
Step 3: Publish the LLMProvider
With the provider registered and policies in place, the platform team publishes the project (or just the selected AI assets). Publishing is the moment the configuration becomes a live, managed endpoint. Just select the catalog and gateway you want to use and you are ready to go:
- The IBM DataPower Interact Gateway now hosts the runtime endpoint and enforces the configured policies on traffic.
- The Developer Portal now lists the watsonx.ai LLM API as a discoverable, "subscribable" asset.
Step 4: Access the gateway-managed endpoint
Once published, the platform team can view the gateway endpoint for the asset: the URL through which watsonx.ai is now invoked via the Nano Gateway. This is what gets shared with consuming teams, through the Developer Portal rather than by email.
Step 5: Dev team consumes the API from the Developer Portal
Now the perspective switches to the consuming team. The social media engineers never touch API Studio and never see a watsonx.ai credential. Their whole journey happens in the Developer Portal, the self-service portal where published AI assets are discoverable and where consumers subscribe so that they can invoke them securely.
- Discover. They browse or search the portal and find the published watsonx.ai LLM API, with its documentation, plans and usage information
- Register an application. They create an application in the portal representing their service, say social-sentiment-service
- Subscribe. They subscribe the application to the LLM API's plan. By default, Client Credentials are used for authentication but this is up to the platform team on how they configure Authorization
- Receive client credentials. The subscription provisions credentials for the application. These uniquely identify their app, are scoped to their subscription, and can be revoked without affecting anyone else.
Step 6: Calling the endpoints
With credentials in hand, the team wires up the pipeline. Every incoming comment goes to the gateway-managed endpoint with a classification prompt. The request below is illustrative, so adapt the URL, headers and body to match your published endpoint and the watsonx.ai API shape you're fronting:
What's missing from that request is the point: there's no watsonx.ai credential anywhere in it. In simplified terms, the gateway validates the client credentials and subscription, enforces the rate limit and other policies, authenticates to watsonx.ai with the platform-held key, forwards the request, records the token usage and returns the response.
The NEGATIVE result lands in the triage queue, and a support agent reaches out about the delayed Ridgeline pack before the customer posts a second, angrier comment.
Step 7: Monitoring it all in Analytics
The last phase of the lifecycle is visibility. IDIG Analytics collects and processes event data from the gateway and gives both the platform team and stakeholders insight into LLM API activity. It's built on the OpenSearch distributed search and analytics engine, and it lets you:
- Monitor LLM API traffic and usage trends. How many comments is the sentiment service actually processing?
- Filter, sort and aggregate event data to answer specific questions
- Visualise activity through charts, tables and reports on the AI usage dashboard
- Analyse performance and operational health: latency, errors, throughput
- Track token consumption for governance and cost management. This is the number that turns into the watsonx.ai invoice.
For the platform team this closes the loop. They can see exactly which applications are consuming the watsonx.ai endpoint, at what volume and at what cost, and adjust policies accordingly.
Beyond watsonx.ai: Other Providers
Everything above used watsonx.ai, but the flow is identical for the other supported providers. You pick a different provider type in the Form View, supply that provider's endpoint and credentials, and the rest (policies, publish, portal, analytics) works the same way.
- OpenAI. Register your OpenAI account so teams can consume GPT models through a governed endpoint instead of sharing an OpenAI key around
- Azure OpenAI. Same idea for organisations consuming OpenAI models via their Azure subscription, with the gateway holding the Azure credentials
- Google Gemini. Governed access to Gemini models, again with the credentials staying on the platform side
- OpenAI-compatible. This is the flexible one. A huge slice of the LLM ecosystem exposes an OpenAI-style API, so this option covers providers and runtimes that aren't on the named list. If it accepts an OpenAI-shaped chat completions request, you can register it and put the same governance in front of it to access platforms such as MistralAI or Groq.
That last option matters more than it might seem. A lot of enterprises end up with a mix: watsonx.ai for some workloads, Google Gemini if you need to batch embed content and so on. With IDIG all of them sit behind the same kind of managed endpoint, with the same subscription model, the same policy enforcement and the same analytics. Your consuming teams don't need to care which is which.
Wrapping Up
The sentiment feature was never the hard part. A single prompt and a foundation model handle that in an afternoon. The hard part in an enterprise is doing it in a way that's secure, governed, observable and repeatable, and that's what the LLM Gateway pattern in IDIG gives you: the platform team registers and publishes once in API Studio, the IBM DataPower Interact Gateway enforces at runtime, the Developer Portal handles consumption, and Analytics keeps everyone honest.
Summit Gear may be fictional, but the payoff is real and transfers directly: the app team ships its feature, and the next team that asks for LLM access gets pointed at the Developer Portal instead of a credentials vault.
Further Reading