How to Set Up Enforcement Tracking for watsonx Orchestrate Agents in watsonx.governance
A step-by-step guide to connecting agent evaluation evidence in watsonx Orchestrate to your GRC system in watsonx.governance.
Agentic AI moves fast, and so does the risks it introduces. If your agents are reasoning across steps, calling tools, and talking to customers, saying “we defined a control” isn’t enough. You need proof that the control is actually being enforced, continuously, in production.
That’s what Enforcement Tracking with watsonx Orchestrate does: it automatically pulls evaluation metrics from your watsonx Orchestrate agents and records them as evidence inside watsonx.governance controls. No manual screenshotting, no spreadsheet exports, no quarterly evidence scramble.
Here’s how to set it up, using a Retail Banking Support agent as our example.
Step 1: Register the Connection (One-Time, Admin Only)
An administrator connects watsonx Orchestrate to watsonx.governance once. This establishes the authenticated link that powers evidence collection for every agent going forward — no per-agent setup required.
-
In watsonx.governance, navigate to Integrations, and select “Agent Platforms”
-
Select watsonx Orchestrate and authenticate the instance
-
Make sure to toggle the “Enable integrations.”
-
Confirm the connection status shows Active.
This step only needs to be done once. Multiple watsonx Orchestrate instances can be connected to watsonx.governance.
Step 2: Link Your Agent to a Use Case and Control
Now connect a specific agent to the governance obligations it needs to meet.
-
Open your Retail Banking Support use case in watsonx.governance.
-
Under the linked Agents in “Related assets” section, select "Import" to bring in your Orchestrate agent.
-
Select the “Retail Banking Support” agent from the dropdown list, and click “Save.”
This mapping is durable: once set, it persists as the agent moves from draft to production. Note: make sure that your use case has been promoted at least into the “Risk Assessment” stage, and has the “Use Case Library” parent business entity.
How evidence flows in depends on where the agent is in its lifecycle:
-
Live agents: watsonx.governance automatically pulls metrics on a scheduled basis. You can toggle the schedule to a daily, weekly, or monthly schedule, with the metrics being aggregated at that interval. These metrics will be available in the “Performance” section.
watsonx.governance is currently retrieving metrics from watsonx Orchestrate Agent Ops APIs, where observability metrics and LLMaaJ metrics like hallucination, toxicity, and helpfulness are available. As new metrics are added into watsonx Orchestrate, those metrics will be automatically retrieved into the system.
Step 4: Review Pass/Breach Status
Each collected metric can be checked against the thresholds defined in the retrieved metric object. You can also configure workflows to have your users be notified to set metric thresholds whenever a new metric is brought in to the system to make sure there is a threshold is set for each new metric of interest to you.
And that’s it! You’ve now configured ingestion for the Retail Banking Support Agent, and it’s now a living part of your GRC workflows.