Building Production Agent Runtimes with watsonx Orchestrate: Architecture, MCP, and Scale

From Prototype to Production

In Part 1, we built a working embedded agent with JWT authentication and context variables. Now we're going to transform that foundation into a production grade agent runtime that can:

• Route different users to specialized agents
• Dynamically discover and use tools via MCP
• Scale across multiple roles and workflows
• Handle complex multi-step processes
• Maintain consistency under load

This is where embedded chat becomes an agent runtime platform, not just a chatbot.

Who Should Read This

• Architects designing multi-agent systems
• Engineers scaling beyond proof-of-concept
• Technical leaders evaluating agent platforms
• Anyone who completed Part 1 and wants to go deeper
  
  

The GrandShield Challenge: Three Users, Three Runtimes

This article documents the real-world implementation of a fictitious GrandShield Insurance company, a production-ready insurance platform built with watsonx Orchestrate. GrandShield Insurance needed to serve three distinct user types, each with completely different needs:

1. Customers → GrandShield Agent

What they do: Get instant auto insurance quotes in under 60 seconds.

Tools: Get quotes, file claims, view policies, make payments.

The key: The agent already knows who you are, what policies you have, and what you're looking at, no repetition needed.

2. Underwriters → Risk Assessment Agent

What they do: Approve a policy without opening 3 different systems.

Tools: Risk assessment, quote approval, application review.

The key: The agent has access to underwriting tools and can execute risk calculations in real-time.

3. Claims Managers → Claims Processing Agent

What they do: Process claim end-to-end in one conversation.

Tools: Process claims, verify documents, calculate settlements.

The key: The agent can orchestrate the entire claims workflow, not just provide information.

The Architecture: Action Loop in Practice

The operational flow shows how agents execute, not just respond:

User → Agent → MCP Tool → Database/API → Agent → User

Here's the full architecture:

How to Read This Architecture

What's non-obvious here:

• JWT is not just auth → it's agent identity
  • The token doesn't just secure the connection, it tells the agent who it's acting on behalf of.
  • Without this, the agent can't safely execute actions.
• MCP is not just tools → it's runtime extensibility
  • Agents discover capabilities dynamically.
  • You can add new tools without redeploying agents.
  • This makes the system feel cutting-edge, not static.
• Agents are not assistants → they are workflow orchestrators
  • They don't just answer questions, they execute multi-step processes.
  • They coordinate between systems.
  • They maintain state across interactions.

Role-Based Agent Routing: Different Users, Different Runtimes

GrandShield isn't just one agent, it has three specialized agents, each with different tools and capabilities:

The pattern:

// Map user role to agent ID
const agentConfig = AGENT_CONFIG[user.role];

// Initialize with role-specific agent
const instance = window.WxOChat.createInstance({
  agentID: agentConfig.agentId,
  authToken: token,
  // ... other config
});

Same embedded chat component, different agent runtime based on authenticated role.

This is the power of the agent runtime: same platform and embedded chat technology, dynamically tailored to provide different capabilities for different users.

MCP: The Tool Layer That Changes Everything

Under the hood, GrandShield agents leverage MCP (Model Context Protocol) servers to dynamically access tools and documentation.

What This Means?

Instead of hardcoding every capability, agents can:

• Discover tools dynamically from MCP servers.
• Access documentation on-demand.
• Connect to enterprise systems through standardized interfaces.
• Extend capabilities without redeploying agents.

Example: The get_customer_by_email Tool

This MCP tool is the foundation of GrandShield's personalization:

What it does:

• Receives user_email from JWT context.
• Queries PostgreSQL database.
• Returns customer data (policies, claims, history)

How the agent uses it:

• Customer asks: "What's my deductible?"
• Agent receives user_email from JWT context.
• Agent calls get_customer_by_email(user_email)
• Tool returns customer data.
• Agent answers: "Your deductible for Policy #12345 is $500"

Impact: Extend agent capabilities seamlessly without changing agent configurations; new functionality is enabled through MCP tool deployments.

MCP Server Architecture

Production Patterns: What We Learned Building GrandShield

1. Tool Specialization Over Tool Sprawl

Problem: Agent with 50+ tools gets confused about which to use.

Solution: Create specialized agents per role. Customer agent has 5 tools, not 50.

Impact: Agent accuracy improved from 60% to 95% by reducing tool confusion.

2. Context Layers: Stable + Dynamic

Problem: Sending all context on every message is wasteful and slow.

Solution: Use JWT for stable context (set once), pre:send for dynamic context (per message).

Impact: Reduced token usage by 40%, improved response time by 200ms.

3. Graceful Degradation

Problem: Tool fails, agent says "I can't help with that" without explanation.

Solution: Return actionable error messages with fallback options (e.g., "Call us at 1-800-SHIELD").

Impact: User satisfaction increased 30% even when tools fail.

4. Token Refresh Without Interruption

Problem: Token expires mid-conversation, chat breaks, user loses context.

Solution: Use authTokenNeeded event to refresh tokens seamlessly in the background.

Impact: Zero user-visible token expiration errors in production.

Advanced Failure Modes

Beyond the basics from Part 1, here are production-level failure modes:

Production Best Practices

Security

• JWT authentication with RS256 (asymmetric encryption).
• 1-hour token expiration with automatic refresh.
• Private keys never exposed to client.
• HTTPS in production.
• Role-based access control.
• Audit logging for all agent actions.

Performance

• Asynchronous script loading.
• Lazy initialization (only when needed.)
• Proper cleanup on logout.
• Session level context for stable data.
• Message level context for dynamic data.
• Tool response caching where appropriate.

Reliability

• Comprehensive error handling.
•  Graceful degradation.
• Token refresh without interruption.
• Health checks for MCP servers.
• Fallback responses for tool failures.

Observability

• Structured logging for all interactions.
• Performance metrics (response time, tool latency).
• Error tracking and alerting.
• User analytics (conversation flows, drop-off points).
  
  

What Makes This Different

The result: Customers don't feel like they are talking to a bot. They feel like the application itself is intelligent and responsive.

Key Takeaways

• Agent Runtime > Chatbot: Embedded chat turns watsonx Orchestrate into an in-application agent runtime capable of executing workflows and taking actions.
• Specialization > Generalization: Multiple focused agents outperform one mega-agent.
• MCP = Dynamic Capabilities: MCP servers enable agents to discover tools dynamically and extend their capabilities without redeployment.
• Role-Based Routing: Different users need different agents with different tools, one platform, multiple specialized runtimes.
• Production is Different: Token refresh, error handling, graceful degradation, and observability are non-negotiable.
• Architecture Matters: Treat agents as first-class backend services, not frontend widgets.

Final Thought

The real design question is no longer "Where do we add a chatbot?"

In the next 2–3 years, users won't navigate applications, they'll express intent.

The only question is whether your product executes that intent… or makes users do the work.

Resources

• watsonx Orchestrate Documentation
• MCP Protocol
• Part 1: Getting Started - JWT, context, and basic setup

See also: Building Intelligence with Intelligence

Built with: IBM Bob, watsonx Orchestrate, Node.js, PostgreSQL, MCP