z/OS Platform Evaluation and Test (zPET) runs customer-like workloads in a Parallel Sysplex environment to perform the final verification of new IBM Z hardware and software.

Summary

A Practical Walkthrough on How to Use Spyre, Mellea, and LangChain for Building Agents in IBM Z. This post explores how we learned how to build custom AI agents using LangChain, IBM Bob, watsonx Assistant for Z (WXA4Z), Cloud Pak for Data (CP4D), z/OS Container Extensions (zCX) and Spyre cards. It showcases the benefits of running AI agents in your hybrid cloud environment.

Intended Audience

The expected audience of this blog post are z/OS system programmers (sysprogs), z/OS & OpenShift administrators, and AI engineers.

Glossary

• LangChain – a JavaScript and Python framework for building LLM-powered applications
• Watsonx Assistant for Z (WXA4Z) – An agentic solution for modernizing Z systems that help assist sysprogs and developers alike in understanding their z/OS components using generative AI.
• Red Hat OpenShift Container Platform (OCP) – An enterprise-grade Kubernetes based hybrid cloud platform to build, deploy, and manage containerized workloads.
• Cloud Pak for Data (CP4D) – A set of containerized services running on Red Hat OpenShift Container Platform (OCP) on IBM Software Hub that allows for installing various types of software like watsonx Orchestrate.
• z/OS Container Extensions (zCX) – A pre-packaged Docker appliance that can run containerized applications within z/OS.
• Spyre AI Card – A high-performance, energy-efficient PCIe add-on designed to enhance AI inferencing capabilities on IBM Z and LinuxONE systems.
• IBM Bob - An agentic coding agent that helps developers write code more efficiently and effectively in enterprise settings.

Introduction

It's an exciting time to be working in tech! However, if you're working on mission critical systems, you may have felt left out since you may not have been able to use 3rd party LLMs due to security reasons. Thankfully, IBM has recently developed two major breakthroughs for IBM Z and z/OS that allow for us to leverage generative AI systems directly on our own infrastructure. That being the Spyre AI cards that give us the hardware possible to inference LLMs within z/OS, and the WXA4Z product that brings AI agents to our z/OS components. Currently, we deployed the IMS, CICS, DB2 and IMS agents in our parallel sysplex which allows us to ask questions in natural language about the state of our environment. If you'd like to learn more about our experience installing WXA4Z on KVM, you can read our report here.

While there are many amazing AI agents provided to us by WXA4Z, each organization may want to build custom agents for their own purposes, which is where this guide comes into place.

Hardware/ Software Requirements

To proceed, we need to ensure that we have WXA4Z installed. You can follow either the previously linked experience report or the official s390x WXA4Z documentation here. I was able to get this working with WXA4Z v3.1 which uses CP4D 5.2 under the hood.

Architectural Overview

Before diving into the technical setup, let's understand how the various components work together to enable custom AI agents on IBM Z infrastructure.

Component Stack

The architecture consists of several layers that work together:

Custom Agent Application (Your Code) (LangChain + Mellea Framework)

↓

Cloud Pak for Data (CP4D) WatsonX.AI (LLM Services) - Model Inference - Project Management

↓

z/OS Container Extensions (zCX) Runtime - Containerized workloads on z/OS

↓

Spyre AI Cards (Hardware Layer) - Accelerated AI inference on IBM Z

Key Components:

Spyre AI Cards

Provide hardware-accelerated AI inference capabilities directly on IBM Z systems, enabling efficient LLM processing without sending data off-platform.

z/OS Container Extensions (zCX)

Allows containerized applications to run within z/OS, bridging traditional mainframe workloads with modern cloud-native architectures.

Cloud Pak for Data (CP4D)

The platform layer that hosts WatsonX.AI services, providing model management, deployment, and inference capabilities. This is where your LLM models are hosted and accessed.

WatsonX Assistant for Z (WXA4Z)

Provides pre-built agents for common z/OS components (IMS, CICS, DB2). While powerful, you may need custom agents for organization-specific workflows.

LangChain

An open-source framework that simplifies building LLM-powered applications by providing abstractions for prompts, chains, and agents.

Mellea

A durable agent framework that implements the IVR (Invoke-Verify-Reflect) loop pattern, ensuring agents can handle complex, multi-step tasks with proper error handling and state management.

Setting up LangChain with CP4D

Retrieving CP4D Credentials

In order to properly authenticate to CP4D with LangChain, you'll need to ensure you have a username, API key, and a project or deployment space id. To follow security best practices, I stored all my credentials in a .env file like so:

# WatsonX Configuration for Cloud Pak for Data (CP4D)
CP4D_URL=https://your-cpd-instance.apps.your-cluster.ibm.com
CP4D_USERNAME=your-cpd-username
CP4D_API_KEY=your-cpd-api-key
CP4D_PROJECT_ID=your-project-id

# Model Configuration
# The model ID to use (default: ibm/granite-3-3-8b-instruct)
MODEL_ID=ibm/granite-3-3-8b-instruct

For the user id and API key that you create, ensure that they have the correct permissions to access CP4D resources. On my CP4D instance with Software Hub, I added my user id to a user group with Administrator authorization. The minimum level of authorization needed is developer.

Another recommendation I have, is to keep your credentials in a secure environment such as Hashicorp Vault, IBM Cloud Secrets Manager, or at the very least a .env file in your project directory.

Python Setup

I used uv Python 3.12, and several packages to create this project. You can create a new Python environment and install the required packages with the following command:

uv init spyre-agents-pocs; cd spyre-agents-pocs

Then install the following packages:

uv add python-dotenv langchain-ibm ibm-watsonx-ai

Now that our Python environment is set up, we can start the client initialization.

Client Initialization

I created the following functions with IBM Bob in order to load my credentials and initialize the client:

"""Shared utilities for CP4D watsonx.ai integration"""
import os
from dotenv import load_dotenv
from ibm_watsonx_ai import APIClient, Credentials


def load_config():
    """Load configuration from .env file"""
    load_dotenv()
    
    config = {
        'url': os.getenv('CP4D_URL'),
        'username': os.getenv('CP4D_USERNAME'),
        'api_key': os.getenv('CP4D_API_KEY'),
        'instance_id': os.getenv('CP4D_INSTANCE_ID', 'openshift'),  # 'openshift' is standard for CP4D on OpenShift
        'project_id': os.getenv('CP4D_PROJECT_ID'),
        'version': os.getenv('CP4D_VERSION', '5.2'),
    }
    
    # Validate required fields
    missing_fields = [key for key, value in config.items()
                     if not value and key not in ['project_id']]
    if missing_fields:
        raise ValueError(f"Missing required configuration: {', '.join(missing_fields)}")
    
    return config


def create_cp4d_client(config):
    """Create and return CP4D API client"""
    credentials = Credentials(
        url=config['url'],
        username=config['username'],
        api_key=config['api_key'],
        instance_id=config['instance_id'],
        version=config['version']
    )
    
    client = APIClient(credentials)
    
    # Set project if provided
    if config['project_id']:
        client.set.default_project(config['project_id'])
    
    return client

Listing Available Models

Now to verify our connection to CP4D is working, let's list out the available models with the watsonx AI python package:

from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

def list_available_models(client):
    """List all available foundation models"""
    try:
        models_details = client.foundation_models.get_model_specs()
        resources = models_details.get('resources', [])
        
        if not resources:
            print("No models found.")
            return
        
        print(f"\nFound {len(resources)} models:\n")
        
        for idx, model in enumerate(resources, 1):
            model_id = model.get('model_id', 'N/A')
            label = model.get('label', 'N/A')
            provider = model.get('provider', 'N/A')
            source = model.get('source', 'N/A')
            
            print(f"{idx}. Model ID: {model_id}")
            print(f"   Label: {label}")
            print(f"   Provider: {provider}")
            print(f"   Source: {source}")
            
            # Show short description if available
            short_desc = model.get('short_description', '')
            if short_desc:
                print(f"   Description: {short_desc}")
            
            # Show supported tasks if available
            tasks = model.get('tasks', [])
            if tasks:
                task_ids = [task.get('id', '') for task in tasks if isinstance(task, dict)]
                if task_ids:
                    print(f"   Tasks: {', '.join(task_ids)}")
            
            # Show model limits if available
            model_limits = model.get('model_limits', {})
            if model_limits:
                max_seq_length = model_limits.get('max_sequence_length', 'N/A')
                max_output_tokens = model_limits.get('max_output_tokens', 'N/A')
                print(f"   Max Sequence Length: {max_seq_length}")
                print(f"   Max Output Tokens: {max_output_tokens}")
            
            print()
        
    except Exception as e:
        print(f"Error: {str(e)}")


def main():
    """Connect to CP4D and list models"""
    try:
        config = load_config()
        client = create_cp4d_client(config)
        list_available_models(client)
    except ValueError as e:
        print(f"Configuration error: {str(e)}")
    except Exception as e:
        print(f"Error: {str(e)}")


if __name__ == "__main__":
    main()

I then ran my example script with the following command: uv run list_models.py

Found 1 models:

1. Model ID: ibm/granite-3-3-8b-instruct
   Label: granite-3-3-8b-instruct
   Provider: IBM
   Source: IBM
   Description: Granite-3.3-8b-Instruct is an IBM-trained, dense decoder-only models, which is particularly well-suited for generative tasks.
   Tasks: question_answering, summarization, retrieval_augmented_generation, classification, generation, code, extraction, translation, function_calling
   Max Sequence Length: 32768
   Max Output Tokens: 4096

This confirms that our WXA4Z install was done correctly and that we can access the models.

LangChain Examples

Basic Inference with Watsonx AI

Building off the previous example, I wanted to see if I could use the base Watsonx AI package to inference granite 3.3 8b:

def main():
    """Minimal inference example"""
    # Load config and create client
    config = load_config()
    client = create_cp4d_client(config)
    
    # Initialize model
    model = ModelInference(
        model_id="ibm/granite-3-3-8b-instruct",
        api_client=client,
        project_id=config['project_id']
    )
    
    # Generate response
    response = model.generate_text(prompt="What is the capital of North Carolina?")
    
    print(response)


if __name__ == "__main__":
    main()

This produces the following output:

The capital of North Carolina is Raleigh. Established in 1

Which is factually correct, and we can see some slight truncation with the max tokens. However, for production-ready agent development, we need a more robust framework such as LangChain.

A Basic LangChain ReAct Agent

The IBM Watsonx.AI Integration doesn't just work with IBM Cloud, but also for CP4D which allows us to do really interesting things. In this case, I created a basic ReAct agent using ChatWatsonx.

"""
ReACT Agent with IBM watsonx.ai
Demonstrates agent reasoning with custom tools in CP4D environments
"""

from langchain_ibm import ChatWatsonx
from langchain_core.tools import tool
from langchain.agents import create_agent
from cp4d_utils import load_config


@tool
def calculate(expression: str) -> str:
    """Evaluate a mathematical expression. Input should be a valid math expression like '2+2' or '10*5'.
    
    SECURITY WARNING: This example uses a restricted eval() for demonstration purposes only.
    In production, use ast.literal_eval() or a proper math expression parser library like 'numexpr' or 'simpleeval'.
    """
    try:
        # Restricted eval with no builtins - still not recommended for production
        result = eval(expression, {"__builtins__": {}}, {})
        return f"Result: {result}"
    except Exception as e:
        return f"Error: {str(e)}"


@tool
def get_string_length(text: str) -> str:
    """Get the length of a text string."""
    return f"Length: {len(text)} characters"


@tool
def reverse_string(text: str) -> str:
    """Reverse a text string."""
    return f"Reversed: {text[::-1]}"


def main():
    """Run ReACT agent with custom tools"""
    print("=" * 80)
    print("ReACT Agent with IBM watsonx.ai")
    print("=" * 80)
    
    # Load configuration
    config = load_config()
    
    # Define tools
    tools = [calculate, get_string_length, reverse_string]
    
    # Initialize chat model
    llm = ChatWatsonx(
        model_id="ibm/granite-3-3-8b-instruct",
        url=config["url"],
        username=config["username"],
        apikey=config["api_key"],
        instance_id=config["instance_id"],
        version=config["version"],
        project_id=config["project_id"],
        params={
            "max_new_tokens": 500,
            "temperature": 0.1,
        }
    )
    
    # Create agent
    agent = create_agent(llm, tools)
    
    # Example queries demonstrating ReACT pattern
    queries = [
        "What is 15 multiplied by 7?",
        "Calculate 100 divided by 4, then tell me the length of the word 'watsonx'",
        "Reverse the string 'hello world' and tell me its length",
    ]
    
    # Run queries
    for query in queries:
        print(f"\n{'=' * 80}")
        print(f"Query: {query}")
        print("=" * 80)
        
        for chunk in agent.stream(
            {"messages": [{"role": "user", "content": query}]},
            stream_mode="values"
        ):
            chunk["messages"][-1].pretty_print()


if __name__ == "__main__":
    try:
        main()
    except ValueError as e:
        print(f"\nConfiguration error: {e}")
        print("Ensure .env file has required CP4D credentials.\n")
    except Exception as e:
        print(f"\nError: {e}\n")

Which when run, will produce the following output:

================================================================================
ReACT Agent with IBM watsonx.ai
================================================================================

================================================================================
Query: What is 15 multiplied by 7?
================================================================================
================================ Human Message =================================

What is 15 multiplied by 7?
================================== Ai Message ==================================
Tool Calls:
  calculate (chatcmpl-tool-bf73d55832554ea5a7eeca8ac8b72423)
 Call ID: chatcmpl-tool-bf73d55832554ea5a7eeca8ac8b72423
  Args:
    expression: 15 * 7
================================= Tool Message =================================
Name: calculate

Result: 105
================================== Ai Message ==================================

15 multiplied by 7 equals 105.

================================================================================
Query: Calculate 100 divided by 4, then tell me the length of the word 'watsonx'
================================================================================
================================ Human Message =================================

Calculate 100 divided by 4, then tell me the length of the word 'watsonx'
================================== Ai Message ==================================
Tool Calls:
  calculate (chatcmpl-tool-555fc71b8a4a4d1889078fb07ff9dde7)
 Call ID: chatcmpl-tool-555fc71b8a4a4d1889078fb07ff9dde7
  Args:
    expression: 100 / 4
  get_string_length (chatcmpl-tool-ffaef7db39ce4c5b98be173aace95429)
 Call ID: chatcmpl-tool-ffaef7db39ce4c5b98be173aace95429
  Args:
    text: watsonx
================================= Tool Message =================================
Name: get_string_length

Length: 7 characters
================================== Ai Message ==================================

The result of 100 divided by 4 is 25.0. The word 'watsonx' has 7 characters.

================================================================================
Query: Reverse the string 'hello world' and tell me its length
================================================================================
================================ Human Message =================================

Reverse the string 'hello world' and tell me its length
================================== Ai Message ==================================
Tool Calls:
  reverse_string (chatcmpl-tool-3405e5cfa66d4389b25de8f422645c8f)
 Call ID: chatcmpl-tool-3405e5cfa66d4389b25de8f422645c8f
  Args:
    text: hello world
  get_string_length (chatcmpl-tool-4cd9a7459f104dabbdb207b6fe09ab73)
 Call ID: chatcmpl-tool-4cd9a7459f104dabbdb207b6fe09ab73
  Args:
    text: hello world
================================= Tool Message =================================
Name: get_string_length

Length: 11 characters
================================== Ai Message ==================================

The reversed string of 'hello world' is 'dlrow olleh' and its length is 11 characters.

This is a great, simple use case of using agents with custom tools. However, in 2026, the bar for agent capabilities has been raised. It's not enough to just define tools, we need to also enforce strict checks to prevent hallucination and destructive actions when performing automated actions in our Z systems. This is where Mellea can help our granite 3.3 8b model achieve performance beyond its typical capabilities.

Introduction to Mellea

Mellea is an AI framework designed to enhance small language model (SLM) performance through durable agent patterns. At the All Things AI conference in Raleigh, NC, Mellea developers demonstrated how the IVR loop (Instruct-Validate-Repair) enables granite-3.3-8b to achieve performance comparable to Llama 3 70b. The IVR loop addresses the primary limitation of SLMs—their inability to zero-shot certain tasks (perform tasks without prior examples)—by iteratively validating and repairing model outputs. The comparison below shows granite-3.3-8b with IVR matching Llama 3 70b performance. The presenter, Luis Lastras, gets full credit for the presentation and workshop where he showcased the power of Mellea and SLMs.

Image courtesy of Edrine Mutebi from Luis Lastras' presentation, All Things AI 2026

Setting up Mellea with CP4D

Mellea's native Watsonx.AI integration was deprecated in version 0.4.x, with the recommendation to use LiteLLM as the backend instead. While this works, integrating LiteLLM with CP4D requires a few extra steps compared to IBM Cloud WatsonX. Specifically, you'll need to:

• Disable SSL warnings for self-signed certificates (common in on-premises CP4D deployments)
• Generate a Bearer token from your CP4D Zen API key (two-step authentication flow)
• Configure LiteLLM-specific environment variables

Prerequisites

Before proceeding, ensure you have:

• Python 3.12+ installed (I used uv for package management, but pip works too)
• Access to a CP4D instance with WatsonX.AI deployed
• The .env file we created earlier with your CP4D credentials

Installation

Install Mellea with the LiteLLM backend support:

# Using uv (recommended)
uv add "mellea[litellm]" requests

The [litellm] extra includes support for multiple LLM providers including WatsonX. You can also combine extras if needed:

uv add "mellea[litellm,tools,telemetry]"

Understanding Mellea's Value Proposition

Before diving into the code, let's talk about why you'd use Mellea instead of calling LiteLLM directly. Mellea's killer feature is requirement enforcement through its IVR (Instruct-Validate-Repair) loop. This means you can specify strict requirements for model outputs, and Mellea will automatically validate and retry until those requirements are met. This is especially valuable when working with smaller models like granite-3.3-8b that might not nail complex instructions on the first try.

In the example below, we'll ask the model to write an email with two specific requirements: include a salutation and write like a pirate. Without Mellea, you'd need to manually check if the model followed both instructions and retry if it didn't. With Mellea, this validation and retry logic is built-in.

Complete Mellea Example

Security Note: This example disables SSL certificate verification for development environments with self-signed certificates. Never use verify=False or disable SSL warnings in production. For production deployments, properly configure SSL certificates or use a certificate bundle.

Below is the Python script I used as a PoC to showcase how Mellea can interact with CP4D and our Spyre cards. This example demonstrates requirement enforcement—a powerful pattern that ensures granite 3.3 8b consistently follows strict instructions, saving you future headaches when building production AI agents.

import os
import warnings
import requests
from dotenv import load_dotenv
from mellea import MelleaSession
from mellea.backends.litellm import LiteLLMBackend
import litellm

#  DEVELOPMENT ONLY: Suppress SSL warnings for self-signed certificates
# Remove these lines for production deployments with proper SSL certificates
warnings.filterwarnings('ignore', message='Unverified HTTPS request')
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
litellm.ssl_verify = False

# Load environment variables from .env file
load_dotenv()

# Get configuration from environment variables (same .env as earlier examples)
cp4d_url = os.getenv('CP4D_URL')
cp4d_username = os.getenv('CP4D_USERNAME')
cp4d_api_key = os.getenv('CP4D_API_KEY')
cp4d_project_id = os.getenv('CP4D_PROJECT_ID')
model_id = os.getenv('MODEL_ID', 'ibm/granite-3-3-8b-instruct')

# Validate required environment variables
if not cp4d_url:
    raise ValueError("CP4D_URL environment variable is required")
if not cp4d_username:
    raise ValueError("CP4D_USERNAME environment variable is required")
if not cp4d_api_key:
    raise ValueError("CP4D_API_KEY environment variable is required")
if not cp4d_project_id:
    raise ValueError("CP4D_PROJECT_ID environment variable is required")

# Note: In production environments, avoid logging sensitive configuration details
print(f"Configuration:")
print(f"  URL: {cp4d_url}")
print(f"  Project ID: {cp4d_project_id}")
print(f"  Model: {model_id}")

# Step 1: Generate Bearer token from CP4D Zen API key
# CP4D requires a two-step auth: API key → Bearer token → LiteLLM
auth_url = f"{cp4d_url}/icp4d-api/v1/authorize"
try:
    auth_response = requests.post(
        auth_url,
        json={
            "username": cp4d_username,
            "api_key": cp4d_api_key
        },
        verify=False  # DEVELOPMENT ONLY
    )
    auth_response.raise_for_status()
    bearer_token = auth_response.json().get('token')
    print("Bearer token generated successfully")
except Exception as e:
    print(f"Failed to generate Bearer token: {e}")
    raise

# Step 2: Configure LiteLLM for CP4D WatsonX
# LiteLLM expects the 'watsonx/' prefix for model routing
litellm_model = f"watsonx/{model_id}"

# Step 3: Set LiteLLM-specific environment variables
# Note: LiteLLM expects WATSONX_* variables (not CP4D_*)
os.environ['WATSONX_URL'] = cp4d_url
os.environ['WATSONX_TOKEN'] = bearer_token
os.environ['WATSONX_PROJECT_ID'] = cp4d_project_id

# Step 4: Create Mellea session with LiteLLM backend
m = MelleaSession(
    LiteLLMBackend(
        model_id=litellm_model,
        model_options={
            "project_id": cp4d_project_id,
            "max_tokens": 1000,
            "temperature": 0.1,
        }
    )
)

print("Mellea session initialized successfully")

# Step 5: Test Mellea's requirement enforcement
# The model must satisfy BOTH requirements or Mellea will retry
try:
    result = m.instruct(
        "Write an email to an enterprise explaining why IBM Z and Spyre are the best thing since sliced bread.",
        requirements=[
            "The email should have a salutation.",
            "Talk like a pirate"
        ],
    )
    print("\nResponse:")
    print(str(result))
except Exception as e:
    print(f"Error: {e}")
    raise

The following output should look something similar to this:

Response:
Subject: Hoist the Anchors, Matey! Discover the Treasures of IBM Z and Spyre











Ahoy there,











I hope this message finds ye well and prosperous. I be writing to share some swashbuckling news about IBM Z and Spyre, the finest solutions since sliced bread, or in our case, the discovery of the New World!











Firstly, IBM Z, a mighty mainframe, be a robust and secure platform that can handle the most demanding enterprise workloads. It be as reliable as a sturdy ship in a storm, offering unparalleled performance, security, and resilience. With IBM Z, ye can navigate the choppy waters of digital transformation with confidence, knowing that your critical applications and data are safe and sound.











Now, let's talk about Spyre, a game-changing tool that simplifies the management of your IBM Z environment. Spyre be like a reliable compass, guiding ye through the complexities of system administration. It offers a unified, intuitive interface that streamlines operations, reduces manual tasks, and enhances productivity. With Spyre, ye can focus on charting your course for innovation, rather than getting bogged down in the minutiae of system management.











Together, IBM Z and Spyre form a formidable duo, akin to a well-crewed galleon. They empower enterprises to sail smoothly through the seas of modern business, ensuring security, efficiency, and agility.











So, there ye have it, me hearty. IBM Z and Spyre be the best thing since sliced bread, or more appropriately, the best thing since the discovery of a treasure trove of gold!











Yours in the pursuit of digital treasure,











[Your Name]
[Your Position]When you run this script, you'll see several warning messages—don't panic! These are expected:

=== 11:12:42-WARNING ======
litellm allows for unknown / non-openai input params; mellea won't validate the following params that may cause issues: project_id

This warning appears because LiteLLM's OpenAI-compatible interface doesn't recognize WatsonX-specific parameters like project_id. This is normal and won't affect functionality—LiteLLM passes these parameters through to WatsonX correctly.

The progress bar shows Mellea's IVR loop in action, validating that both requirements (salutation + pirate speak) are met. If the model fails to satisfy a requirement on the first try, Mellea automatically retries with feedback until success.

Troubleshooting

Authentication Errors

• Verify your CP4D credentials in the .env file
• Ensure your user has access to the specified project ID
• Check that your CP4D Zen API key hasn't expired

SSL Certificate Errors

• If you see SSL verification errors despite disabling warnings, check your Python environment's certificate configuration
• For production, obtain proper SSL certificates from your CP4D administrator

Model Not Found

• Verify the model ID exists in your CP4D instance using the list_models.py script from earlier
• Ensure your project has access to the specified model
• Note: At this point in time, only granite 3.3 8b is able to be used with Spyre cards on WXA4Z 3.1 and 3.2 for s390x systems.

Next Steps

Now that you've seen Mellea's requirement enforcement in action, you can:

1. Build Production Agents: Use Mellea's IVR loop to ensure your AI agents consistently follow complex business rules
2. Integrate with z/OS: Combine this pattern with z/OS REST APIs to create agents that safely interact with mainframe systems
3. Explore Advanced Features: Check out Mellea's documentation for tool use, multi-step reasoning, and telemetry
4. Scale with Spyre: Leverage your Spyre AI cards for high-throughput inference in production workloads

The key takeaway: smaller models like granite-3.3-8b can achieve performance beyond their typical capabilities when paired with durable agent patterns like Mellea's IVR loop. This is especially valuable in enterprise environments where you need predictable, auditable AI behavior.

Conclusion

We've covered a lot of ground in this walkthrough—from setting up LangChain with CP4D, to building ReAct agents, to leveraging Mellea's IVR loop for requirement enforcement. The key takeaway? You don't need massive models or cloud-based LLMs to build powerful AI agents for your z/OS environment.

With the combination of Spyre AI cards, CP4D, and frameworks like LangChain and Mellea, we can run sophisticated AI workloads directly on our own infrastructure. This means we get the benefits of generative AI while maintaining the security, compliance, and control that mission-critical systems demand.

The granite-3.3-8b model, when paired with Mellea's durable agent patterns, can consistently follow complex instructions and business rules—something that's essential when you're automating tasks in production environments. Whether you're building custom agents for IMS, CICS, DB2, or any other z/OS component, these patterns give you a solid foundation.

What excites me most is that this is just the beginning. As we continue to explore what's possible with Spyre cards and WXA4Z, we're discovering new ways to bring AI capabilities to the mainframe without compromising on the reliability and security that IBM Z is known for. The future of AI on Z is here, and it's running on our own hardware.

If you're working on similar projects or have questions about implementing these patterns in your environment, feel free to reach out. The z/OS community is stronger when we share our experiences and learn from each other.

References

1. IBM Bob - https://bob.ibm.com/
2. WXA4Z Experience Report - https://community.ibm.com/community/user/viewdocument/ibm-wxa4z-experience-report?CommunityKey=2a2f855c-5950-4a9d-8485-86645982646a
3. WatsonX Assistant for Z Documentation - https://www.ibm.com/docs/en/watsonx/waz/3.1.0?topic=install-premises-watsonx-assistant-z-s390x
4. LangChain IBM Watsonx.AI Integration - https://docs.langchain.com/oss/python/integrations/tools/ibm_watsonx
5. Mellea Documentation - https://docs.mellea.ai/
6. Mellea IVR Loop Glossary - https://docs.mellea.ai/guide/glossary#ivr-instruct-validate-repair
7. All Things AI Conference - https://allthingsopen.org/events/all-things-ai-2026
8. LiteLLM WatsonX Provider Documentation - https://docs.litellm.ai/docs/providers/watsonx/