z/OS Container Extensions (zCX)

z/OS Container Extensions (zCX)

z/OS Container Extensions (zCX)

Deploying Linux on Z containerized applications inside z/OS

 View Only

Tech Preview: Red Hat AI Inference Server on zCX

By Charles Volzka posted 03/17/26 09:48 AM

  

Technology Preview:

Red Hat® AI Inferencing Server deployment on IBM® z/OS® Container Extensions (zCX) and IBM® zCX Foundation for Red Hat OpenShift will be available as technical preview1 starting 03/20/2026.

Introduction

In this blog, we'll give a brief overview of the technologies involved in utilizing IBM Spyre Accelerators on IBM® z/OS® and easy to replicate examples in your zCX environments so you can begin experimenting1.

With Red Hat AI Inference Server and IBM Spyre Accelerators, you can integrate AI inferencing workloads within IBM z/OS. This brings your AI inferencing closer to your z/OS applications and data, reducing latency and gaining the security of IBM zCX environments in z/OS. You maintain operational control within z/OS and transparently inherit z/OS qualities of service.

Key Technologies

IBM z/OS Container Extensions (zCX)

IBM z/OS Container Extensions (zCX) makes it possible to integrate Linux on IBM Z applications with z/OS. Application developers can develop and data centers can operate popular open source packages, Linux applications, IBM software, and third-party software together with z/OS applications and data.

IBM zCX Foundation for Red Hat OpenShift (zCX for OpenShift)

IBM zCX Foundation for Red Hat OpenShift provides the capability to run Red Hat OpenShift on z/OS®, using IBM z/OS Container Extensions (zCX). Red Hat OpenShift provides open, enterprise-level, cloud-native tools and orchestration capability for your hybrid cloud architecture on z/OS across multiple CPCs and LPARs.

IBM Spyre Accelerator

IBM Spyre Accelerator is a high-performance, energy-efficient PCIe add-on designed to enhance AI inferencing capabilities on IBM Z® and LinuxONE systems.

Red Hat AI Inference Server (RHAIIS)

Red Hat AI Inference Server is a containerized, enterprise-grade, solution designed to optimize the deployment of Large Language Models (LLMs) and generative AI. It optimizes how AI models generate outputs (inference), enabling faster, more efficient, and more cost‑effective model deployment at scale. 

Running RHAIIS on zCX environments

For these examples, we assume that you already have a zCX environment provisioned with Spyre support enabled and a minimum of four IBM Spyre Accelerators set up and assigned to the IBM z/OS LPAR(s) running the zCX environment.

Spyre support will be added to zCX 2.4 maintenance level and to IBM zCX Foundation for Red Hat OpenShift in the March 2026 update. After the APARs are installed, you will need to either provision new instances with Spyre support enabled or run the reconfigure workflow to enable Spyre support. Once the reconfigure workflow is run, you will also need to restart the instance. For more information on how to enable Spyre support using workflows, please see OCP reconfiguration workflow.

Afterward, in the job log, you should see:

GLZP008I zCX instance XXXX: Current Spyre device counts:  YYY
    YYY             Number requested:    4
    YYY             Number obtained:     4
    YYY             Number not obtained: 0

indicating the zCX instance was able to assign Spyre devices from the z/OS LPAR.

Both zCX and zCX for OpenShift use the same image when starting RHAIIS instances. However, the steps to start docker containers on zCX varies from how to start pods on zCX on OpenShift. Below, we'll describe how to start RHAIIS on either platform and sending  a simple inference request to the server.

To download the image you will need a Red Hat login. If you don't have one, you can create one following this Red Hat article. Additionally, if you want trial access to Red Hat's Customer Portal for RHAIIS, you will also need to sign up for a Red Hat AI Inference Server trial.

Running RHAIIS on zCX using docker

  1. Log in to the Red Hat image registry
    1. Run docker login registry.redhat.io
    2. You will be prompted for your Red Hat login and password
  2. (once) Download the model to a Docker Volume which will later be attached to the RHAIIS container

    1. docker run -it --rm --pull=always --user=0 --mount source=rhaiis-zcx-preview-model,target=/models --workdir=/models registry.redhat.io/openshift-builds/openshift-builds-git-cloner-rhel9:1.6.2 --url https://huggingface.co/ibm-granite/granite-3.3-8b-instruct --target /models/granite-3.3-8b-instruct

    2. The download is approximately 16Gb and will take a while depending on your internet connection. 
    3. It will appear stuck at the ".../usr/bin/git -c safe.directory=/models/granite-3.3-8b-instruct clone..." while it downloads. Please be patient
    4. Once downloaded to a Volume, the model can be reused across deployments
  3. Start a RHAIIS container by running:

    1. docker run $(spyre-cdi devices oci --num 4) --env LD_PRELOAD="" -v rhaiis-zcx-preview-model:/models:Z --pids-limit 0 --memory 128G --shm-size 4G -p 8000:8000 --name rhaiis-zcx-preview registry.redhat.io/rhaiis/vllm-spyre-rhel9:3.3.0 --model /models/granite-3.3-8b-instruct -tp 4 --max-model-len 32768 --max-num-seqs 32

    2. Once started, the RHAIIS container needs to prepared the model and expand the weights. Wait until the container logs show Application startup complete before sending inference requests. This takes 5-6 min depending on CPUs
    3. Note: You may also add --detach after docker run to run the container in detached mode. You will immediately return to the CLI once the container starts.
      1. To view container logs you will need to run docker logs -f rhaiis-zcx-preview
      2. To stop and delete the detached container run docker rm -f rhaiis-zcx-preview
  4. Send an inference request
    1. Run: curl -X POST -H "Content-Type: application/json" -d '{"model": "/models/granite-3.3-8b-instruct", "prompt": "What is the capital of France?", "max_tokens": 50}' http://<ZCX_INSTANCE_IP_OR_HOSTNAME>:8000/v1/completions
      1. (optional) add  | jq at the end of the command to make it more human readable. However jq is not installed in the zCX CLI interface
  5. Example results:
    Note: Due to the nondeterministic nature of the model, the text section may differ between runs but should answer the question in the prompt.
    1. {"id":"cmpl-87f4184034f04e83aa729a0635028066","object":"text_completion","created":1773431404,"model":"/models/granite-3.3-8b-instruct","choices":[{"index":0,"text":"\n\nThe capital of France is Paris.\n\nAnd the currency used there is called?\n\nThe currency used in France is the Euro.\n\nWhat about the official language?\n\nThe official language of France is French.","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":7,"total_tokens":57,"completion_tokens":50,"prompt_tokens_details":null},"kv_transfer_params":null}
  6. To remove RHAIIS run the following: 
    1. docker rmiregistry.redhat.io/openshift-builds/openshift-builds-git-cloner-rhel9:1.6.2
    2. docker rmi registry.redhat.io/rhaiis/vllm-spyre-rhel9:v.3.3.0
    3. docker volume rmrhaiis-zcx-preview-model

Running RHAIIS on zCX using OpenShift

  1. Ensure the Spyre Operator for IBM Z and IBM® LinuxONE is installed and setup so your cluster is aware of Spyre devices
  2. Copy rhaiis-zcx-preview.yml (below) to a machine that can run oc commands for your zCX for OpenShift
    1. rhaiis-zcx-preview.yml 
      # Contents: Namespace, Deployment, Service, and Route to deploy Red Hat AI Inference Server on zCX for OpenShift
# Prerequiements: OpenShift installed with Spyre Operator installed, 18GB for the RHAIIS image
# Minimum Requirements:
#   Note: The requirements are per replica pod and a single node must be able to supply all requirements.
#     If you add additional replica pods, each will need that many resources.
#   Storage: 32GB for the model post compilation
#   Memory: 128 GB
#   Spyre: 4 devices

apiVersion: v1
kind: Namespace
metadata:
  name: rhaiis-zcx-preview
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rhaiis-zcx-preview-deployment
  namespace: rhaiis-zcx-preview
  labels:
    app: rhaiis-zcx-preview-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rhaiis-zcx-preview-app
  template:
    metadata:
      labels:
        app: rhaiis-zcx-preview-app
    spec:
      volumes:
      - name: rhaiis-zcx-preview-models
        emptyDir: {}
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 4Gi
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      initContainers:
      - name: rhaiis-zcx-preview-init
        image: registry.redhat.io/openshift-builds/openshift-builds-git-cloner-rhel9:1.6.2
        args:
          - '--url=https://huggingface.co/ibm-granite/granite-3.3-8b-instruct'
          - '--target=/models/granite-3.3-8b-instruct'
        volumeMounts:
        - name: rhaiis-zcx-preview-models
          mountPath: /models
      containers:
      - name: rhaiis-zcx-preview-pod
        image: registry.redhat.io/rhaiis/vllm-spyre-rhel9:latest
        args:
          - '--model=/models/granite-3.3-8b-instruct'
          - '--port=8000'
          - '--max-model-len=32768'
          - '--max-num-seqs=32'
          - '--tensor-parallel-size=4'
        env:
          # This causes container to not load libjemalloc which reduces
          # memory usage on IBM Z Systems
          - name: LD_PRELOAD
            value: ""
        resources:
          limits:
            memory: 128Gi
            ibm.com/spyre_vf: 4
        volumeMounts:
        - name: rhaiis-zcx-preview-models
          mountPath: /models
        - name: dshm
          mountPath: /dev/shm
        ports:
        - containerPort: 8000
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
---
apiVersion: v1
kind: Service
metadata:
  name: rhaiis-zcx-preview-service
  namespace: rhaiis-zcx-preview
spec:
  selector:
    app: rhaiis-zcx-preview-app
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: rhaiis-zcx-preview-route
  namespace: rhaiis-zcx-preview
spec:
  to:
    kind: Service
    name: rhaiis-zcx-preview-service
  port:
    targetPort: 8000
    2. If desired, change the number of "replicas" in the yaml. "2" or higher is required for pod failover. Value in the yaml above is set to "1". 
    3. Run: oc apply -f rhaiis-zcx-preview.yml
  3. Wait for Deployment to fully start:
    1. For each replica pod in the Deployment, the rhaiis-zcx-preview-init container downloads about 16Gb of model weights before the rhaiis-zcx-preview-pod is started. This will take several minutes. Be patient.
    2. Once started, the rhaiis-zcx-preview-pod needs to prep the model and expand the weights. This takes a few minutes after the pod shows as "Ready" in the OpenShift webui. This takes 5-6 min depending on CPUs
    3. Each Deployment Pod is ready for inference requests when Logs for rhaiis-zcx-preview-pod shows Application startup complete.
  4. The rhaiis-zcx-preview.yml creates a Service and Route so you can make an inference request from either the rhaiis-zcx-preview-pod, your zCX bastion node, or externally from the OpenShift cluster via a REST call.
    1. To perform inference from within the rhaiis-zcx-preview-pod, Go to the OpenShift web UI's Terminal tab for a pod and run:
      1. curl -X POST -H "Content-Type: application/json" -d '{ "model": "/models/granite-3.3-8b-instruct", "prompt": "What is the capital of France?", "max_tokens": 50 }' localhost:8000/v1/completions | jq
    2. To perform inference from the zCX bastion node:
      1. Login to bastion with: ssh -t <bastion_ip> docker exec -it zcxocp-cli bash
      2. (one time only, install curl and jq with): apt-get update && apt-get install -y curl jq
      3. To send an inference request, run: curl -X POST -H "Content-Type: application/json" -d '{ "model": "/models/granite-3.3-8b-instruct", "prompt": "What is the capital of France?", "max_tokens": 50 }' http://rhaiis-zcx-preview-route-rhaiis-zcx-preview.apps.$(oc get dnses.config.openshift.io cluster -o jsonpath='{.spec.baseDomain}{"\n"}')/v1/completions | jq
    3. From outside of the OpenShift cluster:
      1. If using zCX bastion node as a haproxy:
        1. (one time only) Add rhaiis-zcx-preview-route-rhaiis-zcx-preview.apps.<bastion_fqdn> to your workstation's /etc/hosts line for the cluster. For example 9.x.y.z rhaiis-zcx-preview-route-rhaiis-zcx-preview.apps.<bastion_fully_qualified_domain_name>
      2. Send a REST request:
        1. curl -X POST -H "Content-Type: application/json" -d '{ "model": "/models/granite-3.3-8b-instruct", "prompt": "What is the capital of France?", "max_tokens": 50 }' http://rhaiis-zcx-preview-route-rhaiis-zcx-preview.apps.<bastion_fqdn>/v1/completions | jq
  5. Example results:
    Note: Due to the nondeterministic nature of the model, the text section may differ between runs but should answer the question in the prompt.
    {
  "id": "cmpl-51607f6d08a647eaa8caf794bce233b8",
  "object": "text_completion",
  "created": 1773431322,
  "model": "/models/granite-3.3-8b-instruct",
  "choices": [
    {
      "index": 0,
      "text": "\nThe capital of France is Paris. Paris is not only the capital of France, but it is also known as the \"City of Light\" (La Ville Lumière) due to its leading role during the Age of Enlight",
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null,
      "prompt_logprobs": null,
      "prompt_token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 7,
    "total_tokens": 57,
    "completion_tokens": 50,
    "prompt_tokens_details": null
  },
  "kv_transfer_params": null
}

Conclusion

Running Red Hat AI Inference Server with IBM Spyre Accelerators on zCX brings high‑performance AI inferencing directly to your z/OS environment, close to your applications, your data, and your existing operational processes. Whether using zCX or zCX for OpenShift, the steps outlined here give you a clear starting point to experiment with generative AI workloads on IBM Z.

We encourage you to explore, iterate, and build on these examples as you bring AI inferencing closer to your mission‑critical workloads. You’ll be able to expand from simple examples to more advanced AI use cases with confidence.

1Disclaimer: 

Technology Preview features are not fully supported, may not be functionally complete, and are not suitable for deployment in production. However, these features are provided to the customer as a courtesy and the primary goal is for the feature to gain wider exposure with the goal of full support in the future. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. Statements regarding IBM plans, directions, and intent are subject to change or withdrawal without notice at the sole discretion of IBM.

0 comments
35 views

Permalink

https://community.ibm.com/community/user/blogs/charles-volzka/2026/03/10/zcx-rhaiis