Prerequisites

The following are dependency operators that need to be installed before installing IBM Spyre Operator

Node Feature Discovery (NFD) Operator is an OpenShift Operator that automates the detection of
hardware features and system configurations across cluster nodes.
Red Hat Cert Manager Operator is a cluster-wide service that automates application certificate
lifecycle management.
Secondary Scheduler Operator: allows to deploy a custom scheduler alongside the default OpenShift
scheduler.

Node Feature Discovery (NFD) Operator

Installation Steps:

• Log in to the OpenShift web console as a cluster administrator.
• In the left panel, navigate to Operators → OperatorHub.
• On the OperatorHub page, type Node Feature Discovery into Filter by Keyword box.

• Click on the Node Feature Discovery provided by Red Hat.
• Select Stable from Channel drop down and choose latest version available from Version drop down.
• Click Install. The Install Operator page opens.

• Choose A specific namespace on the cluster under Installation mode.
• Choose Operator recommended Namespace under Installed Namespace which will be created for you.
• Choose Automatic under Update approval.
• Click Install.

• The Installing Operator pane appears. When the installation finishes, a checkmark appears next to the Operator name.

Verification

• In the OpenShift web console, from the side panel, navigate to Operators → Installed Operators and confirm that the Node Feature Discovery is in Succeeded state.

Configure the Node Feature Discovery Operator

• Click on Node Feature Discovery Operator.
• Navigate to Node Feature Discovery tab.
• Select Create NodeFeatureDiscovery.

• Provide the following YAML contents and then click Create at the bottom of the page.

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  extraLabelNs:
  - ibm.com
  instance: "" # instance is empty by default
  operand:
    image: 'registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.16'
    imagePullPolicy: Always
    servicePort: 12000
  workerConfig:
    configData: |
      #core:
      #  labelWhiteList:
      #  noPublish: false
      #  sleepInterval: 60s
      #  sources: [all]
      #  klog:
      #    addDirHeader: false
      #    alsologtostderr: false
      #    logBacktraceAt:
      #    logtostderr: true
      #    skipHeaders: false
      #    stderrthreshold: 2
      #    v: 0
      #    vmodule:
      ##   NOTE: the following options are not dynamically run-time configurable
      ##         and require a nfd-worker restart to take effect after being changed
      #    logDir:
      #    logFile:
      #    logFileMaxSize: 1800
      #    skipLogHeaders: false
      #sources:
      #  cpu:
      #    cpuid:
      ##     NOTE: whitelist has priority over blacklist
      #      attributeBlacklist:
      #        - "BMI1"
      #        - "BMI2"
      #        - "CLMUL"
      #        - "CMOV"
      #        - "CX16"
      #        - "ERMS"
      #        - "F16C"
      #        - "HTT"
      #        - "LZCNT"
      #        - "MMX"
      #        - "MMXEXT"
      #        - "NX"
      #        - "POPCNT"
      #        - "RDRAND"
      #        - "RDSEED"
      #        - "RDTSCP"
      #        - "SGX"
      #        - "SSE"
      #        - "SSE2"
      #        - "SSE3"
      #        - "SSE4.1"
      #        - "SSE4.2"
      #        - "SSSE3"
      #      attributeWhitelist:
      #  kernel:
      #    kconfigFile: "/path/to/kconfig"
      #    configOpts:
      #      - "NO_HZ"
      #      - "X86"
      #      - "DMI"
      #  pci:
      #    deviceClassWhitelist:
      #      - "0200"
      #      - "03"
      #      - "12"
      #    deviceLabelFields:
      #      - "class"
      #      - "vendor"
      #      - "device"
      #      - "subsystem_vendor"
      #      - "subsystem_device"
      #  usb:
      #    deviceClassWhitelist:
      #      - "0e"
      #      - "ef"
      #      - "fe"
      #      - "ff"
      #    deviceLabelFields:
      #      - "class"
      #      - "vendor"
      #      - "device"
      #  custom:
      #    - name: "my.kernel.feature"
      #      matchOn:
      #        - loadedKMod: ["example_kmod1", "example_kmod2"]
      #    - name: "my.pci.feature"
      #      matchOn:
      #        - pciId:
      #            class: ["0200"]
      #            vendor: ["15b3"]
      #            device: ["1014", "1017"]
      #        - pciId :
      #            vendor: ["8086"]
      #            device: ["1000", "1100"]
      #    - name: "my.usb.feature"
      #      matchOn:
      #        - usbId:
      #          class: ["ff"]
      #          vendor: ["03e7"]
      #          device: ["2485"]
      #        - usbId:
      #          class: ["fe"]
      #          vendor: ["1a6e"]
      #          device: ["089a"]
      #    - name: "my.combined.feature"
      #      matchOn:
      #        - pciId:
      #            vendor: ["15b3"]
      #            device: ["1014", "1017"]
      #          loadedKMod : ["vendor_kmod1", "vendor_kmod2"]

Red Hat Cert Manager Operator

Installation Steps:

• Log in to the OpenShift web console as a cluster administrator.
• In the left panel, navigate to Operators → OperatorHub.
• On the OperatorHub page, type cert-manager Operator for Red Hat OpenShift into Filter by Keyword box.

• Click on cert-manager Operator for Red Hat OpenShift (not community version).
• Select stable-v1 from Channel drop down and choose latest version available from Version drop down.

• Click Install. The Install Operator page opens.
• Select A specific namespace on the cluster under Installation mode.
• Select Operator Recommended Namespace under Installed Namespace.
• Choose Automatic under Update approval.
• Click Install.

• The Installing Operator pane appears. When the installation finishes, a checkmark appears next to the Operator name.

Verification

• In the OpenShift web console, from the side panel, navigate to Operators → Installed Operators and confirm that the cert-manager Operator for Red Hat OpenShift is in Succeeded state.
• Verify if the pods under cert-manager are in running state.

Secondary Scheduler Operator

Installation Steps:

• Log in to the OpenShift web console as a cluster administrator.
• In the left panel, navigate to Operators → OperatorHub.
• On the OperatorHub page, type Secondary Scheduler Operator for Red Hat OpenShift into Filter by Keyword box.

• Click on Secondary Scheduler Operator for Red Hat OpenShift.
• Select stable from Channel drop down and choose latest version available from Version drop down.

• Click Install. The Install Operator page opens.
• Select A specific namespace on the cluster under Installation mode.
• Select openshift-secondary-scheduler-operator from Installed Namespace drop down.
• Choose Automatic under Update approval.
• Click Install.

• The Installing Operator pane appears. When the installation finishes, a checkmark appears next to the Operator name.

Verification

• In the OpenShift web console, from the side panel, navigate to Operators → Installed Operators and confirm that the Secondary scheduler Operator for Red Hat OpenShift is in Succeeded state.

Install IBM Spyre Operator

• Log in to the OpenShift web console as a cluster administrator.
• In the left panel, navigate to Operators → OperatorHub.
• On the OperatorHub page, type IBM Spyre Operator into Filter by Keyword box.

• Click the IBM Spyre Operator tile. The IBM Spyre Operator information pane opens.
• Select stable from channel dropdown.
• Select 1.1.0 from version dropdown.

• Click Install. The Install Operator page opens.

• Select All namespaces on the cluster under Installation mode.
• Select Operator recommended Namespace from Installed Namespace drop down.
• Choose Automatic under Update approval.
• Click Install.
• The Installing Operator pane appears. When the installation finishes, a checkmark appears next to the Operator name.

Verification

• In the OpenShift web console, from the side panel, navigate to Operators → Installed Operators and confirm that the IBM Spyre operator shows one of the following statuses:
  • Installing: Installation is in progress; wait for this to change to Succeeded. This might take several minutes.
  • Succeeded: Installation is successful.
• Navigate to Workloads → Pods from side panel of OpenShift web console and verify if pods under spyre-operator namespace are in running state.

Create Spyre Cluster Policy

• Once the Operator is in Succeeded state, click on IBM Spyre Operator and navigate to Spyre Cluster Policy tab.

• Click on Create SpyreClusterPolicy option and paste the follwing YAML contents under yaml view.

• Click on Create and wait for spyreclusterpolicy to change to Ready state.

Verification

• Navigate to Administration → CustomResourceDefinitions on side panel of OpenShift web console.
• Search for SpyreNodeState in the search box and click on SpyreNodeState from the results.

• Navigate to Instances tab and click on the worker node on which the cards are mounted.

• Click on YAML tab and verify if cards are in healthy state under spec section of the YAML.

Install Red Hat OpenShift AI Operator

Refer Red Hat OpenShift AI operator installation for installing the operator. Proceed to Model Serving only when Red Hat OpenShift AI is successfully installed.

Model Serving

In this guide, We use granite-3.3-8b-instruct model for deployment using the vLLM Spyre s390x ServingRuntime for KServe in RawDeployment mode.

Prerequisites

This guide is based on the Red Hat OpenShift AI Operator version 3.0.0

Ensure that the operator is installed and running successfully before proceeding.

Resources

The following minimum resources are required for the model deployment of the granite-3.3-8b-instruct model.

Model Storage

• To download the model, visit https://huggingface.co/ibm-granite/granite-3.3-8b-instruct and clone the repo.
• Upload the cloned model to one of the supported storage backends in Red Hat OpenShift AI.
  • S3 - compatible object storage
  • URI - based repository
  • OCI - compliant registry
• This guide demonstrates accessing the model from an S3 - Compatible object storage backend.

Serving Runtime

Serving Runtime is a template that defines how a model server should run on KServe, including its container image, supported protocols, and runtime configuration.

You need to create a serving runtime with Red Hat AI inference server 3.2.5 as container image.

Procedure

• Click on Settings option from left panel on dashboard and expand Model resources and operations.

• Click on Serving runtimes and then on Add serving runtime.
• Select the API protocol as REST from API protocol drop down.
• Select Generative AI model from models type drop down.

• Click on Start from scratch in the YAML window and provide the following YAML contents

apiVersion: serving.kserve.io/v1alpha1

kind: ServingRuntime

metadata:

  annotations:

    opendatahub.io/recommended-accelerators: '["ibm.com/spyre_vf"]'

    opendatahub.io/runtime-version: v0.10.2.0

    openshift.io/display-name: vLLM Spyre s390x ServingRuntime for KServe - RHAIIS - 3.2.5

    opendatahub.io/apiProtocol: REST

  labels:

    opendatahub.io/dashboard: "true"

  name: vllm-spyre-s390x-runtime-copy

spec:

  annotations:

    opendatahub.io/kserve-runtime: vllm

    prometheus.io/path: /metrics

    prometheus.io/port: "8080"

  containers:

    - args:

        - --model=/mnt/models

        - --port=8000

        - --served-model-name={{.Name}}

      command:

        - /bin/bash

        - -c

        - source /etc/profile.d/ibm-aiu-setup.sh && exec python3 -m

          vllm.entrypoints.openai.api_server "$@"

        - --

      env:

        - name: HF_HOME

          value: /tmp/hf_home

        - name: FLEX_DEVICE

          value: VF

        - name: TOKENIZERS_PARALLELISM

          value: "false"

        - name: DTLOG_LEVEL

          value: error

        - name: TORCH_SENDNN_LOG

          value: CRITICAL

        - name: VLLM_SPYRE_USE_CB

          value: "1"

        - name: LD_PRELOAD           value: ""

      image: registry.redhat.io/rhaiis/vllm-spyre-rhel9:3.2.5-1765361213

      name: kserve-container

      ports:

        - containerPort: 8000

          protocol: TCP

      volumeMounts:

        - mountPath: /dev/shm

          name: shm

  multiModel: false

  supportedModelFormats:

    - autoSelect: true

      name: vLLM

  volumes:

    - emptyDir:

        medium: Memory

        sizeLimit: 2Gi

      name: shm

• Click on Create and verify if a serving runtime named vLLM Spyre s390x ServingRuntime for KServe - RHAIIS - 3.2.5 is listed under available serving runtimes.

Create a route to gateway to access Red Hat OpenShift AI Dashboard

OpenShift AI 3.0 uses a Gateway API and a dynamically provisioned load balancer service to expose its services. If you are deploying OpenShift AI 3.0 in private and on-premises environments, you must manually configure a route to access OpenShift AI Dashboard.

• Navigate to Networking → Routes in the OpenShift web console.
• Click on Create Route and go to the YAML view.
• Provide the below YAML contents, change the host as per your cluster details and click on create.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: data-science-gateway-data-science-gateway-class
  namespace: openshift-ingress
spec:
  host: data-science-gateway.apps.<CHANGEME>
  port:
    targetPort: https
  tls:
    termination: passthrough
  to:
    kind: Service
    name: data-science-gateway-data-science-gateway-class
    weight: 100
  wildcardPolicy: None

Deployment steps

Accessing Red Hat OpenShift AI Dashboard

• Login to OpenShift web console. Navigate to Operators → Installed Operators on the left panel.
• Verify that the Red Hat OpenShift AI Operator is installed and in the Succeeded state.
• Red Hat OpenShift Service Mesh 3 is a dependency operator and gets installed along with Red Hat OpenShift AI operator.
• Verify Red Hat OpenShift Service Mesh 3 is in succeeded state.
• Click the Application Launcher icon in the top right corner of the console.

• Click on the Red Hat OpenShift AI under OpenShift AI Self Managed Services to open the AI dashboard.
• Verify if the Dashboard is loaded.

Hardware profiles

The default profile available under Hardware Profiles limits vCPU to 2 and Memory to 4 GiB.

Create a new hardware profile to support granite-3.3-8b-instruct's minimum vCPU and memory config.

• Click on Settings option from left panel on dashboard and expand Environment Setup.

• Click on Hardware profiles and then on Create Hardware Profile.
• Provide a unique name and scroll down to edit the default and minimum allowed values.
• Set the default and minimum CPU value to 6
• Set the default and minimum Memory to 160 GiB.
• Set the maximum values of both CPU and Memory based on the resource availability of your OpenShift Cluster.

• Click on Add Resource to add Spyre Accelerator resource.

• Provide a unique name, Resource identifier as ibm.com/spyre_vf, Resource type as Accelerator, Default and maximum allowed to 4.
• Click Update.
• Scroll down to add a toleration to the deployment so that it can be scheduled onto the worker node that was previously tainted. 
• Click on Add toleration and provide the following values to add toleration.

• Click on Add and then on Create Hardware profile.

• Save the profile.

Deploy the model

• On the Dashboard, click on Projects on the left panel.
• Click Create Project, provide a name, and click Create. A project details page will appear with multiple tabs.

• Go to Connections tab.
• Click on Create connection and select S3 compatible object storage from the drop down.

• Enter the following details and click create.
  • Connection name
  • Access Key
  • Secret Key
  • Endpoint URL
  • Region
  • Bucket Name

• Go to Deploy tab.
  • Select Existing Connection under Model Location.
  • Choose the S3 connection created earlier.
  • Provide the path to the model in your S3 bucket.
  • Select Model type as Generative AI model from dropdown.

• • Click Next and give a unique Model Deployment name.
  • Under Hardware Profiles, select the profile you created earlier for granite 3.3 8b.
  • Select vLLM Spyre s390x ServingRuntime for KServe - RHAIIS - 3.2.5  created earlier from dropdown.
  • Set Model server replicas to 1.

• Click Next.
• Under Model Route, enable Make deployed models available through an external route to allow external access.
• For test environments, token authentication is optional.
• For production environments:
  • Select Require Token Authentication.
  • Enter the Service Account Name for token generation.
• Check Add custom runtime arguments check box and enter the following custom runtime arguments in the text box given below.

--max-model-len=32768

--max-num-seqs=32

--tensor-parallel-size=4

• Click on Deploy Model.

• Wait until the deployment reaches the Starting state.
• Once active, verify that the model deployment end points has been generated.

• • Hover over Internal and external endpoint under Inference endpoints and copy the external endpoint to proceed for Inferencing. You can use Internal endpoint to send an inferenece request with in the cluster.

Inferencing

Inference Request

After deployment, use the generated and external endpoint to send inference requests.

curl -k https://<external-endpoint>/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "<model-name>",
  "prompt": "<prompt>",
  "max_tokens": <max-tokens>,
  "temperature": 0
}' | jq

Example Request:

curl -k https://granite-8b-granite-8b.apps.ocpz-standard-spyre.b39-ocpai.pok.stglabs.ibm.com/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "granite-8b",
    "prompt": "What is London famous for?",
    "max_tokens": 300,
    "temperature": 0
  }' | jq