IBM Orchestration Pipelines service is now supported on Linux on IBM Z and LinuxONE (s390x) with the IBM Software Hub Version 5.2.1
Introduction
IBM Orchestration Pipelines is now supported on Linux on IBM Z and LinuxONE platform. This blog helps users understand how to manage and automate their data and flow of AI assets on Linux on IBM Z and LinuxONE by creating, deploying, and managing end-to-end data integration pipelines and AI/ML workflows to generate automated, consistent, and repeatable outcomes.
Capabilities
With the 5.2.1 release, the Orchestration Pipelines service is now fully supported on the Linux on IBM Z and LinuxONE platform. This enhancement allows you to seamlessly create, modify, test, and execute end-to-end data and AI workflows directly on Linux on IBM Z and LinuxONE by leveraging the platform’s strong performance and reliability.
This service now brings support for Orchestration Pipelines on IBM Z and LinuxONE, expanding the range of supported architectures and enhancing cross-platform compatibility. It also enables optimization of hybrid cloud infrastructure by leveraging platform-specific benefits while maintaining the use of trusted IBM services.
Prerequisites
This blog assumes that users are already familiar with:
- Deploying and running other services such as IBM AutoAI, IBM DataStage, and Red Hat OpenShift on Linux on IBM Z and LinuxONE.
- Using the Orchestration Pipelines service to assemble and configure a pipeline.
Environment
The Orchestration Pipelines service on Linux on IBM Z and LinuxONE has been validated in the following environments:
- IBM Software Hub version 5.2.1
- Supported platform: Linux on IBM Z and LinuxONE
- Deployment target: Red Hat OpenShift 4.12 or later on Linux on IBM Z and LinuxONE
- Hypervisor: z/VM or KVM or Bare Metal Linux
Orchestration Pipelines on Linux on IBM Z and LinuxONE
Orchestration Pipelines can be used to create, modify, test, and run an end-to-end flow to achieve the following goals:
- Ingest and process data securely from a variety of internal and external sources.
- Achieve desired outcomes by cohesively building, executing, evaluating, and deploying models or by running scripts.
- Simplify the flow execution by creating branches and using visual tools to gather and interpret results.
You can design a pipeline by dragging nodes onto the canvas, specifying objects and parameters, then running and monitoring the pipeline. Pipelines can run activities including but not limited to:
- AutoAI experiments
- Jupiter Notebook jobs
- Data Refinery jobs
- IBM DataStage jobs (also known as sequence jobs)
- End to End ML pipelines
This service can also be used to automate the process of running tasks to ingest, transform, and save data using IBM DataStage. With the support for Linux on IBM Z and LinuxONE, this unlocks several important possibilities:
- It enables a consistent data engineering experience across multiple architectures.
- With Linux on IBM Z and LinuxONE, organizations can leverage their existing infrastructure and skills to advance AI and analytics projects.
- This enhancement enables tighter integration with other IBM Software Hub services that are already supported on Linux on IBM Z and LinuxONE systems.
- Traditional DataStage sequence jobs can be seamlessly migrated to Orchestration Pipelines, allowing them to be managed and visualized alongside newer, cloud-native data processes.
As part of Cloud Pak for Data 5.2.1 release, the Orchestration Pipelines service on IBM Z is functionally equivalent to x86. There are no changes required in how you deploy or manage the service on Linux on IBM Z and LinuxONE. If you're already familiar with enabling IBM Software Hub services on x86, the process is identical for Linux on IBM Z and LinuxONE.
Installation process
To enable the Orchestration Pipelines service on a Z cluster:
1. Install or upgrade to IBM Software Hub version 5.2.1 on your OpenShift cluster running on Linux on IBM Z and LinuxONE.
2. Navigate to the IBM Software Hub web console, go to the Services/Services catalog, and enable Orchestration Pipelines.
3. Follow the standard installation steps, referring to the IBM Software Hub version 5.2.x documentation and use installation flows, which are now validated for Linux on IBM Z and LinuxONE.
For installing Orchestration Pipelines on IBM® Software Hub version 5.2, refer to https://www.ibm.com/docs/en/software-hub/5.2.x?topic=pipelines-installing
Image and registry support
Orchestration Pipelines images are now available for the s390x architecture and are automatically retrieved by the Orchestration Pipelines operator during installation, eliminating the need for manual handling of architecture-specific image sources.
Architecture advantage
IBM Z is designed for data and I/O intensive workloads. With High memory bandwidth, simultaneous multithreading (SMT), High Availability, Confidential AI, Security, Reliability, Sustainability and Higher TCO (Total Cost of Ownership) with consolidation of workloads. Traditionally IBM Z offers a compelling platform for running long-lived, high-throughput extract, transform, and load (ETL) jobs and how Linux on IBM Z and LinuxONE customers can leverage the benefits of pipelines in their development, test and production environments.
Summary
With the 5.2.1 release of Cloud Pak for Data, the Orchestration Pipelines service is now available on IBM Z (s390x), further enhancing IBM’s multi-architecture support. This addition enables seamless deployment and workload portability between x86 and Linux on IBM Z and LinuxONE platforms, simplifying operations and supporting the modernization of data and ML/AI environments.
By running Orchestration Pipelines on Linux on IBM Z and LinuxONE, you can:
- Leverage your existing Z infrastructure.
- Manage and automate your data and AI lifecycle in s390x architecture.
- Maintain architectural consistency across hybrid and multi-cloud environments.
Call to action (CTA)
Contacts :
Reach out to the following contacts for any assistance.
· Chowdappa R – Senior Engineer – Repana.Chowdappa@ibm.com
· Ravi Gummadi – Technical Architect – ravigumm@in.ibm.com
· Srirama Sharma – STSM, Chief Architect - sriramsh@in.ibm.com
· Srinath Kanukolanu – Offering Manager/Product Manager, IBM Z Cloud Paks – Srinath.kanukolanu@in.ibm.com
· Kavitha Deekaram – Cloud paks on Z Manager – kdeekara@in.ibm.com