File and Object Storage

Software-defined storage for building a global AI, HPC and analytics data platform

View Only

Back to Blog List

IBM Storage Scale on Cloud Made Easy with cloudkit

By Santosh Kowshik posted Tue December 17, 2024 08:55 AM

Traditionally, the IT infrastructure has been on-premise. But cloud offers scalability, new hardware, GPU's, easier access to new marketplace offerings, so business needs best of both worlds. In this scenario, orchestrating workloads like bursting, backup etc, single name space spread across sites is difficult which storage scale deployment tool such as cloudkit aim to solve.

Vision of hybrid cloud for scale

The vision for a hybrid cloud with IBM Storage Scale focuses on delivering a seamless, scalable storage solution that blends the agility of cloud computing with the reliability of on-premises infrastructure.

This unified platform empowers organizations to optimize data management, achieve high performance, and enable workload portability, ensuring flexibility and efficiency.

By integrating cloud capabilities, IBM Storage Scale enhances collaboration, strengthens disaster recovery, and enables cost-effective scaling—meeting the evolving demands of modern enterprises while maintaining control over critical data.

Introduction to cloudkit

As you are already familiar with toolkit, which helps you deploy scale on-premise. cloudkit helps you deploy IBM Storage Scale on public cloud(s). Toolkit make use of Ansible for deploying and configuring the IBM Storage Scale configuration. Where as cloudkit take advantage of terraform to provision resources for IBM Storage Scale on public cloud(s). They are highly customisable and open-sourced in the below repositories.

An open source deployable IaC is available in the below repos:

toolkit: https://github.com/ibm/ibm-spectrum-scale-install-infra

cloudkit: https://github.com/ibm/ibm-spectrum-scale-cloud-install

cloudkit:

Note: Currently, Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure are supported for deployment on public clouds.

The cloudkit is a command line interface that provides automation of installation, deployment, and configuration of IBM Storage Scale on public clouds.

The CloudKit facilitates a wide range of tasks to simplify and enhance the deployment and management of IBM Storage Scale in cloud environments:

Rapid provisioning of IBM Storage Scale in either newly created or existing Virtual Private Cloud (VPC) environments.
Composable architecture of Virtual Machine Images for rapid deployment of IBM Storage Scale on public clouds.
Profile based deployment of diverse cluster configurations, including storage-only, compute-only, and combined compute-and-storage clusters.
Eases remote mount capabilities to mount filesystems on compute/CNSA clusters.
Provides scale-out capabilities for IBM Storage Scale clusters.
Simpler integration to tier data from on-premises to S3 object storage using AFM capabilities.

Cloudkit Architecture

cloudkit follows composable deployment architecture enables incremental enablement of IBM Storage Scale features on your production environments without distribution.

Fitting in the Infrastructure as Code(IaC) model into the existing deployment flow will give a robust and composable approach to install, configure & update Storage Scale on different cloud infrastructures.
The combination of cloudkit and toolkit enables to orchestrate the IBM Storage Scale on different cloud platforms.
Cloudkit IaC model enables us to create/manage the infrastructure required for scale on cloud.
cloudkit leverages toolkit for scale configuration, and sets optimal tuning parameters for scale based on opted cloud platform.

Deployment Profiles

Cloudkit offers the following deployment profiles:

1. Throughput-Performance-Persistent-Storage

This profile operates within a single availability zone and utilizes persistent storage, ensuring that the file system device retains data even after the instance is stopped. The number of storage nodes is dynamically determined by CloudKit based on the specified filesystem capacity, and the system employs gp3 as the disk type.

Note: When using this profile, the file system is configured to replicate only metadata across instances, while the data itself remains unreplicated.

2. Throughput-Performance-Scratch-Storage

This profile operates within a single availability zone and leverages a placement group with a cluster policy, ensuring instances are closely packed within the zone.

This configuration provides the low-latency network performance essential for tightly coupled node-to-node communication, a critical requirement for high-performance computing (HPC) applications. The setup uses instance or temporary storage, meaning the filesystem device does not retain data once the instance is stopped.

In this mode, storage instances are capped at 10, while compute instances are limited to 65.

Note: This profile employs local NVMe SSD or instance storage, delivering high performance and low latency for data-intensive workloads. However, the data is volatile and will be lost if the instance is stopped or terminated. Frequent backups are strongly recommended. This profile is unsuitable for long-term storage or cases where data is not backed up externally.

3. Throughput-Advance-Persistent-Storage

This profile operates within a single availability zone and utilizes persistent storage, ensuring the file system device retains data even after the instance is stopped. Designed primarily for storage capacity rather than performance, this mode supports up to 64 storage instances and 65 compute instances. The available disk types for this profile include gp2, gp3, and io1.

Note: When using this profile, the file system is configured to replicate only metadata across instances, while the actual data is not replicated.

4. Balanced

This profile deploys the IBM Storage Scale cluster across multi availability zones(3 Azs). In this mode, up to 64 storage instances are supported, with a choice of disk types including gp2, gp3, and io1.

Note: The IBM Storage Scale filesystem is configured to replicate both data and metadata across the availability zones, ensuring increased resilience and reliability.

For more information about IBM Storage Scale please visit this page.

#Highlights
#Highlights-home

0 comments

35 views

Permalink

https://community.ibm.com/community/user/blogs/santosh-kowshik/2024/11/25/ibm-storage-scale-on-cloud-with-cloudkit

File and Object Storage

File and Object Storage

IBM Storage Scale on Cloud Made Easy with cloudkit

By Santosh Kowshik posted Tue December 17, 2024 08:55 AM

Vision of hybrid cloud for scale