Automating Ceph Cluster Deployments with IBM Storage Ceph: A Step-by-Step Guide Using Cephadm and Ansible (Part 1)
Introduction
In the era of big data, managing vast amounts of storage efficiently and reliably is a critical challenge for enterprises. Ceph has become a leading software-defined storage solution known for its flexibility, scalability, and robustness. Building on this foundation, IBM Storage Ceph elevates these capabilities, offering seamless integration with enterprise environments and advanced tools for efficiently managing petabytes of data.
This blog post series will delve into the automated deployment of Ceph clusters using IBM Storage Ceph's state-of-the-art orchestrator, Cephadm. Additionally, for those automating their infrastructure with Ansible, we will share an example using an Infrastracture-As-Code approach with the help of Jinja2 templates and Ansible.
Infrastructure as Code
Infrastructure as Code (IaC) revolutionizes infrastructure management by treating infrastructure setups as code. This allows us to apply software development practices such as version control, testing, and continuous integration to infrastructure management, reducing the risk of errors and speeding up deployment and scaling.
With Ceph, tools like Ansible and Cephadm are perfect examples of IaC in action. They allow administrators to define the desired state of their Ceph clusters in code, making it easier to deploy, manage, and scale these clusters across different environments.
A Brief History of RH/IBM Ceph Orchestration Tools
As Ceph became more popular and clusters rapidly grew, the need for an effective orchestration tool became increasingly critical. Over the years, several tools have been developed to simplify and automate the deployment and management of Ceph clusters. Let’s take a brief look at them:
-
Ceph-deploy was one of the first tools introduced to ease the deployment of Ceph clusters. As a lightweight command-line utility, ceph-deploy allowed administrators to quickly set up a basic Ceph cluster by automating many manual steps in configuring Ceph daemons like MONs, OSDs, and MGRs.
-
Ceph-ansible marked a significant step forward by integrating Ceph deployment with Ansible, a popular open-source automation tool. This approach embraced the principles of Infrastructure as Code (IaC), allowing administrators to define the entire Ceph cluster configuration in Ansible playbooks.
Cephadm Introduction
Unlike its predecessors, Cephadm deploys all Ceph daemons as containers using Docker or Podman. This containerized approach ensures consistency across different environments and simplifies the management of dependencies, making deploying, upgrading, and scaling Ceph clusters easier.
Cephadm's use of a declarative spec file to define the cluster's desired state marks a significant improvement in how Ceph clusters are managed. Administrators can now describe their entire cluster configuration upfront, and Cephadm continuously ensures that the cluster matches this desired state.
In addition to its powerful deployment capabilities, Cephadm integrates with the Ceph Dashboard, provides built-in monitoring and alerting, and supports automated upgrades, making it the most comprehensive and user-friendly orchestration tool in the Ceph ecosystem to date.
Repetitive Automated Deployment of IBM Storage Ceph Clusters with Cephadm
Modern IT environments increasingly require repeatedly deploying and scaling storage clusters across different environments: development, testing, and production. This is where Cephadm comes to the rescue. By automating the deployment and management of Ceph clusters, Cephadm eliminates the manual, error-prone processes traditionally involved in setting up distributed storage systems.
Cephadm’s Declarative Approach
Cephadm’s use of a declarative service spec file allows administrators to define the entire configuration of a Ceph cluster in a single, reusable file. This spec file can describe everything from the placement of OSDs, MONs, and MGRs to the setup and configuration of File, Block, and Object Services. By applying this spec file, Cephadm can automatically deploy the cluster to match the desired state, ensuring consistency across multiple deployments.
It’s important to note that Cephadm provides the deployment and lifecycle of the Cluster services. Still, not all day two operations of specific services, like creating a Ceph Object Storage user, are currently covered by Cephadm.
Fitting into Infrastructure as Code (IaC)
Cephadm fits perfectly into an Infrastructure as Code (IaC) paradigm. IaC treats infrastructure configurations like software code—storing them in version control, automating their application, and enabling continuous delivery pipelines. With Cephadm, the spec file acts as the "code" that defines your storage infrastructure.
For example, you could store your Cephadm spec files in a version control system like Git. When changes are made to the cluster configuration, they are committed and pushed, triggering automated pipelines that deploy or update the Ceph cluster based on the updated spec file. This approach streamlines deployments and ensures that your storage infrastructure is always in sync with your application and service needs.
Note that specific Cephadm configuration changes require restarting the corresponding service, which must be coordinated with an automation tool for the changes to take effect once applied.
Walkthrough of a Cephadm Service Spec File that Provides an Automated Deployment of a Ceph Cluster
Below is an example of a Cephadm spec file that enables a complete Ceph cluster deployment during the bootstrap process. This basic example is designed to get you started; a production deployment would require further customization of the spec file.
service_type: host
hostname: ceph1
addr: 10.10.0.2
location:
root: default
datacenter: DC1
labels:
- osd
- mon
- mgr
---
service_type: host
hostname: ceph2
addr: 10.10.0.3
location:
datacenter: DC1
labels:
- osd
- mon
- rgw
---
service_type: host
hostname: ceph3
addr: 10.10.0.4
location:
datacenter: DC1
labels:
- osd
- mds
- mon
---
service_type: mon
placement:
label: "mon"
---
service_type: mgr
service_name: mgr
placement:
label: "mgr"
---
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
spec:
data_devices:
all: true
limit: 1
placement:
label: "osd"
---
service_type: rgw
service_id: objectgw
service_name: rgw.objectgw
placement:
count: 2
label: "rgw"
spec:
rgw_frontend_port: 8080
rgw_frontend_extra_args:
- "tcp_nodelay=1"
---
service_type: ingress
service_id: rgw.external-traffic
placement:
label: "rgw"
spec:
backend_service: rgw.objectgw
virtual_ips_list:
- 172.18.8.191/24
- 172.18.8.192/24
frontend_port: 8080
monitor_port: 1967
The first `service_type` is `host`, which specifies the list of hosts in the cluster, including their hostnames and IP addresses. The `location` field indicates the host's position within the Ceph Crushmap Tree, a hierarchical structure that optimizes data placement and retrieval across the cluster. Check out this link for more info.
By setting specific labels on the host, Cephadm can efficiently schedule and deploy containerized Ceph services on the same nodes. This ensures resource isolation and reduces the number of nodes required, optimizing resource usage and cutting costs in production environments.
Note that the hosts we are adding to the cluster need to have RHEL and a series of prerequisites configured to successfully join the IBM Storage Ceph Cluster. This blog series will also cover automating the prerequisites.
service_type: host
hostname: ceph1
addr: 10.10.0.2
location:
root: default
datacenter: DC1
labels:
- osd
- mon
- mgr
After that, we have the MON and MGR service deployment; we have a simple configuration for the mon and the manager; we only use the placement parameter. With the placement parameter, we are telling Cephadm that it can deploy the mon service on any host with the “mon” label.
---
service_type: mon
placement:
label: "mon"
---
service_type: mgr
service_name: mgr
placement:
label: "mgr"
Next, we have the osd service type; the Cephadm OSD service type is incredibly flexible; it allows you to do almost any OSD configuration you can imagine; for full details on the OSD Service spec, check out this link
In our example, we are going for one of the most straightforward approaches possible; we are telling Cephadm to use as OSDs all free/usable media devices that are available on the nodes, again using the placement parameter, it will only configure OSD devices on nodes that have the “osd” label.
---
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
spec:
data_devices:
all: true
placement:
label: "osd"
In this final section, we configure the Ceph services we want to enable on the cluster, MDS. The metadata service allows us to use IBM Storage Ceph Shared file system “CephFS”. For all available MDS service spec configuration options, check out this link.
Finally, we have the rgw service type to set up IBM Storage Ceph Object Storage Services. The RGW services will provide an S3 and Swift HTTP restful endpoint for users to connect; in this example in the placement section, we are setting the count of RGW services to “2”, this means that the Cephadm scheduler will look to schedule two RGW daemons on available hosts that have the “rgw” label set.
---
service_type: mds
service_id: cephfs
Placement:
count: 2
label: "mds"
---
service_type: rgw
service_id: objectgw
service_name: rgw.objectgw
placement:
count: 2
label: "rgw"
rgw_realm: {{ rgw_realm }}
rgw_zone: {{ rgw_zone }}
rgw_zonegroup: {{ rgw_zonegroup }}
spec:
rgw_frontend_port: 8080
rgw_frontend_extra_args:
- "tcp_nodelay=1"
In IBM Storage Ceph, we also provide an out-of-the-box load balancer based on haproxy and keepalived called the ingress service. In this example, we are using the ingress service to balance the client S3 requests between the different RGW services we have running in our cluster, providing the Object service with HA and Load Balancing. Detailed information is on this link.
We use the rgw label service to co-locate the haproxy/keepalived daemons with the RGW services. We then just set a list of VIPs the clients will use to access the S3 endpoint API with the virtual_ips_list spec parameter.
---
service_type: ingress
service_id: rgw.external-traffic
placement:
label: "rgw"
spec:
backend_service: rgw.objectgw
virtual_ips_list:
172.18.8.191/24
172.18.8.192/24
frontend_port: 8080
monitor_port: 1967
Once we have all the services spec defined and ready, we need to pass the Spec file to the Cephadm bootstrap command to get our cluster deployed and configured as we have described in our file, an example of the bootstrap command using the “--apply-spec” parameter to pass our cluster specification file:
# cephadm bootstrap \
--registry-json /root/registry.json \
--dashboard-password-noupdate \
--ssh-user=cephadm \
--mon-ip {{ admin_host_ip_addr }} \
--apply-spec /root/cluster-spec.yaml
Next Steps
In the next installment of this series, we’ll explore how to leverage Jinja2 templating and Ansible in tandem with Cephadm service specification files. This approach will demonstrate how to build an Infrastructure as Code (IaC) framework for Ceph cluster deployments, facilitating a streamlined Continuous Delivery pipeline with Git as the single source of truth for IBM Storage Ceph configurations management.