Simplifying IBM Storage Ceph Object Deployments: Production-Ready Ceph Object Gateway with New Cephadm Features
Introduction
Deploying a production-ready object storage solution can be challenging, particularly when managing complex requirements such as SSL/TLS encryption, optimal data placement, and multisite replication. During deployment, it’s easy to overlook configuration options that become crucial once the system is live in production.
Traditionally, configuring IBM Storage Ceph for high availability, security, and efficient data handling required users to manually adjust several parameters based on their needs, such as Multisite Replication, Encryption, and High Availability. This initial complexity made it tedious to achieve a production-ready Object Storage configuration.
To tackle these challenges, we have introduced several new features to Ceph's orchestrator that simplify the deployment of IBM Storage Ceph Object and its associated services. Enhancing the Ceph Object Gateway and Ingress service specification files enables an out-of-the-box, production-ready RGW setup with just a few configuration steps. These enhancements include automated SSL/TLS configurations, virtual host bucket access support, erasure coding for cost-effective data storage, and more.
These improvements aim to provide administrators with a seamless deployment experience that ensures secure, scalable, and production-ready configurations for the Ceph Object Gateway and Ingress Service (load balancer).
In this blog, we'll explore each of these new features, discuss the problems they solve, and demonstrate how they can be easily configured using cephadm spec files to achieve a fully operational Ceph Object Gateway setup in minutes.
Feature Highlights
Virtual Host Bucket Access and Self-Signed Certificates
One of the major challenges in deploying RGW is ensuring seamless access to buckets using virtual host-style URLs. For applications and users that rely on virtual host bucket access, proper SSL/TLS certificates that include the necessary Subject Alternative Names (SANs) are crucial. To simplify this, we've added the option to automatically generate self-signed certificates for the Object Gateway if the user does not provide custom certificates. These self-signed certificates include SAN entries that allow TLS/SSL to work seamlessly with virtual host bucket access.
Complete TLS/SSL Client-to-RGW Encryption
Security is a top priority for any production-grade deployment, and customers have increasingly requested full TLS/SSL encryption from the client to the Object Gateway service. Previously, our ingress implementation only supported terminating SSL at the HAProxy level, which meant that communication between HAProxy and RGW could not be encrypted.
To address this, we've added configurable options that allow users to choose whether to re-encrypt traffic between HAProxy and RGW or to use passthrough mode, where the TLS connection remains intact from the client to RGW. This flexibility allows users to achieve complete end-to-end encryption, ensuring sensitive data is always protected in transit.
Multisite replication configuration
In the past, IBM Storage Ceph Multisite deployments involved running a good amount of commands to configure your Realm, zonegroup, and zone, and also establishing the relationship between the zones that will be involved in the Multisite replication, thanks to the RGW manager module, the multisite bootstrap and configuration can be down in two steps, if interested there is an example in this Object Storage Replication blog series.
In the 8.0 release, we have also added the possibility of configuring dedicating Object Gateways just for client traffic purposes through the cephadm spec file with the RGW spec file option:
The advantages of dedicating Ceph Object Gateways to specific tasks are covered here.
Configure your Erasure coded data pool in the Spec file during bootstrap.
Object Storage generally uses an EC profile for the data pool to reduce the TCO of your object storage solution; we have included options for configuring erasure-coded (EC) pools in the spec file. This allows users to define the EC profile, device class, and failure domain for RGW data pools, which provides more control over data placement and storage efficiency.
IBM Storage Ceph Object deployment walkthrough
If you are new to IBM Storage Orchestration and Cephadm, this blog entry will give you a good overview of Cephadm and how we can define the state of our Ceph services in a declarative yaml spec file to deploy and configure our IBM Storage Ceph Cluster.
Below, we'll walk through the CLI commands required to deploy a production-ready RGW setup using the new features added to the cephadm orchestrator.
Enabling the RGW Manager Module
The first step is to enable the RGW manager module. This module is required to manage RGW services through cephadm.
# ceph mgr module enable rgw
Creating the RGW Service Spec File
Next, we'll create a spec file for the Object Gateway service. This spec file includes realm, zone, and zonegroup settings, SSL/TLS, Erasure coding profile for the data pool, etc.
# cat << EOF > /root/rgw-client.spec
service_type: rgw
service_id: client
service_name: rgw.client
placement:
label: rgw
count_per_host: 1
networks:
- 192.168.122.0/24
spec:
rgw_frontend_port: 4443
rgw_realm: multisite
rgw_zone: zone1
rgw_zonegroup: multizg
generate_cert: true
ssl: true
zonegroup_hostnames:
- s3.cephlab.com
data_pool_attributes:
type: ec
k: 2
m: 2
extra_container_args:
- "--stop-timeout=120"
config:
rgw_exit_timeout_secs: "120"
rgw_graceful_stop: true
EOF
In this spec file, we are specifying that the RGW service should use erasure coding with a 2+2 profile (k: 2, m: 2) in the data pool, which helps reduce storage costs compared to a replicated setup. We're also generating a self-signed certificate (generate_cert: true) For the Object gateway service to ensure secure SSL/TLS communication. With zonegroup_hostnames, we are setting virtual host bucket access using the specified domain with bucket.s3.cephlab.com. Thanks to the config parameter rgw_gracefull_stop, we are configuring the graceful stops of the Object gateway services. During a graceful stop, the service will wait until all client connections are closed to the specified 120-second timeout.
Bootstrapping the RGW Realm
Once the spec file is created, we can bootstrap the RGW services. This step will create and deploy the RGW services with the configuration specified in our spec file.
# ceph rgw realm bootstrap -i rgw-client.spec
Verifying the RGW Services
The cephadm bootstrap command will asynchronously apply the configuration defined in our spec file. After a minute, the RGW services will be up and running; we can verify their status using the ceph orch ps command.
# ceph orch ps --daemon_type rgw
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
rgw.client.ceph-node-05.yquamf ceph-node-05.cephlab.com 192.168.122.175:4443 running (32m) 94s ago 32m 91.2M - 19.2.0-53.el9cp fda78a7e8502 a0c39856ddd8
rgw.client.ceph-node-06.zfsutg ceph-node-06.cephlab.com 192.168.122.214:4443 running (32m) 94s ago 32m 92.9M - 19.2.0-53.el9cp fda78a7e8502 82c21d350cb7
This output shows that the RGW services run on the specified nodes and are accessible via the configured 4443 port.
Verifying the Data Pool
To verify that the RGW data pools are correctly configured with erasure coding, we can use the following command:
# ceph osd pool ls detail | grep data
pool 24 'zone1.rgw.buckets.data' erasure profile zone1_zone_data_pool_ec_profile size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 258 lfor 0/0/256 flags hashpspool stripe_width 8192 application rgw
Viewing the Erasure Code Profile
To get more details about the erasure code profile used for the data pool, we can run:
# ceph osd erasure-code-profile get zone1_zone_data_pool_ec_profile
crush-device-class=
crush-failure-domain=host
crush-num-failure-domains=0
crush-osds-per-failure-domain=0
crush-root=default
jerasure-per-chunk-alignment=false
k=2
m=2
plugin=jerasure
technique=reed_sol_van
w=8
This confirms that the erasure code profile is configured with k=2 and m=2, using the Reed-Solomon technique.
Configuring the Ingress Service
Finally, we must configure the ingress service to load balance traffic to the RGW. Again, we create a spec file for the ingress service:
# cat << EOF > rgw-ingress.yaml
service_type: ingress
service_id: rgw
placement:
hosts:
- ceph-node-06.cephlab.com
- ceph-node-07.cephlab.com
spec:
backend_service: rgw.client
virtual_ip: 192.168.122.152/24
frontend_port: 443
monitor_port: 1967
use_tcp_mode_over_rgw: True
EOF
This spec file sets up the ingress service with a virtual IP: 192.168.122.152 and specifies that it should use TCP mode for communication with the Object Gateway, ensuring that SSL/TLS is maintained throughout, with the backend_service we specify what RGW service we want to use as the backend for HAproxy.
Testing Load Balancer and SSL/TLS Configuration
Our ingress service stack is based on Keepalived for the HA of the VIP, and Haproxy takes care of the load balancing:
# ceph orch ps --service_name ingress.rgw
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
haproxy.rgw.ceph-node-06.vooxuh ceph-node-06.cephlab.com *:443,1967 running (58s) 46s ago 58s 5477k - 2.4.22-f8e3218 0d25561e922f 4cd458e1f6b0
haproxy.rgw.ceph-node-07.krdmsb ceph-node-07.cephlab.com *:443,1967 running (56s) 46s ago 56s 5473k - 2.4.22-f8e3218 0d25561e922f 4d18247e7615
keepalived.rgw.ceph-node-06.cwraia ceph-node-06.cephlab.com running (55s) 46s ago 55s 1602k - 2.2.8 6926947c161f 50fd6cf57187
keepalived.rgw.ceph-node-07.svljiw ceph-node-07.cephlab.com running (53s) 46s ago 53s 1598k - 2.2.8 6926947c161f aaab5d79ffdd
If we check the haproxy configuration on ceph-node-06 Where the service is running, we will see that we are using TCP passthrough for the backend configuration of our Object Gateway services.
# ssh ceph-node-06.cephlab.com cat /var/lib/ceph/93d766b0-ae6f-11ef-a800-525400ac92a7/haproxy.rgw.ceph-node-06.vooxuh/haproxy/haproxy.cfg | grep -A 10 "frontend frontend"
...
backend backend
mode tcp
balance roundrobin
option ssl-hello-chk
server rgw.client.ceph-node-05.yquamf 192.168.122.175:4443 check weight 100 inter 2s
server rgw.client.ceph-node-06.zfsutg 192.168.122.214:4443 check weight 100 inter 2s
To verify that the SSL/TLS configuration is working correctly, we can use curl to test the endpoint, we can see that the CA is not trusted in our client system where we are running the curl command:
# curl https://192.168.122.152
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.
To fix this, we need to add the cephadm root CA certificate to the trusted store of our client system:
# ceph orch cert-store get cert cephadm_root_ca_cert > /etc/pki/ca-trust/source/anchors/cephadm-root-ca.crt
# update-ca-trust
After updating the trusted store, we can test again:
# curl https://s3.cephlab.com
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID></Owner><Buckets></Buckets></ListAllMyBucketsResult>
This confirms that the SSL/TLS self-signed certificate configuration works correctly and that the RGW service is accessible using HTTPS. As you can see, we have configured our DNS subdomain s3.cephlab.com and the wildcard *.s3.cephlab.com to point to our VIP address: 192.168.122.152. Also, it's important to mention that you can have more than one VIP address configured so not all the traffic goes through a single haproxy LB node; when using a list of VIP IPs, you need to use the option: virtual_ips_list
Conclusion
These new features in the cephadm orchestrator represents significant steps forward in making Ceph RGW deployments more accessible, secure, and production-ready. By automating complex configurations—such as SSL/TLS encryption, virtual host bucket access, multisite replication, and erasure coding—administrators can now deploy an RGW setup ready for production with minimal manual intervention.
#Featured-area-2