IBM Storage Protect and IBM Storage CEPH S3 ObjectLock

- Configuration Run book - 

Purpose. 2

Prepare storage on CEPH.. 2

Define user on Ceph Object Gateway. 3

Create a bucket with immutability enabled. 5

IBM Storage Protect Setup. 8

Get the CEP RGW Object gateway certificate(s) : 9

Import Ceph Certificate into the Storage Protect TrustStore: 10

Define a connection between Storage Protect and Ceph S3-Object Gateway. 11

Define a Cloud Storage Pool, with immutability enabled. 13

Configure a Policy to use this Storage Pool as a primary destination of the backup. 15

What you can and can’t do with an immutable cloud Object Storage pool 16

Technical appendix: How “immutability” and policy based expiration work together 17

References. 18

Purpose

In this article we explain who to setup IBM Storage CEPH as an IBM Storage Protect Server cloud container storage pool, with immutability.

This simple design helps securing your IBM Storage Protect backup in an immutable repository.

Prepare storage on CEPH

We assume here that the IBM Storage CEPH storage cluster is properly configured (https://www.ibm.com/docs/en/storage-ceph/8.0?topic=getting-started), and its S3 Gateway is properly configured (https://www.ibm.com/docs/en/storage-ceph/8.0?topic=storage-getting-started-object) .

This article covers only the configuration required for the IBM Storage Protect server to communicate and store data in an S3 bucket, enabled for immutability.

We will:

• Define a user on Ceph, dedicated to Storage Protect
• Create a bucket with immutability enabled

Define user on Ceph Object Gateway

From the Ceph User Interface, go under the Object > Users menu , and click on +Create button

Figure 1Ceph - Access to User creation

Specify a meaningful name to you and be sure that the S3 Key Auto-generate-key is ticked On, as shown in Figure 2

This is the S3 key associated with this user which will give you access through the S3 protocol to store and retrieve data.

Figure 2 Ceph - Create User

The userID julien and his associated S3 credential is being used for the rest of the setup.

Figure 3 CEPH - Julien S3 credential

Create a bucket with immutability enabled

Second step of our setup is to create a bucket, dedicated to Storage Protect, where it will store its data.

From the Ceph UI , access the Object > Buckets menu and click +Create button

Figure 4 CEPH - Create bucket

As shown in figure 4 , specify a meaningful name for this bucket and fill in the other options.

Owner is the user we just created : julien

Locking Enabled must be ticked On. This is the option to enable object locking mechanism. Choose among the 2 available modes: COMPLIANCE or GOVERNANCE

In GOVERNANCE mode, users cannot overwrite or delete an object version or alter its lock settings unless they have special permissions.

In COMPLIANCE mode, a protected object version cannot be overwritten or deleted by any user, including the root user in your AWS account. When an object is locked in COMPLIANCE mode, its RETENTION_MODE cannot be changed, and its retention period cannot be shortened. COMPLIANCE mode helps ensure that an object version cannot be overwritten or deleted for the duration of the period.

Locking > Number of Days the data is immutable: this is the number of days while CEPH will prevent data to be deleted. This value must be properly set and aligned with the IBM Storage Protect policy that is going to use this storage destination and the clouddatalockduration parameter being used in the storage pool definition.

Warning: if COMPLIANCE mode is selected, this value cannot be shortened.

Security > Encryption: in this setup, we rely on IBM Storage Protect encryption at storage pool level, hence we do not enable encryption here.

Other options can be left as default, or can be specified according to the Ceph administrator information.

Figure 5 Ceph - Exemple of Bucker creation 

The bucket we are going to configure in this document is named france-s3locked and has the following properties (owner julien, locking enabled/COMPLIANCE ).

Figure 6 CEPH - france-S3locked bucket detailed information 

IBM Storage Protect Setup

We assume here that the IBM Storage Protect Server is already installed and configured as per the best practices (https://www.ibm.com/support/pages/ibm-storage-protect-blueprints), and that Operation Center UI is running.

If Operation Center UI is not available, you can do the same using command lines.

All the command being used can be found in the official IBM Documentation: https://www.ibm.com/docs/en/storage-protect/8.1.25?topic=utilities-administrative-commands

The version we are using here is Storage Protect 8.1.24 running on Linux x86. Ceph (v 7.1) as an immutable object storage target is supported since version 8.1.18. (see support matrix here https://www.ibm.com/support/pages/ibm-spectrum-protect-object-storage-support)

Few steps to be able to use CEPH storage as a Cloud Storage Pool within Storage Protect.

We will:

• Get the TLS certificate from the Ceph S3 Object Storage Interface (Ceph Object Gateway)
• Import the TLS certificate into the Storage Protect TrustStore (required to enable TLS communication between Storage Protect & IBM Ceph Storage)
• Define a connection between Storage Protect and Ceph S3-Object Gateway
• Define a Cloud Storage Pool, with immutability enabled
• Configure a Policy to use this Storage Pool as a primary destination of the backup

Get the CEP RGW Object gateway certificate(s) :

To get the certificate, use the openssl command, from your Storage Protect server operating system.

You will get the certificate information and also - important - the CN information, be sure this is the FQDN information you use later in the connection definition inside Storage Protect .

[root@spta-jsa-sp1 bin]#openssl s_client -connect swat11.pbm.ihost.com:443 -showcerts

CONNECTED(00000003)

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1

verify return:1

depth=1 C = US, O = Let's Encrypt, CN = R10

verify return:1

depth=0 CN = s3.xxx.cephlabs.com

verify return:1

---

Certificate chain

 0 s:CN = s3.xxx.cephlabs.com

   i:C = US, O = Let's Encrypt, CN = R10

-----BEGIN CERTIFICATE-----

…

From the output of this command, locate the Certificate section(s) and copy the Certificate information to a .pem file(s).

Certificate information is the string chain enclosed by "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----"

Note that you may have more than one certificate as the IBM Storage Ceph is a clustered architecture, and the Object Storage Gateway is probably running on multiple nodes. Be sure you prepare the .pem file for each of them.

echo "-----BEGIN CERTIFICATE-----

YOUR KEY CONTENT HERE

-----END CERTIFICATE-----

" > /home/spectrum/cephrgw11-1.pem

Import Ceph Certificate into the Storage Protect TrustStore:

Connect to the IBM Storage Protect server machine using ssh, and use the keytool command installed by IBM Storage Protect server to import the certificate so Storage Protect Server can trust the connection with IBM Storage Ceph when we will define it.

keytool can be found under the /opt/tivoli/tsm/jre/bin folder.

Note: by default the password for the SP Server TrustStore is “changeit” , unless you changed it.

[root@spta-jsa-sp1 Documents]#cd /opt/tivoli/tsm/jre/bin/

[root@spta-jsa-sp1 bin]#./keytool -import -keystore ../lib/security/cacerts -alias cephswat11 -file /home/spectrum/cephrgw11-1.pem

Enter keystore password: changeit

Owner: CN=ceph-dashboard, O=IT

Issuer: CN=ceph-dashboard, O=IT

Serial number: abaca092cbf44e5ca910923c766a82cf

Valid from: 6/26/24 12:59 PM until: 6/24/34 12:59 PM

Certificate fingerprints:

         MD5:  E7:8A:3C:E6:79:B2:11:F1:72:D8:A9:FE:37:6C:E3:58

         SHA1: 06:1E:28:31:C8:C5:D9:B0:09:58:38:D3:E9:B9:1B:B5:A6:44:1C:D6

         SHA256: 1C:96:B3:6D:9F:8D:73:80:97:2B:A4:A1:30:50:7C:63:0C:39:3E:EB:42:6B:AD:5E:E6:5E:A7:56:EC:EF:FE:66

         Signature algorithm name: SHA512withRSA

         Version: 1

Trust this certificate? [no]:  yes

Certificate was added to keystore

Repeat this process for as many .pem files you prepared in the previous step. Be sure to change the -alias value you specified in the keytool command though.

You must restart your Storage Protect server once all the certificates are being added.

Note: Failing to properly import the certificates would cause this error message when trying to use or validate the connection to IBM Storage Ceph bucket.

01/07/2025 11:52:44     ANR3327E The IBM Storage Protect server cannot connect to the cloud service provider due to a Transport Layer Security (TLS) error. (SESSION: 13)

Define a connection between Storage Protect and Ceph S3-Object Gateway

Before created a cloud storage pool, you must define a connection to the object storage endpoint. In this case we point to Storage CEPH, using a HTTPS connection, see the certificate import process in the previous step.

Using the Storage Protect Operation Center UI , you can use the wizard available under  Storage > Cloud Connections menu , then click on the + Create Connection button.

Figure 7 SP - Create Cloud Connection

Once the connection is created, you can verify all is working well using the Test connection button. Doing this validation will ensure communication is successful using the cloudurl and the S3 access key ID and Secret ,  and access to the bucket is ok as well.

Figure 8 SP - Test Cloud Connection

Define a Cloud Storage Pool, with immutability enabled

Note: the definition of a Cloud Storage pool with immutability is not possible through the UI yet. This must be done using the command line.

Open the command line from the operation center as shown in figure 9 below.

Figure 9 SP - Open command line from UI

The command to create a Cloud storage pool using the connection we defined to CEPH is the following:

DEFINE STGPOOL CEPH-LOCKED STGTYPE=CLOUD CONNECTION=CEPH-S3-LOCKED ENABLECLOUDDATALock=Yes CLOUDREADCACHE=ON clouddatalockduration=1

CEPH-LOCKED is the name of that stgpool.

ENABLECLOUDDATALock = Yes this is how we are specifying to SP that the target storage is enable with immutability (S3 Object Lock).

clouddatalockduration=2 this value reflects the number of days specified on the CEPH bucket parameter Locking > Number of Days. This value should be higher than the retention value configured for the data which are going to be stored in this storage pool (aka management class > copy group > retention values). This is how we guarantee the data is immutable during its retention time – until it is due for expiration.

Notes:  

-          2 days here is for the example, this value is usually more than 7 days.

-          You can change it after the initial setup, in this case, be sure to change it also on Ceph bucket side, and note that only new containers being created inside the cloud container pool will get this lock duration. This is not retro-actively applying the change on existing containers.

CLOUDREADCACHE=ON is specified so we use a local cache before moving the data into the cloud object storage. This require the following command to actually configure the cache location.

DEFINE STGPOOLDIRECTORY CEPH-LOCKED /sp01/clouddatacache/ 

/sp01/clouddatacache is a local fast disks (SSD/Flash) that is used as stagging area before Storage Protect transfer the data on the object storage.

More details on how to size this cache and all performance related obersvations can be found here: https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Spectrum%20Protect%20Cloud%20Blueprint%20for%20IBM%20Cloud%20V2.0.pdf

Changing the clouddatalockduration is possible on Storage Protect cloud storage pool.

Using the Operation Center, go to Storage > Storage Pools , select the CEPH-LOCKED storage pool,  go in the Properties (left menu) and edit the Cloud section as show in figure below.

Be sure that you also reflect this change on the CEPH bucket . Remember these values must be aligned. Note that on CEPH side, the value might be increased only ( not decreased) depending on the lock mode you selected (Compliance vs Governance)

Figure 10 SP - Edit Cloud Lock Duration

Configure a Policy to use this Storage Pool as a primary destination of the backup

You are now ready to write and read data to and from the CEPH based Cloud storage Pool.

To do so, configure a copy group in an existing or new management class which points to that new storage pool CEPH-LOCKED.

This can be done using Operation Center, under the Services > Policy menu . Select your domain, then select an existing management class create a new one, and specify the CEP-LOCKED storage pool as a target for writing your data.
An example of this configuration is shown below, where the management class named VM7 , writes the data into the CEPH-LOCKED storage pool, with a 2 days retention.

Remember, we set 2 days in the clouddatalockduration as well as on the bucket locking settings. Thus, the data is going to be immutable during 2 days, until they expire.

Figure 11 SP - Configure Policies

Activate your change and you are now done !

Figure 12 SP - Activate policy

What you can and can’t do with an immutable cloud Object Storage pool

We covered the “backup” or “archive” to a Cloud Object Storage pool with immutability enabled, but this is not the only use case.

Cloud Storage pool with object lock enabled can be used as:

-          Primary target strage pool for backup & archive

-          Target of a Tiering Rule: moving data from a directory container to a cloud container (more information on Tiering here: https://www.ibm.com/docs/en/storage-protect/8.1.25?topic=storage-tiering-data )

What is NOT possible today (as of SP 8.1.25 ):

-          Storage Protect database backup to an S3 Cloud storage pool with object lock enabled

o   List of supported S3 target (non immutable): look for IBM Spectrum Protect Database Backup to Object Storage in that page

-          Retention Pool to an S3 Cloud storage pool with object lock enabled (https://www.ibm.com/docs/en/storage-protect/8.1.25?topic=storage-copying-retention-sets)

o   List of supported S3 target (non immutable): look for IBM Spectrum Protect Cloud Retention Storage Pools in that page

-          Doing space Reclamation on S3 Cloud storage pool when object lock is enabled  

-          Disabling enableclouddatalock option on the Storage Protect server side is not possible (nor disabling the S3 lock on the CEPH side.. ) .

Technical appendix: How “immutability” and policy based expiration work together

Policy & clouddatalockduration are independent

•        These are deduplicated containers being locked, not inventory objects

•        An inventory object can be set to X days, while the clouddatalockduration in the storage pool can be set to Y days

•        Best to set these intelligently based on what data you expect to store in the pool

•        There could be different policies in use, in the same pool

How data are being written/expired when using immutable object storage:

1.      Data is backed up to the SP server, assigned a policy and deduped chunks are staged in accelerator cache (which is a type of directory container, the STGPOOLDIRECTORY we specified when defining the cloud storage pool )

2.  Data chunks are uploaded as a ‘container’ to immutable the bucket

–      Lock is set based on “clouddatalockduration”

3.  SP will mark data as expired in the SP db when the SP policy expiration period is reached

4 . SP will check daily to see if S3 lock is about to expire

–      If SP is not ready to expire data, but S3 lock is about to expire, SP will extend the lock on COS container for another “clouddatalockduration”

5. Once all the data chunks in a container are marked as expired by SP:

•        If lock in S3 has also expired: SP will do normal container deletion

•        If S3 lock has not expired:  SP will put the container into a pending state – telling SP that the container is no longer needed but container is locked

•        Once S3 lock expires, SP will do normal container deletion

References

IBM Storage Protect supported Object Storage provider :

IBM Storage CEPH compatibility matrix : https://www.ibm.com/docs/en/storage-ceph/7.1?topic=compatibility-matrix#compatibility_matrix__backup_target

An introduction to the integration between IBM Storage Ceph and IBM Storage Protect can be found here: https://community.ibm.com/community/user/storage/blogs/daniel-alexander-parkes/2024/08/28/ibm-storage-ceph-object-storage-integration-with-i

FAQ on Cloud Container Storage Pools: https://www.ibm.com/docs/en/storage-protect/8.1.25?topic=cs-frequently-asked-questions-faqs-about-cloud-container-storage-pools#r_cloud_container_stgpool_faqs__section_tss_tvy_nbc

Certification of Immutability on IBM Storage Ceph :

Consulting company Cohasset concludes that IBM Storage Ceph, when properly configured and upon satisfying the additional considerations, meets the electronic record-keeping system requirements of SEC Rules 17a-4(f)(2), 18a-6(e)(2), and FINRA Rule 4511(c), as well as, supports the regulated entity in its compliance with the audit system requirements in SEC Rules 17a-4(f)(3)(iii) and 18a-6(e)(3)(iii). In addition, the assessed capabilities meet the principles-based electronic records requirements of CFTC Rule 1.31(c)-(d). See the Cohasset certification for more information.

Source: https://www.ibm.com/docs/en/storage-ceph/7.1?topic=lifecycle-enabling-object-lock-s3