This blog explains how to integrate IBM Storage Protect (formerly Spectrum Protect) with Amazon Web Services (AWS) S3 using Cloud Container Storage Pools.

Architecture Overview

IBM Storage Protect uses AWS S3 as a cloud container to store backup data securely and cost-effectively. The configuration requires setting up an S3 bucket, IAM user, and appropriate policies to allow read/write/delete operations from AWS end.

Steps to perform at AWS end:

Step 1: Create an S3 Bucket (AWS Management Console)

Log in to the AWS Management Console and create an S3 bucket. Choose the appropriate region and specify a unique bucket name (S3 → Buckets → Create Bucket).

Step 2: Create IAM User & Attach Policy (AWS Management Console)

Create an IAM user and assign S3 policy as this user will be used by IBM Storage Protect to access AWS S3. (IAM→Users→Create User)

Step 3: Define IAM Policy (AWS Management Console)

We need the following permissions to configure the AWS cloud. During the Validate command, apart from verifying the connection to the URL, the system also performs write, read, and delete operations on a test object to ensure that basic operations are working properly

Note: We can generate Customer Managed Policies using AWS Policy Generator

Sample Json Script

Attach Policy

Step 4: Generate Access Keys (AWS Management Console)

Generate Access Key ID and Secret Access Key for the IAM user. These credentials will be configured in IBM Storage Protect. 

Steps to perform on IBM Storage Protect Server:

Step 5: Define and Validate the Cloud Connection in IBM Storage Protect

To configure the cloud connection, provide the following details: Cloud URL (https://s3.amazonaws.com), Bucket Name, Access Key ID, and Secret Access Key. Then validate the connection to ensure successful read, write, and delete operations.

Note: Amazon S3 mainly supports 2 types of URLs to access objects:

Here is an example of a path style URL:

https://s3.region-code.amazonaws.com/bucket-name/key-name

Starting with 8.1.18, IBM Storage Protect supports the use of AWS Hosted Style URLs. Here is an example:

https://bucket-name.s3.region-code.amazonaws.com/key-name

Step 6: Define Cloud Storage Pool for client backups:

Create a cloud storage pool and select the appropriate AWS storage class such as Standard, Intelligent-Tiering, or Glacier Instant Retrieval based on cost and access requirements.

Step 7: Define the Cloud Storage Pool Directory for Cloudcache:

Bucket with Object lock (WORM):

The basic steps remain the same as those followed for a normal bucket, with a few changes that include bucket creation, defining bucket policies, and configuring storage pools.

A)    Create an S3 Bucket (AWS Management Console)

While creating an Object Lock bucket, you must update the retention mode and retention period as shown below (S3 → Buckets → Bucket name → Edit Object Lock).

Note: Once Object Lock is enabled, it cannot be disabled after creation.

Retention modes:

Governance: Users with specific IAM permissions can overwrite or delete protected object versions during the retention period.

Compliance: No users can overwrite or delete protected object versions during the retention period.

B)   IAM POLICY( IBM Storage Protect administrative interface)

An Object Lock bucket requires elevated privileges compared to a standard bucket, as it enables the management of immutability (WORM) and versioning features

C) Defining Cloud storage pool (IBM Storage Protect administrative interface)

          When defining a cloud storage pool for Object Lock storage, the following parameters must be configured

·       ENABLECLOUDDATALOCK: Specifies whether the data in the cloud storage pool is locked.

·       CLOUDDATALOCKDURATION: Specifies the number of days the server retains the data.

If ENABLECLOUDDATALOCK is set to Yes, it cannot be changed back to No. For production systems, CLOUDDATALOCKDURATION must be set to a minimum of 7 days, and the server verifies that the cloud data lock settings are synchronized between the storage pool and the cloud bucket If they are not synchronized, you will see the following warning:

ANR3816W – The default retention settings for bucket <Name> are not set or do not match the cloud lock duration settings for the storage pool.

Troubleshooting:

Common issues include permission mismatches, invalid access keys, and certificate errors. Ensure that the bucket’s retention settings match the cloudlockduration value, always synchronize server time with AWS, and validate policies carefully. Here are few example error:

Issue1: ANR3327E

Validate cloud command fail with following error:

ANR3327E The IBM Storage Protect server cannot connect to the cloud service provider due to a Transport Layer Security (TLS) error.

Dsmffdc.log reports:

[ FFDC_GENERAL_SERVER_ERROR ]: [984](jvm.c:1906)(SESSION: 12) ANR0161I Displaying segment 1 for message insert [ID1:03:03:36.730] with  text com.tivoli.dsm.cloud.api.CloudHandler verifyCredentials Exception during verifyCredentials:javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.j: PKIX path building failed: com.ibm.security.cert.

IBMCertPathBuilderException: unable to find valid certification path to requested target

Solution:

https://www.ibm.com/support/pages/node/3241809#self%20signed( Refer : How can I set up self-signed certificates for my object storage system?)

Issue2: ANR4140E

Validate cloud command fail with following error:

ANR4140E The specified credentials failed verification because they do not have permission for list object operations on the bucket or vault.

Dsmffdc.log reports:

[ FFDC_GENERAL_SERVER_ERROR ]: [19](jvm.c:1906) com.tivoli.dsm.cloud.api.ProviderS3Retry

handleException com.amazonaws.services.s3.model.AmazonS3Exception Access Denied

(Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: Q0AZPX600NC227Y5; S3 Extended Request ID: K1sxMKxAmFDVn6rMxIf4S61NeLGFfeTw11tWWUHcfp0dy5lbBeUx7q86N9AVhxGWdPCFXztkwuQ=; Proxy: null) HTTP Status Code: 403

Solution: https://www.ibm.com/support/pages/node/7261501  

Issue3: Restore slowness

Solution: To improve restore performance in cloud-container storage pools, it is advisable to enable CLOUDREADCACHE, which caches restored data locally before client delivery.

Ex: update stgpool <cloud_stgpool> cloudreadcache=yes

FAQ:

https://www.ibm.com/docs/en/storage-protect/8.2.0?topic=cs-frequently-asked-questions-faqs-about-cloud-container-storage-pools

Contributors: Rama Krishna Deevela, Bharat Vyas
Acknowledgment: Special thanks to Ramchandra Sonawane for reviewing this blog