Authors: @RAVI KUMAR KOMANDURI @Anandhu Karattuparambil

In the era of AI Data & Analytics workloads, the Object access is a key differentiator for any product in the industry. This Object access, would enhance the Research & Academia, Retail, Financial, Telecom sectors etc etc to analyze their businesses and aid in coming up with innovative strategies for enhancing their end users experience. IBM Storage Scale Product[1] (aka Spectrum scale as previously called), coined a feature, known as "S3 protocol"[2], designed for addressing such use cases. As data is everywhere, so is the need to maintain single source of data while making them available across varied environments, consumed in different forms based on their usage. 

In this blog our main focus would be on how the data across different S3users is shared in a secure manner with the help of Linux Standard Access and Bucket policies. Here in the S3 environment, the S3users are the end users, would run their workloads to upload/download data, which resides under respective buckets in the underlying IBM Storage Scale filesystems. Now, this data can be shared (like in other use cases) for the S3 user's to perform their Visualization / analytics experiments based on their requirements. 

Let us now delve into further detail on achieving the same. 

Pre-requisites:

- Deploy an IBM Storage Scale cluster 
- Enable the S3 protocol in the environment
- Create the Accounts and Buckets using the mms3 Command Line Interface (CLI)

Use case 1: Create two S3 accounts with the same group id(gid) and apply bucket policies

 In this use case, we describe about the data share across S3users
Step 1:

  Create S3 accounts using mms3 CLI as shown below 

Step 2:

Create aliases on the Application or the Client node for the respective accounts created i.e account-first, account-second, account-third with their S3 access and secret keys. Use the AWS S3 CLI for performing the IO uploads and downloads for the S3users and then create a bucket using the alias for account-first as shown below 

Note: Aliases in examples are referred as "aws-account-*" (i.e first, second, third) for easy command usage than specifying the entire S3 CLI command with the access, secret keys. 

Step 3:

 Now list the bucket created for the S3user, using the mms3 CLI command

Step 4:

Upload an object to the bucket via the S3 CLI for the S3user account-first with the alias created earlier 

Step 5:

Even though the S3user account-second  has the same group id (gid), the bucket created by s3user account-first cannot list the contents of the bucket. Now using a bucket policy setup by the account-first, grant access to all the data in the bucket for the users in the same group. Sample bucket policy is shown below 

Step 6:

Now list the bucket policy with S3api call to check the policy is applied to the bucket 

Step 7:

As S3user account-second, now try to upload and list the contents from the bucket

As you have worked through the above steps, it is evident that the S3users can share the data across same group (gid) by having the bucket policies in place which is a 2 step security in the S3 world. 

Use case 2: Create two S3 accounts with the different group id(gid) and apply bucket policies to share data 

This scenario is an extension to the Use case 1, where  S3user "account-third" is created using the mms3 CLI with a different group id. 

Step 1:

Create an S3user "account-third" with different group id (gid) 

Step 2:

S3user account-third cannot list the content of the bucket "bucket-first"  as this has a different group id (gid).  System administrator needs to set the appropriate permissions on the NewBucketsPath of the "account-first" user bucket for this S3user to access the data

Step 3:

Now account-third will be able to list the bucket

Note:  Even though account-third is able to access the bucket-first, it is still unable to list the objects inside it. Reason is by default S3 uploads take place with default mode bits 660 on the objects. 

Step 4:

Now sysadmin will modify the permissions on the objects to 664 of the s3user bucket-first directory contents on relevant files, so the other group users can access the objects

Step 5:

Now list the objects again using account-third, displays all the objects in the bucket "bucket-first"

This is how the S3users can share the data across different users with the help of bucket policies. There are other methods which exist today in the S3 world, which will be explored in future blogs. However, this blog gives an high level overview on how this is achieved. 

Conclusions:

In summary, the S3users can share the data across them by applying the bucket policies at a fine grain level. 

References:
[1] https://www.ibm.com/docs/en/storage-scale/5.2.0
[2] https://community.ibm.com/community/user/storage/blogs/madhu-punjabi/2024/04/26/ibm-storage-scale-ces-s3