File and Object Storage

 View Only

Spectrum Scale Object access with Swift and S3

By MAARTEN KREUGER posted 26 days ago

  

Spectrum Scale Object with S3 and Swift User ACLs

 

This blog shows how to install and create a Spectrum Scale object store, amd shows how the access controls work on Swift and S3 access.

Installation and Configuration (30 minutes)

We start with a single node server with one disk for simplicity, more can be configured at install time, or added dynamically later. This example install is done on RHEL 8.5 on ppc64le. I'm using the Data Management edition here, but you can use the Data Access edition or the free (max 12TB) developer edition (x86 only) as well. More info in the Spectrum Scale FAQ. We will use the ansible toolkit to quickly create a cluster.

Prepare Linux:

ssh-keygen -t rsa

cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

cat >>/etc/hosts <<EOF

172.17.146.43 dmiscale

172.17.146.44 dmiobject

EOF

yum install ntp-utils gcc gcc-c++ kernel-headers kernel-devel ntp cpp binutils nfs-utils ethtool rpcbind psmisc iputils boost-regex elfutils-devel make numactl

Install Spectrum Scale:

chmod +x /root/Spectrum_Scale_Data_Management-5.1.3.1-ppc64LE-Linux-install

/root/Spectrum_Scale_Data_Management-5.1.3.1-ppc64LE-Linux-install

cd /usr/lpp/mmfs/5.1.3.1/ansible-toolkit

./spectrumscale  setup -s 172.17.146.43

./spectrumscale node add dmiscale -p -a -n -q -m -g

./spectrumscale nsd add /dev/dm-1

./spectrumscale nsd add -p dmiscale -fs scale /dev/dm-1

./spectrumscale callhome disable

./spectrumscale config gpfs -e 61000-62000

./spectrumscale install

Deploy protocols:

./spectrumscale config protocols -f scale -m /ibm/scale/ces

./spectrumscale config protocols -e 172.17.146.44

./spectrumscale enable nfs smb object

./spectrumscale config object -f scale -m /ibm/scale -o object -e dmiobject -au admin -ap adminsecret -su swiftadmin -sp swiftsecret -dp dbsecret -s3 on

./spectrumscale deploy

Basic Object access

Further authentication configuration (object was configured as type local by the installer):

/usr/lpp/mmfs/gui/cli/mkuser admin -g SecurityAdmin -p adminsecret

mmuserauth service create --data-access-method file --type userdefined

mmuserauth service list

 

The swift protocol works from command line with the defined admin user:

source openrc

swift upload test_container openrc

swift list -l test_container

   364 2022-04-25 09:20:29 application/octet-stream openrc

   364

 This creates a bucket in the default admin project, which is OK for testing, but we'll need to use a different project later.

 The S3 API is enabled for single region (us-east-1) with protocol version V2. We need to create new S3 credentials to use the S3 protocol:

source openrc

openstack credential create --type ec2 --project admin admin '{"access": "s3admin", "secret": "s3adminsecret"}'

 

You can use Cyberduck with the The AWS2 signature version profile.  The aws command line tool is not available on Power by default, so download and build it:

git clone https://github.com/aws/aws-cli.git; git checkout

cd aws-cli/

python3 ./setup.py sdist

pip3 install --upgrade ./dist/awscli-*.tar.gz

aws configure

AWS Access Key ID [None]: s3admin

AWS Secret Access Key [None]: s3adminsecret

Default region name [None]: us-east-1

Default output format [None]: text

 

The Object store on Spectrum Scale listens on port 8080, without SSL, so the aws command is as follows:

aws --endpoint-url=http://172.17.146.44:8080 s3 ls test_container

2009-02-03 17:45:09 test_container

Terms and Definitions

The basic access is done, but we're using the default project and default admin user. Let's go into more detail now. We need to discuss some terms and how these apply to Spectrum Scale. The following definitions (in italics) are copied from the Openstack Documentation. (see reference links at the bottom of this document)

 

Domain

 An Identity service API v3 entity. Domains are a collection of projects and users that define administrative boundaries for managing Identity entities. Domains can represent an individual, company, or operator-owned space. They expose administrative activities directly to system users. Users can be granted the administrator role for a domain. A domain administrator can create projects, users, and groups in a domain and assign roles to users and groups in a domain.

One Spectrum Scale cluster is one domain, the preconfigured name is "Default".

Group

An Identity service API v3 entity. Groups are a collection of users owned by a domain. A group role, granted to a domain or project, applies to all users in the group. Adding or removing users to or from a group grants or revokes their role and authentication to the associated domain or project.

There are no groups defined by default in Spectrum Scale.

Project

A container that groups or isolates resources or identity objects. Depending on the service operator, a project might map to a customer, account, organization, or tenant.

Spectrum Scale has a default admin project which should be used for administrative purposes only. Container creation, quotas, users and their role assignments should be done on projects created for that specific purpose.

Region

An Identity service API v3 entity. Represents a general division in an OpenStack deployment. You can associate zero or more sub-regions with a region to make a tree-like structured hierarchy. Although a region does not have a geographical connotation, a deployment can use a geographical name for a region, such as us-east.

Spectrum Scale Object store is deployed as a single Region, but can be converted to multi-region which allows for object replication to other clusters.

Storage

PolicyStorage policies enable segmenting of the Object Storage within a single cluster for various use cases. OpenStack Swift supports storage policies in which you can define the replication settings and location of objects in a cluster.

IBM Spectrum Scale enhances storage policies to add the following functions for Object Storage:

  • The file-access object capability (unified file and object access)
  • Compression
  • Encryption
See the link to Nishaan Docrat's whitepaper in the reference section at the bottom of this blog on how to configure this.

User

A digital representation of a person, system, or service that uses OpenStack cloud services. The Identity service validates that incoming requests are made by the user who claims to be making the call. Users have a login and can access resources by using assigned tokens. Users can be directly assigned to a particular project and behave as if they are contained in that project.

The keystone admin user is defined during the ansible-toolkit installation or the manual initialization using mmobj. The "admin" user account name is often used. Users need a role assigned to them to be able to perform actions in a project. The user definitions can be stored in an external database like LDAP, or in a local keystone database.

Role

A personality with a defined set of user rights and privileges to perform a specific set of operations. The Identity service issues a token to a user that includes a list of roles. When a user calls a service, that service interprets the user role set, and determines to which operations or resources each role grants access. A user can have different roles in different projects.

  • Reader: read-only access
  • Member: reader+creation
  • Admin: member+deletion
  • ResellerAdmin: Quota administrator

 You will need to assign the Admin and ResellerAdmin role within a project to a user so the project and its buckets can be managed. Regular users should be assigned the member role if they only need swift object access. Members cannot create swift containers, they need ACLs set to allow for this.

  

Credentials

Within the S3 environment credentials consist of an access key (similar to a username) and a secret key (similar to a password). Credentials can be stored in profiles in the ~/.aws/credentials file. Credentials are defined inside a project (tenant).`

 S3 credentials can only exist in the local keystone EC2 repository, unlike users that can come from an external keystone, LDAP or AD server. See: https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-configuring-openstack-ec2-credentials

 

We can create Swift users and S3 credentials to access the containers/buckets. These are two ways of accessing the same objects.  The role a user is assigned in a project has an effect on what the default permissions are, but roles are primarily an openstack thing, not specifically a swift object access thing. This means that specific access user have to buckets in their  projects(accounts) is defined using either swift ACLS definition commands, or set by S3api calls that define S3 ACLs. These ACLs are not the same and not compatible.  

To illustrate permission settings, we'll create a project, users and credentials. First we create the swift users and assign roles to them, then we'll create S3 credentials and set ACLs for them. 

Create the project:

openstack project create --description "Shared Project 1" project1

Create Swift users and assign different roles to them using CLI: (GUI is also OK)

openstack user create --password "p1secret" p1admin

openstack user create --password "user1secret" user1

openstack user create --password "user2secret" user2

openstack user create --password "user3secret" user3

openstack role add --project project1 --user p1admin admin

openstack role add --project project1 --user p1admin ResellerAdmin

openstack role add --project project1 --user user1 admin

openstack role add --project project1 --user user2 member

openstack role add --project project1 --user user3 reader

 

openstack role assignment list --project project1 --names

+---------------+-----------------+-------+------------------+--------+--------+-----------+

| Role          | User            | Group | Project          | Domain | System | Inherited |

+---------------+-----------------+-------+------------------+--------+--------+-----------+

| admin         | p1admin@Default |       | project1@Default |        |        | False     |

| ResellerAdmin | p1admin@Default |       | project1@Default |        |        | False     |

| admin         | user1@Default   |       | project1@Default |        |        | False     |

| member        | user2@Default   |       | project1@Default |        |        | False     |

| reader        | user3@Default   |       | project1@Default |        |        | False     |

+---------------+-----------------+-------+------------------+--------+--------+-----------+

 

We can now use the Swift interface or the Spectrum Scale GUI to create containers and objects in the project. We use the p1admin user to create a container and set Quota:

export OS_USERNAME=p1admin

export OS_PASSWORD=p1secret

export OS_PROJECT_NAME=project1

swift post container1 -m 'quota-bytes:59055800320'

 

The user1 account has the admin role, so can create containers and upload objects.

export OS_USERNAME=user1

export OS_PASSWORD=user1secret

swift upload container1 /root/openrc.user1

swift list container1

/root/openrc

/root/openrc.user1

 

The user2 user is only a member, so does not have any innate permissions:

export OS_USERNAME=user2

export OS_PASSWORD=user2secret

swift list container1

... 403 forbidden ... Access was denied ...

 

We can add an ACL to the container to allow read and write access for user2:

swift --os-username p1admin --os-password p1secret post 'container1' --read-acl 'project1:user2' --write-acl 'project1:user2'

swift list container1

/root/openrc

/root/openrc.user1

swift upload container1 /root/openrc.user2

 

The user3 user only has the reader role, so also no privileges on the bucket. We can add a read ACL in addition to the existing ACLs:

swift --os-username p1admin --os-password p1secret post 'container1' --read-acl 'project1:user2,project1:user3' --write-acl 'project1:user2'

swift --os-username p1admin --os-password p1secret stat 'container1' | grep ACL

             Read ACL:  project1:user2,project1:user3

             Write ACL: project1:user2

 

export OS_USERNAME=user2

export OS_PASSWORD=user2secret

swift list container1

root/openrc

root/openrc.user1

root/openrc.user2

swift upload container1 /root/openrc.user3

... 403 Forbidden ... Access was denied ...

 

In the GUI we can set Swift permissions a little easier than on the command line:

Spectrum Scale GUI Swift ACLs

 

So in summary, for swift object access you should assign the admin role only if you want the user to be able to manage all containers within a project, otherwise assign the member role and allow access to specific containers only using ACL settings. Be careful when using the command line when settings ACLs, the input validation is limited. The GUI is recommended.

S3 access

Next we'll create S3 credentials for the project, with unique access and secret keys. These could be the same as the Swift username and password, but not necessarily. As a swift admin user in the project, run the following commands to create the credentials in the project:

openstack credential create --type ec2 --project project1 user1 '{"access": "S3user1", "secret": "S3user1secret"}'

openstack credential create --type ec2 --project project1 user2 '{"access": "S3user2", "secret": "S3user2secret"}'

openstack credential create --type ec2 --project project1 user3 '{"access": "S3user3", "secret": "S3user3secret"}'

 

Add profile stanzas for the new users in /root/.aws/credentials using the aws command:

aws configure --profile user4

AWS Access Key ID [None]: S3user4

AWS Secret Access Key [None]: S3user4secret

Default region name [None]: us-east-1

Default output format [None]: text

 

We'll set an alias to make command line access easier.

alias dmi-aws="aws --endpoint-url=http://172.17.146.44:8080 "

 

We have not set any S3 ACLs, so let's see what the aws commands are allowed to do to.
All three S3user credentials allow the listing of the buckets and their contents:

dmi-aws --profile S3user1 s3 ls s3://

2009-02-03 17:45:09 container1

 

All three S3user credentials allow copying files into the buckets, not what we expect from the ACLs set in Swift :, which shows mixing access using swift and s3 is a problem.

dmi-aws --profile S3user1 s3 cp file1 s3://container1

upload: ./file1 to s3://container1/file1                           

dmi-aws --profile S3user1 s3 cp file1 s3://container1

upload: ./file1 to s3://container1/file1                          

dmi-aws --profile S3user1 s3 cp file1 s3://container1

upload: ./file1 to s3://container/file1                           

 

Only S3user1 is allowed to create a new bucket, as this requires an admin role in the project.

dmi-aws --profile S3user1 s3 mb  s3://S3bucket1

make_bucket: S3user1

dmi-aws --profile S3user2 s3 mb  s3://S3bucket2

make_bucket failed: s3://S3bucket2 An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied.

Even though we set roles to allow admin, member and reader privilege levels, these roles are only relevant for general openstack commands. the ACLs when mixing AWS and Swift can get confusing, as they are not compatible. So either keep these two use cases separate, or only use admin role users for both the swift user and the user used for the credentials, so no ACLs need to be set on either. as the user is allowed to do anything with its bucket anyway.

The S3 protocol uses its own ACLs to set access rights. If the bucket is created using swift, no ACLs are set for either Swift or the S3 credentials(obviously): 

swift post container1

swift stat bucky | grep ACL

  RequestsDependencyWarning)

              Read ACL:

             Write ACL:

dmi-aws s3api --profile S3user1 get-bucket-acl --bucket container1

{

    "Owner": {

        "DisplayName": "",

        "ID": ""

    },

    "Grants": []

}

 

However, if the bucket is created using s3 commands, the creator of the bucket has all rights in S3 ACLs:

dmi-aws --profile S3user1 s3 mb s3://container1

make_bucket: bucky2

dmi-aws s3api --profile S3user1 get-bucket-acl --bucket S3bucket1

{

    "Owner": {

        "DisplayName": "project1:user1",

        "ID": "project1:user1"

    },

    "Grants": [

        {

            "Grantee": {

                "DisplayName": "project1:user1",

                "ID": "project1:user1",

                "Type": "CanonicalUser"

            },

            "Permission": "FULL_CONTROL"

        }

    ]

}

 

As the user2 swift user which has the S3user2 credential is not listed in the ACLs, it cannot create a file there:

dmi-aws --profile S3user2 s3 cp openrc s3://S3bucket1

upload failed: ./openrc to s3://S3bucket1/openrc An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.

 

Set the ACLs for the bucket using S3API to allow user2 access:

dmi-aws s3api --profile S3user1 put-bucket-acl --bucket S3bucket1 --grant-read id=project1:user2 --grant-full-control id=project1:user2

dmi-aws --profile S3user2 s3 cp openrc s3://S3bucket1

upload: ./openrc to s3://S3bucket1/openrc   

 

To summarize, the permissions to access a container/bucket are dependent on the role of the swift user account for which the credential was created. The admin role always has access, users with other roles depend on the ACLs set. Only ACLs that are set for the appropriate protocol are relevant for access, and setting both swift and S3 ACLs on a bucket does not work.

When using the admin role to access a bucket using both S3 and Swift the S3 access key and swift username may be the same, but are not required to be.

If a bucket is only used for S3 access, permissions are set using S3API, not via openstack or the GUI. The GUI only supports swift ACLs.

 

To configure the Spectrum Scale Object Store further, IBM's Nishaan Docrat has written an excellent whitepaper on the subject:

https://community.ibm.com/community/user/storage/viewdocument/setting-up-an-s3-compliant-object-s?CommunityKey=1142f81e-95e4-4381-95d0-7977f20d53fa&tab=librarydocuments

 

Other reference Links:

https://docs.openstack.org/keystone/pike/admin/identity-concepts.html

https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-how-manage-openstack-s3-api

https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-configuring-openstack-ec2-credentials

1 comment
26 views

Permalink

Comments

22 days ago

Great write-up of Swift and S3 ACLs. Will definitely come in handy!