Spectrum Scale Object with S3 and Swift User ACLs
This blog shows how to install and create a Spectrum Scale object store, amd shows how the access controls work on Swift and S3 access.
Installation and Configuration (30 minutes)
We start with a single node server with one disk for simplicity, more can be configured at install time, or added dynamically later. This example install is done on RHEL 8.5 on ppc64le. I'm using the Data Management edition here, but you can use the Data Access edition or the free (max 12TB) developer edition (x86 only) as well. More info in the Spectrum Scale FAQ. We will use the ansible toolkit to quickly create a cluster.
Prepare Linux:
ssh-keygen -t rsa
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
cat >>/etc/hosts <<EOF
172.17.146.43 dmiscale
172.17.146.44 dmiobject
EOF
yum install ntp-utils gcc gcc-c++ kernel-headers kernel-devel ntp cpp binutils nfs-utils ethtool rpcbind psmisc iputils boost-regex elfutils-devel make numactl
Install Spectrum Scale:
chmod +x /root/Spectrum_Scale_Data_Management-5.1.3.1-ppc64LE-Linux-install
/root/Spectrum_Scale_Data_Management-5.1.3.1-ppc64LE-Linux-install
cd /usr/lpp/mmfs/5.1.3.1/ansible-toolkit
./spectrumscale setup -s 172.17.146.43
./spectrumscale node add dmiscale -p -a -n -q -m -g
./spectrumscale nsd add /dev/dm-1
./spectrumscale nsd add -p dmiscale -fs scale /dev/dm-1
./spectrumscale callhome disable
./spectrumscale config gpfs -e 61000-62000
./spectrumscale install
Deploy protocols:
./spectrumscale config protocols -f scale -m /ibm/scale/ces
./spectrumscale config protocols -e 172.17.146.44
./spectrumscale enable nfs smb object
./spectrumscale config object -f scale -m /ibm/scale -o object -e dmiobject -au admin -ap adminsecret -su swiftadmin -sp swiftsecret -dp dbsecret -s3 on
./spectrumscale deploy
Basic Object access
Further authentication configuration (object was configured as type local by the installer):
/usr/lpp/mmfs/gui/cli/mkuser admin -g SecurityAdmin -p adminsecret
mmuserauth service create --data-access-method file --type userdefined
mmuserauth service list
The swift protocol works from command line with the defined admin user:
source openrc
swift upload test_container openrc
swift list -l test_container
364 2022-04-25 09:20:29 application/octet-stream openrc
364
This creates a bucket in the default admin project, which is OK for testing, but we'll need to use a different project later.
The S3 API is enabled for single region (us-east-1) with protocol version V2. We need to create new S3 credentials to use the S3 protocol:
source openrc
openstack credential create --type ec2 --project admin admin '{"access": "s3admin", "secret": "s3adminsecret"}'
You can use Cyberduck with the The AWS2 signature version profile. The aws command line tool is not available on Power by default, so download and build it:
git clone https://github.com/aws/aws-cli.git; git checkout
cd aws-cli/
python3 ./setup.py sdist
pip3 install --upgrade ./dist/awscli-*.tar.gz
aws configure
AWS Access Key ID [None]: s3admin
AWS Secret Access Key [None]: s3adminsecret
Default region name [None]: us-east-1
Default output format [None]: text
The Object store on Spectrum Scale listens on port 8080, without SSL, so the aws command is as follows:
aws --endpoint-url=http://172.17.146.44:8080 s3 ls test_container
2009-02-03 17:45:09 test_container
Terms and Definitions
The basic access is done, but we're using the default project and default admin user. Let's go into more detail now. We need to discuss some terms and how these apply to Spectrum Scale. The following definitions (in italics) are copied from the Openstack Documentation. (see reference links at the bottom of this document)
Domain
An Identity service API v3 entity. Domains are a collection of projects and users that define administrative boundaries for managing Identity entities. Domains can represent an individual, company, or operator-owned space. They expose administrative activities directly to system users. Users can be granted the administrator role for a domain. A domain administrator can create projects, users, and groups in a domain and assign roles to users and groups in a domain.
One Spectrum Scale cluster is one domain, the preconfigured name is "Default".
Group
An Identity service API v3 entity. Groups are a collection of users owned by a domain. A group role, granted to a domain or project, applies to all users in the group. Adding or removing users to or from a group grants or revokes their role and authentication to the associated domain or project.
There are no groups defined by default in Spectrum Scale.
Project
A container that groups or isolates resources or identity objects. Depending on the service operator, a project might map to a customer, account, organization, or tenant.
Spectrum Scale has a default admin project which should be used for administrative purposes only. Container creation, quotas, users and their role assignments should be done on projects created for that specific purpose.
Region
An Identity service API v3 entity. Represents a general division in an OpenStack deployment. You can associate zero or more sub-regions with a region to make a tree-like structured hierarchy. Although a region does not have a geographical connotation, a deployment can use a geographical name for a region, such as us-east.
Spectrum Scale Object store is deployed as a single Region, but can be converted to multi-region which allows for object replication to other clusters.
Storage
PolicyStorage policies enable segmenting of the Object Storage within a single cluster for various use cases. OpenStack Swift supports storage policies in which you can define the replication settings and location of objects in a cluster.
IBM Spectrum Scale enhances storage policies to add the following functions for Object Storage:
- The file-access object capability (unified file and object access)
- Compression
- Encryption
See the link to
Nishaan Docrat's whitepaper in the reference section at the bottom of this blog on how to configure this.
User
A digital representation of a person, system, or service that uses OpenStack cloud services. The Identity service validates that incoming requests are made by the user who claims to be making the call. Users have a login and can access resources by using assigned tokens. Users can be directly assigned to a particular project and behave as if they are contained in that project.
The keystone admin user is defined during the ansible-toolkit installation or the manual initialization using mmobj. The "admin" user account name is often used. Users need a role assigned to them to be able to perform actions in a project. The user definitions can be stored in an external database like LDAP, or in a local keystone database.
Role
A personality with a defined set of user rights and privileges to perform a specific set of operations. The Identity service issues a token to a user that includes a list of roles. When a user calls a service, that service interprets the user role set, and determines to which operations or resources each role grants access. A user can have different roles in different projects.
- Reader: read-only access
- Member: reader+creation
- Admin: member+deletion
- ResellerAdmin: Quota administrator
You will need to assign the Admin and ResellerAdmin role within a project to a user so the project and its buckets can be managed. Regular users should be assigned the member role if they only need swift object access. Members cannot create swift containers, they need ACLs set to allow for this.
Credentials
Within the S3 environment credentials consist of an access key (similar to a username) and a secret key (similar to a password). Credentials can be stored in profiles in the ~/.aws/credentials file. Credentials are defined inside a project (tenant).`
S3 credentials can only exist in the local keystone EC2 repository, unlike users that can come from an external keystone, LDAP or AD server. See: https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-configuring-openstack-ec2-credentials
We can create Swift users and S3 credentials to access the containers/buckets. These are two ways of accessing the same objects. The role a user is assigned in a project has an effect on what the default permissions are, but roles are primarily an openstack thing, not specifically a swift object access thing. This means that specific access user have to buckets in their projects(accounts) is defined using either swift ACLS definition commands, or set by S3api calls that define S3 ACLs. These ACLs are not the same and not compatible.
To illustrate permission settings, we'll create a project, users and credentials. First we create the swift users and assign roles to them, then we'll create S3 credentials and set ACLs for them.
Create the project:
openstack project create --description "Shared Project 1" project1
Create Swift users and assign different roles to them using CLI: (GUI is also OK)
openstack user create --password "p1secret" p1admin
openstack user create --password "user1secret" user1
openstack user create --password "user2secret" user2
openstack user create --password "user3secret" user3
openstack role add --project project1 --user p1admin admin
openstack role add --project project1 --user p1admin ResellerAdmin
openstack role add --project project1 --user user1 admin
openstack role add --project project1 --user user2 member
openstack role add --project project1 --user user3 reader
openstack role assignment list --project project1 --names
+---------------+-----------------+-------+------------------+--------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+---------------+-----------------+-------+------------------+--------+--------+-----------+
| admin | p1admin@Default | | project1@Default | | | False |
| ResellerAdmin | p1admin@Default | | project1@Default | | | False |
| admin | user1@Default | | project1@Default | | | False |
| member | user2@Default | | project1@Default | | | False |
| reader | user3@Default | | project1@Default | | | False |
+---------------+-----------------+-------+------------------+--------+--------+-----------+
We can now use the Swift interface or the Spectrum Scale GUI to create containers and objects in the project. We use the p1admin user to create a container and set Quota:
export OS_USERNAME=p1admin
export OS_PASSWORD=p1secret
export OS_PROJECT_NAME=project1
swift post container1 -m 'quota-bytes:59055800320'
The user1 account has the admin role, so can create containers and upload objects.
export OS_USERNAME=user1
export OS_PASSWORD=user1secret
swift upload container1 /root/openrc.user1
swift list container1
/root/openrc
/root/openrc.user1
The user2 user is only a member, so does not have any innate permissions:
export OS_USERNAME=user2
export OS_PASSWORD=user2secret
swift list container1
... 403 forbidden ... Access was denied ...
We can add an ACL to the container to allow read and write access for user2:
swift --os-username p1admin --os-password p1secret post 'container1' --read-acl 'project1:user2' --write-acl 'project1:user2'
swift list container1
/root/openrc
/root/openrc.user1
swift upload container1 /root/openrc.user2
The user3 user only has the reader role, so also no privileges on the bucket. We can add a read ACL in addition to the existing ACLs:
swift --os-username p1admin --os-password p1secret post 'container1' --read-acl 'project1:user2,project1:user3' --write-acl 'project1:user2'
swift --os-username p1admin --os-password p1secret stat 'container1' | grep ACL
Read ACL: project1:user2,project1:user3
Write ACL: project1:user2
export OS_USERNAME=user2
export OS_PASSWORD=user2secret
swift list container1
root/openrc
root/openrc.user1
root/openrc.user2
swift upload container1 /root/openrc.user3
... 403 Forbidden ... Access was denied ...
In the GUI we can set Swift permissions a little easier than on the command line:
So in summary, for swift object access you should assign the admin role only if you want the user to be able to manage all containers within a project, otherwise assign the member role and allow access to specific containers only using ACL settings. Be careful when using the command line when settings ACLs, the input validation is limited. The GUI is recommended.
S3 access
Next we'll create S3 credentials for the project, with unique access and secret keys. These could be the same as the Swift username and password, but not necessarily. As a swift admin user in the project, run the following commands to create the credentials in the project:
openstack credential create --type ec2 --project project1 user1 '{"access": "S3user1", "secret": "S3user1secret"}'
openstack credential create --type ec2 --project project1 user2 '{"access": "S3user2", "secret": "S3user2secret"}'
openstack credential create --type ec2 --project project1 user3 '{"access": "S3user3", "secret": "S3user3secret"}'
Add profile stanzas for the new users in /root/.aws/credentials using the aws command:
aws configure --profile user4
AWS Access Key ID [None]: S3user4
AWS Secret Access Key [None]: S3user4secret
Default region name [None]: us-east-1
Default output format [None]: text
We'll set an alias to make command line access easier.
alias dmi-aws="aws --endpoint-url=http://172.17.146.44:8080 "
We have not set any S3 ACLs, so let's see what the aws commands are allowed to do to.
All three S3user credentials allow the listing of the buckets and their contents:
dmi-aws --profile S3user1 s3 ls s3://
2009-02-03 17:45:09 container1
All three S3user credentials allow copying files into the buckets, not what we expect from the ACLs set in Swift :, which shows mixing access using swift and s3 is a problem.
dmi-aws --profile S3user1 s3 cp file1 s3://container1
upload: ./file1 to s3://container1/file1
dmi-aws --profile S3user1 s3 cp file1 s3://container1
upload: ./file1 to s3://container1/file1
dmi-aws --profile S3user1 s3 cp file1 s3://container1
upload: ./file1 to s3://container/file1
Only S3user1 is allowed to create a new bucket, as this requires an admin role in the project.
dmi-aws --profile S3user1 s3 mb s3://S3bucket1
make_bucket: S3user1
dmi-aws --profile S3user2 s3 mb s3://S3bucket2
make_bucket failed: s3://S3bucket2 An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied.
Even though we set roles to allow admin, member and reader privilege levels, these roles are only relevant for general openstack commands. the ACLs when mixing AWS and Swift can get confusing, as they are not compatible. So either keep these two use cases separate, or only use admin role users for both the swift user and the user used for the credentials, so no ACLs need to be set on either. as the user is allowed to do anything with its bucket anyway.
The S3 protocol uses its own ACLs to set access rights. If the bucket is created using swift, no ACLs are set for either Swift or the S3 credentials(obviously):
swift post container1
swift stat bucky | grep ACL
RequestsDependencyWarning)
Read ACL:
Write ACL:
dmi-aws s3api --profile S3user1 get-bucket-acl --bucket container1
{
"Owner": {
"DisplayName": "",
"ID": ""
},
"Grants": []
}
However, if the bucket is created using s3 commands, the creator of the bucket has all rights in S3 ACLs:
dmi-aws --profile S3user1 s3 mb s3://container1
make_bucket: bucky2
dmi-aws s3api --profile S3user1 get-bucket-acl --bucket S3bucket1
{
"Owner": {
"DisplayName": "project1:user1",
"ID": "project1:user1"
},
"Grants": [
{
"Grantee": {
"DisplayName": "project1:user1",
"ID": "project1:user1",
"Type": "CanonicalUser"
},
"Permission": "FULL_CONTROL"
}
]
}
As the user2 swift user which has the S3user2 credential is not listed in the ACLs, it cannot create a file there:
dmi-aws --profile S3user2 s3 cp openrc s3://S3bucket1
upload failed: ./openrc to s3://S3bucket1/openrc An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.
Set the ACLs for the bucket using S3API to allow user2 access:
dmi-aws s3api --profile S3user1 put-bucket-acl --bucket S3bucket1 --grant-read id=project1:user2 --grant-full-control id=project1:user2
dmi-aws --profile S3user2 s3 cp openrc s3://S3bucket1
upload: ./openrc to s3://S3bucket1/openrc
To summarize, the permissions to access a container/bucket are dependent on the role of the swift user account for which the credential was created. The admin role always has access, users with other roles depend on the ACLs set. Only ACLs that are set for the appropriate protocol are relevant for access, and setting both swift and S3 ACLs on a bucket does not work.
When using the admin role to access a bucket using both S3 and Swift the S3 access key and swift username may be the same, but are not required to be.
If a bucket is only used for S3 access, permissions are set using S3API, not via openstack or the GUI. The GUI only supports swift ACLs.
To configure the Spectrum Scale Object Store further, IBM's Nishaan Docrat has written an excellent whitepaper on the subject:
https://community.ibm.com/community/user/storage/viewdocument/setting-up-an-s3-compliant-object-s?CommunityKey=1142f81e-95e4-4381-95d0-7977f20d53fa&tab=librarydocuments
Other reference Links:
https://docs.openstack.org/keystone/pike/admin/identity-concepts.html
https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-how-manage-openstack-s3-api
https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=storage-configuring-openstack-ec2-credentials