Authors: @RAVI KUMAR KOMANDURI @Madhu Punjabi
Data is everywhere and the need for Data Management and accessibility in Distributed/Hybrid (On-/Off-Prem) Computing environments is playing a key role in the current era. High Performance Computing environments (Seismic, Computational Fluid Dynamics, Aerospace, Weather Forecasting etc.) refine the data for performing Computational or Analytics workloads to deliver data insights. IBM Storage Scale Data Access Services (DAS S3) [1] is a Containerized S3 Object protocol solution, built on top of IBM Storage Scale Containerized Native Storage Access [2], which accesses underlying IBM Storage Scale [3] File system. In-order to address emerging industry requirements on data sharing, a comprehensive data access is being solutioned via multi-protocol enablement across (DAS S3, NFS, POSIX and IBM Spectrum Scale Container Storage Interface (CSI) [4]).
This blog describes use cases of data sharing across multiple sites using DAS S3 with CSI, IBM Storage Scale (with NFS and POSIX client)[5]. DAS S3 is an object storage solution that runs in a secure environment on top of Red Hat OpenShift Container Platform [6] with IBM Storage Scale file system serving as the backend storage. Let us now delve into further details on multi-protocol data access between DAS(S3) - NFS - CSI - POSIX.
Note: SELinux is disabled in IBM Storage Scale cluster to provide the multi-protocol data access capability
Note: In the blog we will refer to DAS as DAS-S3 Cluster and a second cluster termed as NFS - CSI - DAS Cluster.
Figure 1: DAS-S3 Cluster that is deployed and accessing the IBM Storage Scale as backend Storage Cluster (SELinux is Disabled on Storage Cluster)
Note: Two key considerations to note from the above figure: 1) Access/Secret keys, and 2) SELinux disabled in the backend Storage Cluster.
With these data points at hand, the IBM Storage Scale Cluster has NFS protocol (NFS-Ganesha Server) configured, and the same data can be accessed with NFS client and DAS S3. Similarly, the same data on the IBM Storage Scale Cluster can be accessed with POSIX and DAS-S3. IBM Storage Scale CSI can be on a separate cluster, or it can be part of the same DAS-S3 cluster with appropriate user credentials configured. In the Figure 2, SELinux is disabled on IBM Storage Cluster to have multi-protocol configuration enabled for data sharing.
Figure 2: DAS-S3 Cluster and IBM Storage Scale cluster with protocols enabled using the same backend Storage Scale file system with SELinux Disabled.
Let us now discuss two use cases in detail.
Use case 1: Write data from DAS S3 cluster, read and update from (POSIX, NFS, CSI) environments and finally re-read from DAS Cluster
The primary goal in this case is the S3 application creates objects (i.e uses S3 PUT) which the DAS-S3 cluster stores as file in the backend Scale File System. It is further illustrated how the different environments (POSIX, NFS and CSI) are accessing the same data and perform operations. With POSIX (NSD client), the user of the Storage Scale Cluster will access the data with the required credentials and does read/write. Further the same directory is exported via NFS-Ganesha protocol on the Scale environment, in order to provide access for the users via the NFS client. Further on the data can be accessed by CSI (with static provisioning [7]) with appropriate user credentials. Finally, the DAS-S3 Cluster will be able to re-read the data after all updates.
Step wise procedure:
Listed below are the steps that are required to be performed in each environment to achieve the data sharing capability.
Storage Cluster:
1 Check the SELinux setting.
2 Create the directory on the Storage Cluster with appropriate permissions for DAS S3 user.
$ mkdir /ibm/fs1/s3user10005-dir
$ ls -lad /ibm/fs1/s3user10005-dir drwxr-xr-x 2 root root 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
$ chmod 770 /ibm/fs1/s3user10005-dir
$ chown 10005:11000 /ibm/fs1/s3user10005-dir
$ ls -lad /ibm/fs1/s3user10005-dir drwxrwx--- 2 10005 11000 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
|
3. Create the user on the Storage Cluster for POSIX access to the data directory
$ useradd nfsuser10005 -d /home/nfsuser10005 -u 10005 -g 11000
$ id nfsuser10005 uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ ls -lad /home/nfsuser10005 drwx------ 2 nfsuser10005 nfsgrp11000 90 May 24 03:32 /home/nfsuser10005
$ ls -lad /ibm/fs1/s3user10005-dir drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
|
DAS-S3 Cluster:
4. Create S3 user on the OpenShift cluster with respective uid, gid along with option '--newBucketsPath'
$ mmdas account create s3user10005-hpo@ibm.com --uid 10005 --gid 11000 --newBucketsPath "/mnt/remote-sample/s3user10005-dir" |
5. Create an alias for the S3 user, create a bucket and upload an object
$ alias s3u10005='AWS_ACCESS_KEY_ID=<access-key> AWS_SECRET_ACCESS_KEY=<secret-key> aws --endpoint https://10.13.20.97 --no-verify-ssl s3'
$ s3u10005 mb s3://data-share-das-posix-nfs-csi make_bucket: data-share-das-posix-nfs-csi
$ echo "this is file written from hpo as user 10005" > ./object-das-posix-nfs-csi
$ s3u10005 cp /root/object-das-posix-nfs-csi s3://data-share-das-posix-nfs-csi upload: ./object-das-posix-nfs-csi to s3://data-share-das-posix-nfs-csi/object-das-posix-nfs-csi
$ s3u10005 ls s3://data-share-das-posix-nfs-csi 2023-05-24 04:00:58 44 object-das-posix-nfs-csi
|
Storage Cluster (via NSD client for POSIX access):
6. As the nfsuser10005 user that is created in Step 3, read and update the object
$ id uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami nfsuser10005
$ ls -lrt /ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi/ total 1 -rw-rw---- 1 nfsuser10005 nfsgrp11000 44 May 24 04:00 object-das-posix-nfs-csi
$ echo "this is an update from posix client as the nfsuser10005" >> ./object-das-posix-nfs-csi
$ ls -lrt total 1 -rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 24 06:00 object-das-posix-nfs-csi
|
7. Create a NFS-Ganesha export on the cluster.
Note: Reference to the IBM Storage Scale documentation
$ mmnfs export add /ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi --client "*(Access_Type=RW,Protocols=3:4,Squash=no_root_squash,anonymous_uid=10005,anonymous_gid=11000,Transports=TCP:UDP)" mmnfs: The NFS export was created successfully.
|
Standalone NFS client:
8. On the NFS client system as shown in fig 2, the system administrator would mount the respective NFS export from Storage Cluster via CES IP and create the NFS user with required credentials
$ mkdir -p /mnt/nfsuser10005-data-share
$ mount -o vers=3 -t nfs 10.11.28.204:/ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi /mnt/nfsuser10005-data-share Note: NFS client can mount with vers=4.0 as well
$ id uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami nfsuser10005
$ cd /mnt/nfsuser10005-data-share/
$ pwd /mnt/nfsuser10005-data-share
$ ls -lrt total 1 -rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 24 06:00 object-das-posix-nfs-csi
9. Read and update the file as nfsuser10005 user
$ echo "this is an update from the NFS client user nfs10005" >> ./object-das-posix-nfs-csi
$ cat object-das-posix-nfs-csi this is file written from hpo as user 10005 this is an update from posix client as the nfsuser10005 this is an update from the NFS client user nfs10005
$ ls -lrt object-das-posix-nfs-csi -rw-rw---- 1 nfsuser10005 nfsgrp11000 152 May 24 06:44 object-das-posix-nfs-csi
|
CSI namespace access with the userid 10005 on the DAS cluster (or any other CSI cluster, configured to access the same backend Storage):
Note: Refer to the IBM Storage Scale CSI documentation for creating static volumes
10. Pre-requisite for the access is to create the new namespace then Persistent Volume(PV) , Persistent Volume Claim(PVC) and pod creation with respective user credentials
i) Create a new project and describe the namespace
$ oc new-project hpo-nfs-csi-posix-datashare
$ oc describe ns hpo-nfs-csi-posix-datashare Name: hpo-nfs-csi-posix-datashare Labels: kubernetes.io/metadata.name=hpo-nfs-csi-posix-datashare pod-security.kubernetes.io/audit=restricted pod-security.kubernetes.io/audit-version=v1.24 pod-security.kubernetes.io/warn=restricted pod-security.kubernetes.io/warn-version=v1.24 Annotations: openshift.io/description: openshift.io/display-name: openshift.io/requester: kube:admin openshift.io/sa.scc.mcs: s0:c27,c14 openshift.io/sa.scc.supplemental-groups: 1000730000/10000 openshift.io/sa.scc.uid-range: 1000730000/10000 Status: Active
No resource quota.
No LimitRange resource.
|
ii) Create the persistent volume(pv) for the directory path that is required for access and apply the pv
$ cat pv-hpo-nfs-csi-posix.yaml apiVersion: v1 kind: PersistentVolume metadata: name: static-pv-hpo-csi-nfsuser10005 spec: capacity: storage: 1Gi accessModes: - ReadWriteMany csi: driver: spectrumscale.csi.ibm.com volumeHandle: 0;0;601738680925587765;CE1C0B0A:642E8A09;;;/mnt/remote-sample/s3user10005-dir/data-share-das-posix-nfs-csi
$ oc apply -f ./pv-hpo-nfs-csi-posix.yaml persistentvolume/static-pv-hpo-csi-nfsuser10005 created
$ oc get pv static-pv-hpo-csi-nfsuser10005 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE static-pv-hpo-csi-nfsuser10005 1Gi RWX Retain Available 7s
|
iii) Create a PVC which will bound to the pv
$ cat pvc-hpo-nfs-csi-posix.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: static-pvc-hpo-csi-fornfsuser10005 spec: volumeName: static-pv-hpo-csi-nfsuser10005 accessModes: - ReadWriteMany resources: requests: storage: 1Gi
$ oc apply -f ./pvc-hpo-nfs-csi-posix.yaml persistentvolumeclaim/static-pvc-hpo-csi-fornfsuser10005 created.
$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE static-pvc-hpo-csi-fornfsuser10005 Bound static-pv-hpo-csi-nfsuser10005 1Gi RWX 3s
|
iv) Create the Pod with user id and group id to access the directory provided via the above PVC (showing only user group)
$ cat pod-hpo-nfs-csi-posix.yaml apiVersion: v1 kind: Pod metadata: name: csi-hpo-datashare-nfsuser10005 spec: containers: - name: sec-ctx-demo-2 image: busybox:1.28 command: [ "sh", "-c", "sleep 1h"] securityContext: runAsUser: 10005 runAsGroup: 11000 fsGroup: 6000 volumeMounts: - name: mypvc mountPath: /tmp/busy volumes: - name: mypvc persistentVolumeClaim: claimName: static-pvc-hpo-csi-fornfsuser10005 readOnly: false
$ oc apply -f ./pod-hpo-nfs-csi-posix.yaml
$ oc get pods NAME READY STATUS RESTARTS AGE csi-hpo-datashare-nfsuser10005 1/1 Running 0 57s |
11. Login to the pod using 'oc' command and then read and update the file
$ oc exec --stdin --tty csi-hpo-datashare-nfsuser10005 -- /bin/sh $ df -h /tmp/busy Filesystem Size Used Available Use% Mounted on remote-sample 3.9T 134.9G 3.8T 3% /tmp/busy $ umask 0022
$ ls -lrt /tmp/busy total 1 -rw-rw---- 1 10005 11000 152 May 24 13:44 object-das-posix-nfs-csi
/tmp/busy $ id uid=10005(10005) gid=11000 groups=11000
/tmp/busy $ whoami 10005
/tmp/busy $ echo "this is an update from the CSI pod as user id 10005 and group id 11000" >>./object-das-posix-nfs-csi
/tmp/busy $ ls -lrt total 1 -rw-rw---- 1 10005 11000 223 May 25 06:08 object-das-posix-nfs-csi
|
DAS-S3 Cluster:
12. Now re-read the object via S3 user with the required access and secret key
$ s3u10005 ls s3://data-share-das-posix-nfs-csi 2023-05-24 23:08:32 223 object-das-posix-nfs-csi |
Overall, the above procedure explains how the data can be shared across different environments (DAS-S3, POSIX, NFS, CSI).
Use case 2: Create a file from NFS client, update the file from (DAS-S3 cluster, POSIX, CSI) environments and finally re-read from NFS client.
Scenario is to create a file from NFS client system which has mounted an NFS export created in IBM Storage Scale cluster using NFS-Ganesha protocol, access from DAS-S3 cluster as S3 user to perform the operations. Further access via POSIX (NSD client) with appropriate user credentials and then via the CSI environment and then finally re-read from NFS environment.
Storage Cluster:
1. Create a directory with appropriate user id and group id set to NFS user nfsuser10005 and group 11000
$ mkdir nfsuser10005-dir
$ chmod 770 nfsuser10005-dir
$ chown 10005:11000 nfsuser10005-dir
$ ls -lrtd nfsuser10005-dir drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 25 01:39 nfsuser10005-dir
|
2. Make the directory available as a NFS export from the cluster
$ mmnfs export add /ibm/fs1/nfsuser10005-dir --client "*(Access_Type=RW,Protocols=3:4,Squash=no_root_squash,anonymous_uid=10005,anonymous_gid=11000,Transports=TCP:UDP)" mmnfs: The NFS export was created successfully.
$ mmnfs export list --nfsdefs /ibm/fs1/nfsuser10005-dir
Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit Pseudo Path ------------------------- ----------- ------- ----------- --------- ---------- -------------- ------------- ------------- ------- -------------- ------------------ ----------- ---------- ------------------------- /ibm/fs1/nfsuser10005-dir NONE * RW 3,4 TCP,UDP NO_ROOT_SQUASH 10005 11000 SYS FALSE NONE FALSE FALSE /ibm/fs1/nfsuser10005-dir
|
Standalone NFS client:
3. System administrator will mount the NFS exported directory on the NFS client system
$ mount -o vers=4.0 -t nfs 10.11.28.206:/ibm/fs1/nfsuser10005-dir /mnt/nfsuser10005-dir-share Note: NFS client can mount with vers=3.0 as well |
4. Create a sub-directory and a file under it
$ id uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami nfsuser10005
$ mkdir nfscreated-dir-for-das-share
$ ls -lrt total 1 drwxr-xr-x 2 nfsuser10005 nfsgrp11000 4096 May 25 01:48 nfscreated-dir-for-das-share
$ chmod 770 nfscreated-dir-for-das-share
$ ls -lrt total 1 drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 25 01:48 nfscreated-dir-for-das-share
$ echo "this is new file created from NFS client node as nfsuser10005" > ./nfsuser-data-share
$ ls -lrt total 1 -rw-r--r-- 1 nfsuser10005 nfsgrp11000 62 May 25 01:50 nfsuser-data-share
$ umask 0022
|
DAS-S3 Cluster:
5. Create a bucket using 'mmdas export' CLI command for the same directory, which was made available as NFS export, step 4 from above:
$ mmdas export create das-nfs-share --filesystemPath "/mnt/remote-sample/nfsuser10005-dir/nfscreated-dir-for-das-share" Export is successfully created.
$ mmdas export list das-nfs-share
Name Filesystem Path ----- ----------------- das-nfs-share /mnt/remote-sample/nfsuser10005-dir/nfscreated-dir-for-das-share/
|
6. List the file inside the S3 bucket and download the file. Modify and upload the file from S3 user perspective.
$ s3u10005 ls s3://das-nfs-share 2023-05-25 01:50:50 62 nfsuser-data-share
$ s3u10005 cp s3://das-nfs-share/nfsuser-data-share /tmp/nfsuser-data-share $ echo "this is das update from S3 user side " >> /tmp/nfsuser-data-share
$ ls -lrt /tmp/nfsuser-data-share -rw-r--r-- 1 root root 100 May 25 05:00 /tmp/nfsuser-data-share
$ s3u10005 cp /tmp/nfsuser-data-share s3://das-nfs-share
$ s3u10005 ls s3://das-nfs-share 2023-05-25 05:00:57 100 nfsuser-data-share
|
7. From NSD client, read and update the content of the file with respective user credentials
$ id uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ pwd /ibm/fs1/nfsuser10005-dir/nfscreated-dir-for-das-share
$ ls -lrt nfsuser-data-share -rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 25 05:00 nfsuser-data-share
$ echo "this is an update from the POSIX client as the user id 10005" >> ./nfsuser-data-share
$ ls -lrt total 1 -rw-rw---- 1 nfsuser10005 nfsgrp11000 161 May 25 22:32 nfsuser-data-share
|
CSI namespace access with the userid 10005 on the DAS-S3 cluster (or any other OpenShift/Kubernetes cluster, configured to access the same backend storage):
8. Pre-requisite for the access is to create a new namespace, Persistent Volume (PV), Persistent Volume Claim (PVC) and create a pod with respective user credentials.
As we have already created the namespace under use case 1 (step 10(i)), use the same one for this. Let us create new PV, PVC and pod for the static path.
i) Create the PV with the required directory path that needs to be accessed from the pod:
$ cat pv-nfs-hpo-posix-csi-share.yaml apiVersion: v1 kind: PersistentVolume metadata: name: static-pv-nfs-hpo-csi-10005 spec: capacity: storage: 1Gi accessModes: - ReadWriteMany csi: driver: spectrumscale.csi.ibm.com volumeHandle: 0;0;601738680925587765;CE1C0B0A:642E8A09;;;/mnt/remote-sample/nfsuser10005-dir/nfscreated-dir-for-das-share
$ oc apply -f ./pv-nfs-hpo-posix-csi-share.yaml persistentvolume/static-pv-nfs-hpo-csi-10005 created
$ oc get pv static-pv-nfs-hpo-csi-10005 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE static-pv-nfs-hpo-csi-10005 1Gi RWX Retain Available 6s
|
ii) Create the PVC to bound to the above PV
$ cat pvc-nfs-hpo-posix-csi-share.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: static-pvc-nfs-hpo-csi-user10005 spec: volumeName: static-pv-nfs-hpo-csi-10005 accessModes: - ReadWriteMany resources: requests: storage: 1Gi
$ oc apply -f ./pvc-nfs-hpo-posix-csi-share.yaml persistentvolumeclaim/static-pvc-nfs-hpo-csi-user10005 created
$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE static-pvc-nfs-hpo-csi-user10005 Pending static-pv-nfs-hpo-csi-10005 0 2s
$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE static-pvc-nfs-hpo-csi-user10005 Bound static-pv-nfs-hpo-csi-10005 1Gi RWX 16s
|
iii) Create a pod to access the directory that got available with the above PVC
$ cat pod-nfs-hpo-posix-csi.yaml apiVersion: v1 kind: Pod metadata: name: nfs-hpo-posix-csi-user10005 spec: containers: - name: sec-ctx-demo-2 image: busybox:1.28 command: [ "sh", "-c", "sleep 1h"] securityContext: runAsUser: 10005 runAsGroup: 11000 fsGroup: 6000 volumeMounts: - name: mypvc mountPath: /tmp/busy volumes: - name: mypvc persistentVolumeClaim: claimName: static-pvc-nfs-hpo-csi-user10005 readOnly: false
$ oc apply -f ./pod-nfs-hpo-posix-csi.yaml pod/nfs-hpo-posix-csi-user10005 created.
$ oc get pod NAME READY STATUS RESTARTS AGE nfs-hpo-posix-csi-user10005 0/1 ContainerCreating 0 7s
$ oc get pod NAME READY STATUS RESTARTS AGE nfs-hpo-posix-csi-user10005 1/1 Running 0 21s
|
9. Login to the pod for accessing the file to read and update
$ oc exec --stdin --tty nfs-hpo-posix-csi-user10005 -- /bin/sh
$ df -h /tmp/busy Filesystem Size Used Available Use% Mounted on remote-sample 3.9T 134.9G 3.8T 3% /tmp/busy
$ ls -lrt /tmp/busy/ total 1 -rw-rw---- 1 10005 11000 161 May 26 05:32 nfsuser-data-share
/tmp/busy $ id uid=10005(10005) gid=11000 groups=11000
/tmp/busy $ echo "this is an update from the CSI Pod with user id 10005 " >> ./nfsuser-data-share
/tmp/busy $ ls -lrt total 1 -rw-rw---- 1 10005 11000 217 May 26 05:53 nfsuser-data-share
/tmp/busy $ cat nfsuser-data-share this is new file created from NFS client node as nfsuser10005 this is das update from S3 user side this is an update from the POSIX client as the user id 10005 this is an update from the CSI Pod with user id 10005
|
Standalone NFS client:
10. Finally re-read the file and check the size from NFS client
$ whoami nfsuser10005
$ id uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ pwd /mnt/nfsuser10005-dir-share/nfscreated-dir-for-das-share
$ cat nfsuser-data-share this is new file created from NFS client node as nfsuser10005 this is das update from S3 user side this is an update from the POSIX client as the user id 10005 this is an update from the CSI Pod with user id 10005
$ ls -lrt total 1 -rw-rw---- 1 nfsuser10005 nfsgrp11000 217 May 25 22:53 nfsuser-data-share
|
In this scenario, the data sharing has been performed via NFS to DAS-S3 with all environments in-between.
Conclusion:
As per the above use cases, data access could be provided across varied environments (NFS, S3, POSIX, CSI) with read/write operations, by integrating the IBM Storage Scale DAS Solution with IBM Storage Scale environment.
Limitation:
Providing concurrent write data access across the environments is a challenge and it needs to be addressed through other S3 and NFS protocol stack features.
References:
[1] https://www.ibm.com/docs/en/ssdas?topic=spectrum-scale-data-access-services-517
[2] https://www.ibm.com/docs/en/scalecontainernative?topic=spectrum-scale-container-native-storage-access-517
[3] https://www.ibm.com/docs/en/storage-scale/5.1.7?topic=overview-spectrum-scale
[4] https://www.ibm.com/docs/en/spectrum-scale-csi
[5] https://www.ibm.com/docs/en/storage-scale/5.1.7?topic=configuring-ces-protocol-configuration
[6] https://www.redhat.com/en/technologies/cloud-computing/openshift/container-platform
[7] https://www.ibm.com/docs/en/spectrum-scale-csi?topic=driver-static-provisioning
#Highlights#Highlights-home