Authors: @RAVI KUMAR KOMANDURI @Madhu Punjabi
Data is everywhere and the need for Data Management and accessibility in Distributed/Hybrid (On-/Off-Prem) Computing environments is playing a key role in the current era. High Performance Computing environments (Seismic, Computational Fluid Dynamics, Aerospace, Weather Forecasting etc.) refine the data for performing Computational or Analytics workloads to deliver data insights. IBM Storage Scale Data Access Services (DAS S3) [1] is a Containerized S3 Object protocol solution, built on top of IBM Storage Scale Containerized Native Storage Access [2], which accesses underlying IBM Storage Scale [3] File system. In-order to address emerging industry requirements on data sharing, a comprehensive data access is being solutioned via multi-protocol enablement across (DAS S3, NFS, POSIX and IBM Spectrum Scale Container Storage Interface (CSI) [4]).
This blog describes use cases of data sharing across multiple sites using DAS S3 with CSI, IBM Storage Scale (with NFS and POSIX client)[5]. DAS S3 is an object storage solution that runs in a secure environment on top of Red Hat OpenShift Container Platform [6] with IBM Storage Scale file system serving as the backend storage. Let us now delve into further details on multi-protocol data access between DAS(S3) - NFS - CSI - POSIX.
Note: SELinux is disabled in IBM Storage Scale cluster to provide the multi-protocol data access capability
Note: In the blog we will refer to DAS as DAS-S3 Cluster and a second cluster termed as NFS - CSI - DAS Cluster.
Figure 1: DAS-S3 Cluster that is deployed and accessing the IBM Storage Scale as backend Storage Cluster (SELinux is Disabled on Storage Cluster)
Note: Two key considerations to note from the above figure: 1) Access/Secret keys, and 2) SELinux disabled in the backend Storage Cluster.
With these data points at hand, the IBM Storage Scale Cluster has NFS protocol (NFS-Ganesha Server) configured, and the same data can be accessed with NFS client and DAS S3. Similarly, the same data on the IBM Storage Scale Cluster can be accessed with POSIX and DAS-S3. IBM Storage Scale CSI can be on a separate cluster, or it can be part of the same DAS-S3 cluster with appropriate user credentials configured. In the Figure 2, SELinux is disabled on IBM Storage Cluster to have multi-protocol configuration enabled for data sharing.
Figure 2: DAS-S3 Cluster and IBM Storage Scale cluster with protocols enabled using the same backend Storage Scale file system with SELinux Disabled.
Let us now discuss two use cases in detail.
Use case 1: Write data from DAS S3 cluster, read and update from (POSIX, NFS, CSI) environments and finally re-read from DAS Cluster
The primary goal in this case is the S3 application creates objects (i.e uses S3 PUT) which the DAS-S3 cluster stores as file in the backend Scale File System. It is further illustrated how the different environments (POSIX, NFS and CSI) are accessing the same data and perform operations. With POSIX (NSD client), the user of the Storage Scale Cluster will access the data with the required credentials and does read/write. Further the same directory is exported via NFS-Ganesha protocol on the Scale environment, in order to provide access for the users via the NFS client. Further on the data can be accessed by CSI (with static provisioning [7]) with appropriate user credentials. Finally, the DAS-S3 Cluster will be able to re-read the data after all updates.
Step wise procedure:
Listed below are the steps that are required to be performed in each environment to achieve the data sharing capability.
Storage Cluster:
1 Check the SELinux setting.
2 Create the directory on the Storage Cluster with appropriate permissions for DAS S3 user.
|
$ ls -lad /ibm/fs1/s3user10005-dir
drwxr-xr-x 2 root root 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
$ chmod 770 /ibm/fs1/s3user10005-dir
$ chown 10005:11000 /ibm/fs1/s3user10005-dir
$ ls -lad /ibm/fs1/s3user10005-dir
drwxrwx--- 2 10005 11000 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
|
3. Create the user on the Storage Cluster for POSIX access to the data directory
|
$ id nfsuser10005
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ ls -lad /home/nfsuser10005
drwx------ 2 nfsuser10005 nfsgrp11000 90 May 24 03:32 /home/nfsuser10005
$ ls -lad /ibm/fs1/s3user10005-dir
drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 24 03:23 /ibm/fs1/s3user10005-dir
|
DAS-S3 Cluster:
4. Create S3 user on the OpenShift cluster with respective uid, gid along with option '--newBucketsPath'
| $ mmdas account create s3user10005-hpo@ibm.com --uid 10005 --gid 11000 --newBucketsPath "/mnt/remote-sample/s3user10005-dir" |
|
$ s3u10005 mb s3://data-share-das-posix-nfs-csi
make_bucket: data-share-das-posix-nfs-csi
$ echo "this is file written from hpo as user 10005" > ./object-das-posix-nfs-csi
$ s3u10005 cp /root/object-das-posix-nfs-csi s3://data-share-das-posix-nfs-csi
upload: ./object-das-posix-nfs-csi to s3://data-share-das-posix-nfs-csi/object-das-posix-nfs-csi
$ s3u10005 ls s3://data-share-das-posix-nfs-csi
2023-05-24 04:00:58 44 object-das-posix-nfs-csi
|
Storage Cluster (via NSD client for POSIX access):
6. As the nfsuser10005 user that is created in Step 3, read and update the object
|
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami
nfsuser10005
$ ls -lrt /ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi/
total 1
-rw-rw---- 1 nfsuser10005 nfsgrp11000 44 May 24 04:00 object-das-posix-nfs-csi
$ echo "this is an update from posix client as the nfsuser10005" >> ./object-das-posix-nfs-csi
$ ls -lrt
total 1
-rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 24 06:00 object-das-posix-nfs-csi
|
7. Create a NFS-Ganesha export on the cluster.
Note: Reference to the IBM Storage Scale documentation
$ mmnfs export add /ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi --client "*(Access_Type=RW,Protocols=3:4,Squash=no_root_squash,anonymous_uid=10005,anonymous_gid=11000,Transports=TCP:UDP)"
mmnfs: The NFS export was created successfully.
|
Standalone NFS client:
8. On the NFS client system as shown in fig 2, the system administrator would mount the respective NFS export from Storage Cluster via CES IP and create the NFS user with required credentials
|
$ mount -o vers=3 -t nfs 10.11.28.204:/ibm/fs1/s3user10005-dir/data-share-das-posix-nfs-csi /mnt/nfsuser10005-data-share
Note: NFS client can mount with vers=4.0 as well
$ id
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami
nfsuser10005
$ cd /mnt/nfsuser10005-data-share/
$ pwd
/mnt/nfsuser10005-data-share
$ ls -lrt
total 1
-rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 24 06:00 object-das-posix-nfs-csi
9. Read and update the file as nfsuser10005 user
$ echo "this is an update from the NFS client user nfs10005" >> ./object-das-posix-nfs-csi
$ cat object-das-posix-nfs-csi
this is file written from hpo as user 10005
this is an update from posix client as the nfsuser10005
this is an update from the NFS client user nfs10005
$ ls -lrt object-das-posix-nfs-csi
-rw-rw---- 1 nfsuser10005 nfsgrp11000 152 May 24 06:44 object-das-posix-nfs-csi
|
CSI namespace access with the userid 10005 on the DAS cluster (or any other CSI cluster, configured to access the same backend Storage):
Note: Refer to the IBM Storage Scale CSI documentation for creating static volumes
10. Pre-requisite for the access is to create the new namespace then Persistent Volume(PV) , Persistent Volume Claim(PVC) and pod creation with respective user credentials
i) Create a new project and describe the namespace
|
$ oc describe ns hpo-nfs-csi-posix-datashare
Name: hpo-nfs-csi-posix-datashare
Labels: kubernetes.io/metadata.name=hpo-nfs-csi-posix-datashare
pod-security.kubernetes.io/audit=restricted
pod-security.kubernetes.io/audit-version=v1.24
pod-security.kubernetes.io/warn=restricted
pod-security.kubernetes.io/warn-version=v1.24
Annotations: openshift.io/description:
openshift.io/display-name:
openshift.io/requester: kube:admin
openshift.io/sa.scc.mcs: s0:c27,c14
openshift.io/sa.scc.supplemental-groups: 1000730000/10000
openshift.io/sa.scc.uid-range: 1000730000/10000
Status: Active
No resource quota.
No LimitRange resource.
|
ii) Create the persistent volume(pv) for the directory path that is required for access and apply the pv
|
$ cat pv-hpo-nfs-csi-posix.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-pv-hpo-csi-nfsuser10005
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
csi:
driver: spectrumscale.csi.ibm.com
volumeHandle: 0;0;601738680925587765;CE1C0B0A:642E8A09;;;/mnt/remote-sample/s3user10005-dir/data-share-das-posix-nfs-csi
$ oc apply -f ./pv-hpo-nfs-csi-posix.yaml
persistentvolume/static-pv-hpo-csi-nfsuser10005 created
$ oc get pv static-pv-hpo-csi-nfsuser10005
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
static-pv-hpo-csi-nfsuser10005 1Gi RWX Retain Available 7s
|
iii) Create a PVC which will bound to the pv
|
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: static-pvc-hpo-csi-fornfsuser10005
spec:
volumeName: static-pv-hpo-csi-nfsuser10005
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
$ oc apply -f ./pvc-hpo-nfs-csi-posix.yaml
persistentvolumeclaim/static-pvc-hpo-csi-fornfsuser10005 created.
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
static-pvc-hpo-csi-fornfsuser10005 Bound static-pv-hpo-csi-nfsuser10005 1Gi RWX 3s
|
iv) Create the Pod with user id and group id to access the directory provided via the above PVC (showing only user group)
apiVersion: v1
kind: Pod
metadata:
name: csi-hpo-datashare-nfsuser10005
spec:
containers:
- name: sec-ctx-demo-2
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h"]
securityContext:
runAsUser: 10005
runAsGroup: 11000
fsGroup: 6000
volumeMounts:
- name: mypvc
mountPath: /tmp/busy
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: static-pvc-hpo-csi-fornfsuser10005
readOnly: false
$ oc apply -f ./pod-hpo-nfs-csi-posix.yaml
$ oc get pods
NAME READY STATUS RESTARTS AGE
csi-hpo-datashare-nfsuser10005 1/1 Running 0 57s |
|
$ df -h /tmp/busy
Filesystem Size Used Available Use% Mounted on
remote-sample 3.9T 134.9G 3.8T 3% /tmp/busy
$ umask
0022
$ ls -lrt /tmp/busy
total 1
-rw-rw---- 1 10005 11000 152 May 24 13:44 object-das-posix-nfs-csi
/tmp/busy $ id
uid=10005(10005) gid=11000 groups=11000
/tmp/busy $ whoami
10005
/tmp/busy $ echo "this is an update from the CSI pod as user id 10005 and group id 11000" >>./object-das-posix-nfs-csi
/tmp/busy $ ls -lrt
total 1
-rw-rw---- 1 10005 11000 223 May 25 06:08 object-das-posix-nfs-csi
|
DAS-S3 Cluster:
12. Now re-read the object via S3 user with the required access and secret key
2023-05-24 23:08:32 223 object-das-posix-nfs-csi |
Use case 2: Create a file from NFS client, update the file from (DAS-S3 cluster, POSIX, CSI) environments and finally re-read from NFS client.
Scenario is to create a file from NFS client system which has mounted an NFS export created in IBM Storage Scale cluster using NFS-Ganesha protocol, access from DAS-S3 cluster as S3 user to perform the operations. Further access via POSIX (NSD client) with appropriate user credentials and then via the CSI environment and then finally re-read from NFS environment.
Storage Cluster:
1. Create a directory with appropriate user id and group id set to NFS user nfsuser10005 and group 11000
|
$ chmod 770 nfsuser10005-dir
$ chown 10005:11000 nfsuser10005-dir
$ ls -lrtd nfsuser10005-dir
drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 25 01:39 nfsuser10005-dir
|
2. Make the directory available as a NFS export from the cluster
|
mmnfs: The NFS export was created successfully.
$ mmnfs export list --nfsdefs /ibm/fs1/nfsuser10005-dir
Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort DefaultDelegations Manage_Gids NFS_Commit Pseudo Path
------------------------- ----------- ------- ----------- --------- ---------- -------------- ------------- ------------- ------- -------------- ------------------ ----------- ---------- -------------------------
/ibm/fs1/nfsuser10005-dir NONE * RW 3,4 TCP,UDP NO_ROOT_SQUASH 10005 11000 SYS FALSE NONE FALSE FALSE /ibm/fs1/nfsuser10005-dir
|
Standalone NFS client:
3. System administrator will mount the NFS exported directory on the NFS client system
$ mount -o vers=4.0 -t nfs 10.11.28.206:/ibm/fs1/nfsuser10005-dir /mnt/nfsuser10005-dir-share
Note: NFS client can mount with vers=3.0 as well |
4. Create a sub-directory and a file under it
|
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ whoami
nfsuser10005
$ mkdir nfscreated-dir-for-das-share
$ ls -lrt
total 1
drwxr-xr-x 2 nfsuser10005 nfsgrp11000 4096 May 25 01:48 nfscreated-dir-for-das-share
$ chmod 770 nfscreated-dir-for-das-share
$ ls -lrt
total 1
drwxrwx--- 2 nfsuser10005 nfsgrp11000 4096 May 25 01:48 nfscreated-dir-for-das-share
$ echo "this is new file created from NFS client node as nfsuser10005" > ./nfsuser-data-share
$ ls -lrt
total 1
-rw-r--r-- 1 nfsuser10005 nfsgrp11000 62 May 25 01:50 nfsuser-data-share
$ umask
0022
|
DAS-S3 Cluster:
5. Create a bucket using 'mmdas export' CLI command for the same directory, which was made available as NFS export, step 4 from above:
|
Export is successfully created.
$ mmdas export list das-nfs-share
Name Filesystem Path
----- -----------------
das-nfs-share /mnt/remote-sample/nfsuser10005-dir/nfscreated-dir-for-das-share/
|
6. List the file inside the S3 bucket and download the file. Modify and upload the file from S3 user perspective.
|
$ s3u10005 ls s3://das-nfs-share
2023-05-25 01:50:50 62 nfsuser-data-share
$ s3u10005 cp s3://das-nfs-share/nfsuser-data-share /tmp/nfsuser-data-share
$ echo "this is das update from S3 user side " >> /tmp/nfsuser-data-share
$ ls -lrt /tmp/nfsuser-data-share
-rw-r--r-- 1 root root 100 May 25 05:00 /tmp/nfsuser-data-share
$ s3u10005 cp /tmp/nfsuser-data-share s3://das-nfs-share
$ s3u10005 ls s3://das-nfs-share
2023-05-25 05:00:57 100 nfsuser-data-share
|
7. From NSD client, read and update the content of the file with respective user credentials
|
$ id
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ pwd
/ibm/fs1/nfsuser10005-dir/nfscreated-dir-for-das-share
$ ls -lrt nfsuser-data-share
-rw-rw---- 1 nfsuser10005 nfsgrp11000 100 May 25 05:00 nfsuser-data-share
$ echo "this is an update from the POSIX client as the user id 10005" >> ./nfsuser-data-share
$ ls -lrt
total 1
-rw-rw---- 1 nfsuser10005 nfsgrp11000 161 May 25 22:32 nfsuser-data-share
|
CSI namespace access with the userid 10005 on the DAS-S3 cluster (or any other OpenShift/Kubernetes cluster, configured to access the same backend storage):
8. Pre-requisite for the access is to create a new namespace, Persistent Volume (PV), Persistent Volume Claim (PVC) and create a pod with respective user credentials.
As we have already created the namespace under use case 1 (step 10(i)), use the same one for this. Let us create new PV, PVC and pod for the static path.
i) Create the PV with the required directory path that needs to be accessed from the pod:
|
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-pv-nfs-hpo-csi-10005
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
csi:
driver: spectrumscale.csi.ibm.com
volumeHandle: 0;0;601738680925587765;CE1C0B0A:642E8A09;;;/mnt/remote-sample/nfsuser10005-dir/nfscreated-dir-for-das-share
$ oc apply -f ./pv-nfs-hpo-posix-csi-share.yaml
persistentvolume/static-pv-nfs-hpo-csi-10005 created
$ oc get pv static-pv-nfs-hpo-csi-10005
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
static-pv-nfs-hpo-csi-10005 1Gi RWX Retain Available 6s
|
ii) Create the PVC to bound to the above PV
|
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: static-pvc-nfs-hpo-csi-user10005
spec:
volumeName: static-pv-nfs-hpo-csi-10005
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
$ oc apply -f ./pvc-nfs-hpo-posix-csi-share.yaml
persistentvolumeclaim/static-pvc-nfs-hpo-csi-user10005 created
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
static-pvc-nfs-hpo-csi-user10005 Pending static-pv-nfs-hpo-csi-10005 0 2s
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
static-pvc-nfs-hpo-csi-user10005 Bound static-pv-nfs-hpo-csi-10005 1Gi RWX 16s
|
iii) Create a pod to access the directory that got available with the above PVC
|
$ cat pod-nfs-hpo-posix-csi.yaml
apiVersion: v1
kind: Pod
metadata:
name: nfs-hpo-posix-csi-user10005
spec:
containers:
- name: sec-ctx-demo-2
image: busybox:1.28
command: [ "sh", "-c", "sleep 1h"]
securityContext:
runAsUser: 10005
runAsGroup: 11000
fsGroup: 6000
volumeMounts:
- name: mypvc
mountPath: /tmp/busy
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: static-pvc-nfs-hpo-csi-user10005
readOnly: false
$ oc apply -f ./pod-nfs-hpo-posix-csi.yaml
pod/nfs-hpo-posix-csi-user10005 created.
$ oc get pod
NAME READY STATUS RESTARTS AGE
nfs-hpo-posix-csi-user10005 0/1 ContainerCreating 0 7s
$ oc get pod
NAME READY STATUS RESTARTS AGE
nfs-hpo-posix-csi-user10005 1/1 Running 0 21s
|
9. Login to the pod for accessing the file to read and update
|
$ df -h /tmp/busy
Filesystem Size Used Available Use% Mounted on
remote-sample 3.9T 134.9G 3.8T 3% /tmp/busy
$ ls -lrt /tmp/busy/
total 1
-rw-rw---- 1 10005 11000 161 May 26 05:32 nfsuser-data-share
/tmp/busy $ id
uid=10005(10005) gid=11000 groups=11000
/tmp/busy $ echo "this is an update from the CSI Pod with user id 10005 " >> ./nfsuser-data-share
/tmp/busy $ ls -lrt
total 1
-rw-rw---- 1 10005 11000 217 May 26 05:53 nfsuser-data-share
/tmp/busy $ cat nfsuser-data-share
this is new file created from NFS client node as nfsuser10005
this is das update from S3 user side
this is an update from the POSIX client as the user id 10005
this is an update from the CSI Pod with user id 10005
|
Standalone NFS client:
10. Finally re-read the file and check the size from NFS client
|
nfsuser10005
$ id
uid=10005(nfsuser10005) gid=11000(nfsgrp11000) groups=11000(nfsgrp11000)
$ pwd
/mnt/nfsuser10005-dir-share/nfscreated-dir-for-das-share
$ cat nfsuser-data-share
this is new file created from NFS client node as nfsuser10005
this is das update from S3 user side
this is an update from the POSIX client as the user id 10005
this is an update from the CSI Pod with user id 10005
$ ls -lrt
total 1
-rw-rw---- 1 nfsuser10005 nfsgrp11000 217 May 25 22:53 nfsuser-data-share
|
In this scenario, the data sharing has been performed via NFS to DAS-S3 with all environments in-between.
Conclusion:
As per the above use cases, data access could be provided across varied environments (NFS, S3, POSIX, CSI) with read/write operations, by integrating the IBM Storage Scale DAS Solution with IBM Storage Scale environment.
Limitation:
Providing concurrent write data access across the environments is a challenge and it needs to be addressed through other S3 and NFS protocol stack features.
References:
[1] https://www.ibm.com/docs/en/ssdas?topic=spectrum-scale-data-access-services-517
[2] https://www.ibm.com/docs/en/scalecontainernative?topic=spectrum-scale-container-native-storage-access-517
[3] https://www.ibm.com/docs/en/storage-scale/5.1.7?topic=overview-spectrum-scale
[4] https://www.ibm.com/docs/en/spectrum-scale-csi
[5] https://www.ibm.com/docs/en/storage-scale/5.1.7?topic=configuring-ces-protocol-configuration
[6] https://www.redhat.com/en/technologies/cloud-computing/openshift/container-platform
[7] https://www.ibm.com/docs/en/spectrum-scale-csi?topic=driver-static-provisioning
#Highlights#Highlights-home