Authors: @Madhu Punjabi @VENKATESWARA RAO PUVVADA
IBM Spectrum Scale CSI driver allows IBM Spectrum Scale to be used as persistent storage for containerised applications. Using the IBM Spectrum Scale CSI driver, Kubernetes persistent volumes (PVs) can be provisioned based on filesets or directories.
IBM Spectrum Scale has an asynchronous caching mechanism, Active File Management (AFM), which can be used to implement multisite use cases. Currently for using an AFM fileset with CSI driver, the AFM fileset can be static provisioned. This will ensure that data in the fileset will be used as is and not deleted after the application pod is deleted.
Let us see how to static provision a persistent volume for AFM when running stateful applications on Kubernetes or Red Hat OpenShift.
Deploying the IBM Spectrum Scale CSI Driver
To install the v2.5.0 of IBM Spectrum Scale CSI Driver, please check the IBM Documentation at https://www.ibm.com/docs/en/spectrum-scale-csi?topic=250-installation
Deploying the IBM Spectrum Scale AFMTo install AFM with IBM Spectrum Scale v5.1.3-0, please check the IBM Documentation at
https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-active-file-management-afm
Let's Get Started !
Let us create sample filesets for AFM at IBM Spectrum Scale cluster. These filesets would be referred further in CSI volume provisioning examples.
Creating a sample AFM target fileset
$ mmcrfileset fs2 afm-target --inode-space new
Fileset afm-target created with id 11 root inode 3670019.
$ mmlinkfileset fs2 afm-target -J /ibm/fs2/afm-target
Fileset afm-target linked at /ibm/fs2/afm-target
$ mmafmconfig enable /ibm/fs2/afm-target
Creating a sample AFM cache fileset
Before creating an AFM cache fileset, set one of the nodes in the IBM Spectrum Scale cluster as gateway node.
$ mmchnode --gateway -N mt-node-1.fyre.ibm.com
Wed Mar 23 01:55:08 PDT 2022: mmchnode: Processing node mt-node-1.fyre.ibm.com
mmchnode: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Create a sample AFM cache fileset
$ mmcrfileset fs1 afm-cache --inode-space new -p afmtarget=gpfs:///ibm/fs2/afm-target -p afmmode=sw
Fileset afm-cache created with id 140 root inode 16252931.
$ mmlinkfileset fs1 afm-cache -J /ibm/fs1/afm-cache
Fileset afm-cache linked at /ibm/fs1/afm-cache
Verify the the relationship between the AFM cache and target filesets
$ mmafmctl fs1 getstate
Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec
------------ -------------- ------------- ------------ ------------ -------------
afm-cache gpfs:///ibm/fs2/afm-target Active mt-node-1 0 2
Create a sample directory under the AFM cache fileset
$ mkdir /ibm/fs1/afm-cache/sample
$ ls /ibm/fs2/afm-target/
sample
Creating PV and PVC for using AFM fileset with IBM Spectrum Scale CSI driver
Now let us create static provisioned PV and PVC for using the AFM cache fileset on a Kubernetes or Redhat Openshift cluster.
Sample manifests for static provisioning of fileset based volume
To generate static provisioning manifests (PV and PVC), download the following script:
$ curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.5.0/tools/generate_static_provisioning_yamls.sh
To know how to use this script, please check https://www.ibm.com/docs/en/spectrum-scale-csi?topic=provisioning-generating-static-manifests
Below command shows an example to generate manifests for PV and PVC for the fileset 'afm-cache' within the filesystem 'fs1':
$ ./generate_static_provisioning_yamls.sh --filesystem fs1 --fileset afm-cache --size 10 --pvname afmstaticpv --guihost mt-csi-scalegui
GUI Username: admin
GUI Password:
INFO: volumeHandle: 0;2;8809226123290901444;0A0B42BC:60D87CA7;;afm-cache;/ibm/fs1/afm-cache
INFO: Successfully created afmstaticpv.yaml
INFO: Successfully created pvc-afmstaticpv.yaml
Creating a Persistent Volume (PV)
Issue the below command to create a persistent volume (PV) using the manifest created in the previous step
$ kubectl apply -f afmstaticpv.yaml
persistentvolume/afmstaticpv created
Note: All YAML files presented in this blog can be applied with kubectl apply -f <filename> on Kubernetes. Replace ‘kubectl’ with ‘oc’ for Red Hat OpenShift.
Creating a Persistent Volume Claim (PVC)
Update the manifest generated for creating PVC in a previous step to comment the below line or specify a namespace. When namespace is not specified, the PVC will get created in the current namespace.
# namespace: <PVC namespace>
Issue the below command to create a persistent volume claim (PVC) using the manifest updated in the previous step
$ kubectl apply -f pvc-afmstaticpv.yaml
persistentvolumeclaim/pvc-afmstaticpv created
Creating a pod
Below example shows a sample pod definition for creating a nginx container using the PVC created for the AFM cache fileset.
$ cat afm-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-afm-staticdemo-pod
labels:
app: nginx
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- name: mypvc
mountPath: /usr/share/nginx/html/scale
ports:
- containerPort: 80
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: pvc-afmstaticpv
readOnly: false
Issue below command to create the pod.
$ kubectl apply -f afm-pod.yaml
Issue below commands to access the container inside the pod to read and write at the mounted path.
$ kubectl exec -it csi-afm-staticdemo-pod -c web-server -- bash
root@csi-afm-staticdemo-pod:/# cd /usr/share/nginx/html/scale
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# ls
sample
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# mkdir dirA
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# ls
dirA sample
Now verify the contents of the AFM cache fileset and AFM target fileset at the Spectrum Scale cluster.
$ cd /ibm/fs1/afm-cache
[root@mt-csi-scalegui 2022_04_05-18:18:36 afm-cache]$ ls
dirA sample
$ ls /ibm/fs2/afm-target
dirA sample
For more information on static provisioning of fileset or directory please refer https://www.ibm.com/docs/en/spectrum-scale-csi?topic=driver-static-provisioning
For information on creating pods referencing the persistent volume claim (PVC) check https://www.ibm.com/docs/en/spectrum-scale-csi?topic=provisioning-creating-pods
References:1. IBM Spectrum Scale Container Storage Interface Driver Documentation
https://www.ibm.com/docs/en/spectrum-scale-csi2. IBM Spectrum Scale AFM Documentation
https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-active-file-management-afm#afm
#csi#spectrumscale
#redhatopenshift