File and Object Storage

 View Only

Using IBM Spectrum Scale AFM along with CSI driver !

By Madhu Punjabi posted Tue April 12, 2022 10:45 AM

  


Authors: @Madhu Punjabi @VENKATESWARA RAO PUVVADA


IBM Spectrum Scale CSI driver allows IBM Spectrum Scale to be used as persistent storage for containerised applications. Using the IBM Spectrum Scale CSI driver, Kubernetes persistent volumes (PVs) can be provisioned based on filesets or directories.

IBM Spectrum Scale has an asynchronous caching mechanism, Active File Management (AFM), which can be used to implement multisite use cases. Currently for using an AFM fileset with CSI driver, the AFM fileset can be static provisioned. This will ensure that data in the fileset will be used as is and not deleted after the application pod is deleted.

Let us see how to static provision a persistent volume for AFM when running stateful applications on Kubernetes or Red Hat OpenShift.


Deploying the IBM Spectrum Scale CSI Driver

To install the v2.5.0 of IBM Spectrum Scale CSI Driver, please check the IBM Documentation at https://www.ibm.com/docs/en/spectrum-scale-csi?topic=250-installation


Deploying the IBM Spectrum Scale AFM

To install AFM with IBM Spectrum Scale v5.1.3-0, please check the IBM Documentation at https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-active-file-management-afm

Let's Get Started !

Let us create sample filesets for AFM at IBM Spectrum Scale cluster. These filesets would be referred further in CSI volume provisioning examples.

Creating a sample AFM target fileset

$ mmcrfileset fs2 afm-target --inode-space new
Fileset afm-target created with id 11 root inode 3670019.

$ mmlinkfileset fs2 afm-target -J /ibm/fs2/afm-target
Fileset afm-target linked at /ibm/fs2/afm-target     

$ mmafmconfig enable /ibm/fs2/afm-target


Creating a sample AFM cache fileset

Before creating an AFM cache fileset, set one of the nodes in the IBM Spectrum Scale cluster as gateway node.

$ mmchnode --gateway -N mt-node-1.fyre.ibm.com
Wed Mar 23 01:55:08 PDT 2022: mmchnode: Processing node mt-node-1.fyre.ibm.com
mmchnode: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.


Create a sample AFM cache fileset

$ mmcrfileset fs1 afm-cache --inode-space new -p afmtarget=gpfs:///ibm/fs2/afm-target -p afmmode=sw 
Fileset afm-cache created with id 140 root inode 16252931.

$ mmlinkfileset fs1 afm-cache -J /ibm/fs1/afm-cache
Fileset afm-cache linked at /ibm/fs1/afm-cache

 
Verify the the relationship between the AFM cache and target filesets

$ mmafmctl fs1 getstate 
Fileset Name Fileset Target Cache State Gateway Node Queue Length Queue numExec
------------ -------------- ------------- ------------ ------------ -------------
afm-cache gpfs:///ibm/fs2/afm-target Active mt-node-1 0 2


Create a sample directory under the AFM cache fileset

$ mkdir /ibm/fs1/afm-cache/sample
$ ls /ibm/fs2/afm-target/
sample

Creating PV and PVC for using AFM fileset with IBM Spectrum Scale CSI driver

Now let us create static provisioned PV and PVC for using the AFM cache fileset on a Kubernetes or Redhat Openshift cluster.

Sample manifests for static provisioning of fileset based volume
To generate static provisioning manifests (PV and PVC), download the following script:

$ curl -O https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-csi/v2.5.0/tools/generate_static_provisioning_yamls.sh

To know how to use this script, please check https://www.ibm.com/docs/en/spectrum-scale-csi?topic=provisioning-generating-static-manifests

Below command shows an example to generate manifests for PV and PVC for the fileset 'afm-cache' within the filesystem 'fs1':

$ ./generate_static_provisioning_yamls.sh --filesystem fs1 --fileset afm-cache --size 10 --pvname afmstaticpv --guihost mt-csi-scalegui
GUI Username: admin
GUI Password:
INFO: volumeHandle: 0;2;8809226123290901444;0A0B42BC:60D87CA7;;afm-cache;/ibm/fs1/afm-cache
INFO: Successfully created afmstaticpv.yaml
INFO: Successfully created pvc-afmstaticpv.yaml


Creating a Persistent Volume (PV)
Issue the below command to create a persistent volume (PV) using the manifest created in the previous step

$ kubectl apply -f afmstaticpv.yaml 
persistentvolume/afmstaticpv created

Note: All YAML files presented in this blog can be applied with kubectl apply -f <filename> on Kubernetes. Replace ‘kubectl’ with ‘oc’ for Red Hat OpenShift.

Creating a Persistent Volume Claim (PVC)
Update the manifest generated for creating PVC in a previous step to comment the below line or specify a namespace. When namespace is not specified, the PVC will get created in the current namespace.

# namespace: <PVC namespace>


Issue the below command to create a persistent volume claim (PVC) using the manifest updated in the previous step

$ kubectl apply -f pvc-afmstaticpv.yaml 
persistentvolumeclaim/pvc-afmstaticpv created


Creating a pod
Below example shows a sample pod definition for creating a nginx container using the PVC created for the AFM cache fileset.

$ cat afm-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: csi-afm-staticdemo-pod
labels:
app: nginx
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- name: mypvc
mountPath: /usr/share/nginx/html/scale
ports:
- containerPort: 80
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: pvc-afmstaticpv
readOnly: false


Issue below command to create the pod.

$ kubectl apply -f afm-pod.yaml


Issue below commands to access the container inside the pod to read and write at the mounted path.

$ kubectl exec -it csi-afm-staticdemo-pod -c web-server -- bash 
root@csi-afm-staticdemo-pod:/# cd /usr/share/nginx/html/scale
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# ls
sample
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# mkdir dirA
root@csi-afm-staticdemo-pod:/usr/share/nginx/html/scale# ls
dirA sample


Now verify the contents of the AFM cache fileset and AFM target fileset at the Spectrum Scale cluster.

$ cd /ibm/fs1/afm-cache
[root@mt-csi-scalegui 2022_04_05-18:18:36 afm-cache]$ ls
dirA sample

$ ls /ibm/fs2/afm-target
dirA sample


For more information on static provisioning of fileset or directory please refer https://www.ibm.com/docs/en/spectrum-scale-csi?topic=driver-static-provisioning
For information on creating pods referencing the persistent volume claim (PVC) check https://www.ibm.com/docs/en/spectrum-scale-csi?topic=provisioning-creating-pods


References:

1. IBM Spectrum Scale Container Storage Interface Driver Documentation
https://www.ibm.com/docs/en/spectrum-scale-csi

2. IBM Spectrum Scale AFM Documentation 
https://www.ibm.com/docs/en/spectrum-scale/5.1.3?topic=reference-active-file-management-afm

#afm #csi#spectrumscale #redhatopenshift
​​​​​​​​​​​
0 comments
30 views

Permalink