This post was originally posted to link

A prerequisite for this blog, you should be familiar with seccomp profiles and security context, or read my blog related to Configuring Seccomp Profile on OpenShift Container Platform for Security and Compliance on Power. Like the seccomp profile, securing attached storage also falls under container security and its main aim is to secure the attached block or shared storage.

This document outlines the OpenShift Container Platform NFS (Network File System) shared storage provider, how to mount the PersistentVolume (PV), securing and scoping the same to a single project, how to mount the NFS to container and methods by which we can restrict the usage of NFS inside container.

Persistent storage using NFS

When you deploy OpenShift Container Platform Cluster you can check whether the NFS storage has been provisioned by checking the contents of /etc/exports file which contains all the accessible NFS directories. Each NFS export is associated with POSIX Owner and Group IDs.

You request the NFS storage by either providing the PersistentVolumeClaim (PVC) name or the NFS plugin details directly in volume section in pod definition. You can check the Ownership and SELinux permissions of the NFS storage using below commands:

# cat /etc/exports
/export *(rw,sync,root_squash)

# ls -ld /export
drwxrwxrwx. 3 nobody nobody 92 Nov 21 06:33 /export

# id nobody
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

To achieve NFS volume security the container must match SELinux options, and either run with a UID 65534, the nobody owner, or with 65534 in its supplemental groups to access the directory.

NFS Volume Security

Once the NFS file system is mounted read/write by a remote host, the only protection each shared file has its permissions. By default, access control lists (ACLs) are supported by NFS under Red Hat Enterprise Linux.

Following are the series of steps to achieve the same:

1. Create a Persistent Volume (PV):

# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /export
    server: <nfs-server-ip>
  persistentVolumeReclaimPolicy: Recycle

# oc create -f pv.yaml 
persistentvolume/pv0001 created

# oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                   STORAGECLASS              REASON   AGE
pv0001                                     5Gi        RWO            Recycle          Available                                                                              14s
pvc-0134f991-8be5-406f-962f-ece86d26b21d   20Gi       RWX            Delete           Bound       openshift-image-registry/registry-pvc   nfs-storage-provisioner            6d22h

2. Create a Persistent Volume Claim (PVC) which will bind to above created PV:

# cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-claim1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: pv0001
  storageClassName: ""

# oc create -f pvc.yaml
persistentvolumeclaim/nfs-claim1 created

# oc get pvc
NAME         STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfs-claim1   Bound    pv0001   5Gi        RWO                           4s

After successful creation of PV having PVC bounded to it, you use it in Pod definition.

For the cluster administrator, granting pods access to PVs involves:

• knowing the group ID and/or user ID assigned to the actual storage,
• understanding SELinux considerations, and
• ensuring that these IDs are allowed in the range of legal IDs defined for the project and/or the SCC that matches the requirements of the pod.

However, to provide group id, user id, SELinux options you must create custom SCC (Security Context Constraint) which matches the pod requirements and has permissions for the ID ranges that are specified in Pod definition. For more details of how to create custom SCC please check blogpost Lessons Learned using Security Context
Constraints OpenShift

After following the steps mentioned in above blogpost you can create the SCC that matches your requirements and can add it to custom Service Account. To attach the custom SCC to a pod, you specify the Service Account name in Pod. You can find example later in this blog.

Group ID

This is the most recommended way to handle access to NFS storage. For this you can use supplemental groups under the securityContext definition of the pod.

spec:
  containers:
    - name:
    ...
  securityContext: 
    supplementalGroups: []

Here securityContext should be specified at pod level and supplementalGroups is an array. You can specify multiple Group IDS separated by comma as per your requirement.
Note: If group ID range checking is desired, a custom SCC is the preferred solution.

User ID

This can be defined in container image or pod definition. Here the UID should match the permissions as that of NFS storage. When we specify the user id in pod or container then every process will run with that user id.

Like supplemental groups user id should also be specified under the securityContext definition of the pod.

spec:
  containers: 
  - name:
  ...
    securityContext:
      runAsUser: 65534

SELinux

To assign SELinux labels to a Container, include the seLinuxOptions field in the securityContext section of your Pod or Container manifest.

securityContext:
  seLinuxOptions:
    level: "s0:c123,c456"

Volumes that support SELinux labelling are relabeled to be accessible by the label specified under seLinuxOptions. Usually, you only need to set the level section.

Now that you know basics of how to use above parameters to secure your pod/container and how PVC will get bind to only one PV at a time for a namespace, you can then use it to secure your storage by limiting access to it by specifying specific user id, group id and SELinux options in pod definition as follows

# cat test-pod_selinux.yaml
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo-1
spec:
  securityContext:
    seLinuxOptions:   
      level: "s0:c26,c15"
    runAsUser: 3333
    supplementalGroups: [5555]
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: sec-ctx-demo-1
    image: docker.io/library/tomcat:9.0
    volumeMounts:
    - name: my-volume
      mountPath: "/data"
    securityContext:
      seLinuxOptions:
        level: "s0:c26,c15"
      runAsUser: 3333
      supplementalGroups: [5555]
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop: ["ALL"]
      allowPrivilegeEscalation: false
  serviceAccountName: my-custom-sa
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: nfs-claim1

As you can see in above example, we have specified the UID, GID, SELinux options and also used service account having custom SCC attached and also specified PVC which needs to be mounted in container.

To validate this, you can check the container logs as follows:

1. Check if custom SCC has been attached to Pod:

# oc get pod security-context-demo-1 -oyaml | grep scc
openshift.io/scc: my-custom-scc

2. Check the user id, group id and selinux options attached to container

# oc get pods -owide
NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE                                       NOMINATED NODE   READINESS GATES
security-context-demo-1         1/1     Running   0          7m29s   10.128.2.194   osa21-worker-0.storage-2111-db9d.ibm.com   <none>           <none>
# oc exec -it security-context-demo-1 -- sh
$ id
uid=3333(3333) gid=0(root) groups=0(root),5555,1000680000
$ df -h | grep data
Filesystem               Size  Used Avail Use% Mounted on
192.168.100.253:/export  300G  2.2G  298G   1% /data
$ ls -Z /data
system_u:object_r:nfs_t:s0 hsperfdata_3333  system_u:object_r:nfs_t:s0 hsperfdata_5555
$ ls -Z | grep bin
system_u:object_r:container_file_t:s0:c26,c15 bin

As you can see here all the processes inside container will have 3333 as user id, 5555 supplemental group has been added in groups, SELinux option set, NFS volume attached, and custom SCC attached.

In this post you have learned how to attach user id, supplemental group, SELinux options, NFS and custom SCC to a pod which will enhance the Pod/Container security.

Thanks for reading! I hope you found this helpful:)