Using external storages for MQ Containers
An IBM MQ container can run without persisting data. i.e., its data will be limited to the lifetime of the container until its shutdown. These are often referred to as stateless containers and do not save any data. But this is not the usual way of dealing with IBM MQ in production container environments. MQ containers typically use storage to save data that can be accessed or retained after the container is stopped or restarted.
So in case of containers using persisted data, a fundamental requirement for the successful operation of IBM MQ running in such containers is ensuring that it has necessary permissions to write its files. When these permissions are absent, functioning of IBM MQ is negatively affected resulting into potential disruption in a messaging infrastructure. This demand becomes particularly important when one uses external storage solutions that have their own set permission settings and configurations.
Permissions issues with external storages in OpenShift Container Platform(OCP) often stem from mismatches in user and group IDs, security policies, or SELinux settings or SCCs(Storage Class Constraints). These issues can prevent pods from accessing the required storage, leading to application failures or degraded performance.
External storages use uid (user id) and gid (group id) for file permissions. If the uids/gids on the OpenShift nodes don't match those on the provider’s file system, permission issues can arise.
There are external storages classes like the nfs storage providers or AWS’s efs-sc or ibmc-file-gold-gid from IBM, Azure’s storage classes & so on. In such cases of using external storages, one may be required to specify a supplementalGroup and/or an fsGroup in order for a seamless functioning.
An example of this kind can be found in this section of IBM Knowledge Center, where the ibmc-file-gold-gid has been used as the storage class and an appropriate value for supplementalGroups has been set. The value being 65534. This is to give MQ the correct permissions on NFS volumes underlying the storage class. Else the container will fail with the error, AMQ7077E: You are not authorized to perform the requested operation.
When logging is set to debug mode, the user may also see errors like below,
Detected 'nfs4' volume mounted to /mnt/mqm
Using queue manager name: QM1
mkdir /mnt/mqm/data: permission denied ,
which is a clear indication of permissions issues on respective storage class used.
The API reference for QueueManager page has a separate section - API reference for QueueManager (mq.ibm.com/v1beta1), for securityContext where many such helpful attributes are made available in QueueManager yaml. Either using one of them or combination of these would be helpful for anyone facing permissions issue related to external storages.
- And lastly according to this support page - https://www.ibm.com/support/pages/testing-statement-ibm-mq-multi-instance-queue-manager-file-systems, it provides details for someone to know what file systems work and which all do not work for MQ containers. The page also lists the ones which were tested with the product.
Important: For any external storage class used, a user has to reach out to their storage provider in order to know what are the right values are for fsGroup and supplementalGroup.
There are instances where one can find out the values for these by debugging the container and know the ownership groups that were set by the PVC and thus which supplementalGroups they needed to set to grant write access to the container.
In that aspect, the following can come handy. This OCP(OpenShift Container Platform) v4.14’s documentation contains in detail about NFS storages and about creation of fsgroups/supplemental groups for storage usage - Persistent storage using NFS
In case of AWS using Docker Compose
If you are trying to setup an MQ container on AWS using Docker Compose, without external storage, it may seem ok, but with external storage one would encounter the problems discussed before. It will throw the permission denied error and after which docker compose decides to tear everything down.
For this scenario, there is a already a way to handle insecure storages. One has to run an init container to prepare the storage by setting the necessary permissions. i.e., run the container with a “-i” argument (i.e. runmqserver -i), which changes the storage permissions and exits. Later the main container can just run as normal. The step by step procedure has been documented in the following github page - Using Docker Compose to deploy IBM MQ Queue Manager
Other best practices for the discussed topic include,
1) Ensuring SCCs are configured to allow the necessary permissions for pods
2) Network & firewall settings between the OCP cluster & NFS server
3) Monitoring the mount points and setting of alerts for permission issues
By carefully managing these aspects, an user can mitigate permissions issues with external storages in an OpenShift environment, ensuring reliable and secure accesses.