Cloud Pak for Integration

 View Only
Expand all | Collapse all

Permission denied on filesystem - CP4I MQ 9.3.0.0-r1 MultiInstance - IBM Cloud ROKS 4.10

  • 1.  Permission denied on filesystem - CP4I MQ 9.3.0.0-r1 MultiInstance - IBM Cloud ROKS 4.10

    Posted Fri November 11, 2022 09:23 AM
    I'm trying to deploy MQ 9.3.0.0-r1 in MultiInstance mode but when pod starts I receiving the following error:

    Error 71 creating queue manager: AMQ6239E: Permission denied attempting to access filesystem location
    '/mnt/mqm/data/mqs.ini'.
    AMQ7062E: Permission denied attempting to access an INI file.

    I followed this official guide to configure the storage in the correct way: link

    My environment:

    • IBM Cloud ROKS 4.10
    • IBM CP4I v2022.2.1
    • MQ image version: cp.icr.io/cp/ibm-mqadvanced-server-integration:9.3.0.0-r1
    • MQ Operator Version: 2.0.0

    My QueueManager instance:
    apiVersion: mq.ibm.com/v1beta1
    kind: QueueManager
    metadata:
      name: test
      namespace:test
    spec:
      license:
        accept: true
        license: L-RJON-CD3JKX
        use: Production
      pki:
        keys:
          - name: servercert
            secret:
              items:
                - personalcert.key
                - personalcert.crt
              secretName: mq-personal-cert
        trust:
          - name: intermediate
            secret:
              items:
                - ca.crt
              secretName: mq-ca-signcert
      queueManager:
        metrics:
          enabled: false
        availability:
          type: MultiInstance
        name: QM_PROD
        storage:
          defaultClass: ibmc-file-gold-gid
          persistedData:
            class: ibmc-file-gold-gid
            enabled: true
            size: 20Gi
            type: persistent-claim
          queueManager:
            class: ibmc-file-gold-gid
            size: 20Gi
            type: persistent-claim
          recoveryLogs:
            class: ibmc-file-gold-gid
            enabled: true
            size: 20Gi
            type: persistent-claim
      securityContext:
        initVolumeAsRoot: false
        supplementalGroups:
          - 99
      template:
        pod:
          containers:
            - env:
                - name: MQSNOAUT
                  value: 'yes'
                - name: DEBUG
                  value: 'true'
              name: qmgr
              resources: {}
      version: 9.3.0.0-r1
      web:
        enabled: true​


    Persistent volume (automatically provided) in which error occurs:
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      annotations:
        ibm.io/provisioning-status: >-
          {"status":"complete","time":"2022-11-11T10:56:44Z","attempt":1,"retry":false,"pluginid":"ibm-file-plugin-1667815828","pvcid":"74a4586c-e17a-4778-b643-c204be4bcc48"}
        pv.kubernetes.io/bind-completed: 'yes'
        pv.kubernetes.io/bound-by-controller: 'yes'
        volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-file
        volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-file
      name: data-test-ibm-mq-0
      namespace: test
      finalizers:
        - kubernetes.io/pvc-protection
      labels:
        app.kubernetes.io/component: integration
        app.kubernetes.io/instance: test
        app.kubernetes.io/managed-by: operator
        app.kubernetes.io/name: ibm-mq
        region: eu-de
        zone: mil01
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      volumeName: pvc-74a4586c-e17a-4778-b643-c204be4bcc48
      storageClassName: ibmc-file-gold-gid
      volumeMode: Filesystem
    status:
      phase: Bound
      accessModes:
        - ReadWriteOnce
      capacity:
        storage: 20Gi
    ​


    MQ Pod Logs:

    2022-11-11T10:57:03.562Z CPU architecture: amd64
    2022-11-11T10:57:03.562Z Linux kernel version: 4.18.0-372.26.1.el8_6.x86_64
    2022-11-11T10:57:03.562Z Container runtime: kube
    2022-11-11T10:57:03.562Z Base image: Red Hat Enterprise Linux 8.6 (Ootpa)
    2022-11-11T10:57:03.563Z Running as user ID 1000650000 with primary group 0, and supplementary groups 99,65531,1000650000
    2022-11-11T10:57:03.563Z Capabilities: none
    2022-11-11T10:57:03.563Z seccomp enforcing mode: disabled
    2022-11-11T10:57:03.563Z Process security attributes: system_u:system_r:container_t:s0:c0,c26
    2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm-log
    2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm
    2022-11-11T10:57:03.564Z Detected 'nfs4' volume mounted to /mnt/mqm-data
    2022-11-11T10:57:03.564Z Multi-instance queue manager: enabled
    2022-11-11T10:57:03.564Z Integration-Image created: 2022-06-16T10:35:04+00:00
    2022-11-11T10:57:03.564Z Integration-Image tag: ibm-mqadvanced-server-integration:9.3.0.0-r1.20220616103034.e5ef27d-amd64
    2022-11-11T10:57:03.582Z Open Tracing is disabled
    2022-11-11T10:57:03.613Z Using queue manager name: QM_PROD
    2022-11-11T10:57:03.648Z DEBUG: --- Start Diagnostics ---
    2022-11-11T10:57:03.660Z DEBUG: /mnt/:
    total 12
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log
    
    2022-11-11T10:57:03.671Z DEBUG: /mnt/mqm:
    total 4
    drwxr-xr-x. 2 1000650000 nobody 4096 Nov 11 10:57 data
    
    2022-11-11T10:57:03.676Z DEBUG: /mnt/mqm/data:
    total 0
    
    2022-11-11T10:57:03.682Z DEBUG: /mnt/mqm-log/log:
    total 0
    
    2022-11-11T10:57:03.688Z DEBUG: /mnt/mqm-data/qmgrs:
    total 0
    
    2022-11-11T10:57:03.694Z DEBUG: /var/mqm:
    lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data
    
    2022-11-11T10:57:03.703Z DEBUG: /var/mqm/errors:
    ls: cannot access '/var/mqm/errors': No such file or directory
    
    2022-11-11T10:57:03.719Z DEBUG: /etc/mqm:
    total 24
    -rw-r--r--. 1 1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
    drwxr-sr-x. 1 1001 root 4096 Jun 16 10:35 MQOpenTracing
    -rw-r--r--. 1 1001 root  591 Jun 15 13:07 native-ha.ini.tpl
    drwxr-sr-x. 4 root root 4096 Nov 11 10:57 pki
    drwxrwsr-x. 1 1001 root 4096 Jun 15 13:19 web
    
    2022-11-11T10:57:03.719Z DEBUG: ffstsummary:
    
    2022-11-11T10:57:03.719Z DEBUG: ---  End Diagnostics  ---
    2022-11-11T10:57:11.706Z Warning creating directory structure: 
    
    2022-11-11T10:57:11.706Z Created directory structure under /var/mqm
    2022-11-11T10:57:11.706Z DEBUG: --- Start Diagnostics ---
    2022-11-11T10:57:11.718Z DEBUG: /mnt/:
    total 12
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log
    
    2022-11-11T10:57:11.756Z DEBUG: /mnt/mqm:
    total 4
    drwxrwsr-x. 13 1000650000 nobody 4096 Nov 11 10:57 data
    
    2022-11-11T10:57:11.804Z DEBUG: /mnt/mqm/data:
    total 52
    drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 config
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 conv
    drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 errors
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits64
    drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 log
    -rw-rw-r--. 1 1000650000 nobody  571 Nov 11 10:57 mqclient.ini
    drwxrwsr-x. 5 1000650000 nobody 4096 Nov 11 10:57 mqft
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 qmgrs
    -rw-rw-r--. 1 1000650000 nobody 1941 Nov 11 10:57 service.env
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 sockets
    drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 trace
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 web
    
    2022-11-11T10:57:11.812Z DEBUG: /mnt/mqm-log/log:
    total 0
    
    2022-11-11T10:57:11.815Z DEBUG: /mnt/mqm-data/qmgrs:
    total 0
    
    2022-11-11T10:57:11.823Z DEBUG: /var/mqm:
    lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data
    
    2022-11-11T10:57:11.853Z DEBUG: /var/mqm/errors:
    total 100
    -rw-rw----. 1 1000650000 nobody 89309 Nov 11 10:57 AMQ28.0.FDC
    -rw-rw-r--. 1 1000650000 nobody  2735 Nov 11 10:57 AMQERR01.LOG
    -rw-rw-r--. 1 1000650000 nobody  1808 Nov 11 10:57 AMQERR01.json
    -rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR02.LOG
    -rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR03.LOG
    
    2022-11-11T10:57:11.863Z DEBUG: /etc/mqm:
    total 24
    -rw-r--r--. 1 1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
    drwxr-sr-x. 1 1001 root 4096 Jun 16 10:35 MQOpenTracing
    -rw-r--r--. 1 1001 root  591 Jun 15 13:07 native-ha.ini.tpl
    drwxr-sr-x. 4 root root 4096 Nov 11 10:57 pki
    drwxrwsr-x. 1 1001 root 4096 Jun 15 13:19 web
    
    2022-11-11T10:57:12.431Z DEBUG: ffstsummary:
     AMQ28.0.FDC 2022/11/11 10:57:11.302805 Installation1 crtmqdir 28 1 XY019009 xufOpenIniEdit xecU_W_INI_ACCESS_DENIED OK
    
    2022-11-11T10:57:12.431Z DEBUG: ---  End Diagnostics  ---
    2022-11-11T10:57:12.431Z Image created: 2022-06-15T13:16:28+00:00
    2022-11-11T10:57:12.431Z Image tag: ibm-mqadvanced-server:9.3.0.0-r1.20220615130653.3111d48-amd64
    2022-11-11T10:57:13.335Z MQ version: 9.3.0.0
    2022-11-11T10:57:13.336Z MQ level: p930-L220606
    2022-11-11T10:57:13.336Z MQ license: Production
    2022-11-11T10:57:16.429Z Creating queue manager QM_PROD
    2022-11-11T10:57:16.429Z Starting web server
    2022-11-11T10:57:19.083Z Error 71 creating queue manager: AMQ6239E: Permission denied attempting to access filesystem location
    '/mnt/mqm/data/mqs.ini'.
    AMQ7062E: Permission denied attempting to access an INI file.
    
    2022-11-11T10:57:19.083Z DEBUG: Writing termination message: /opt/mqm/bin/crtmqm: exit status 71
    2022-11-11T10:57:19.083Z /opt/mqm/bin/crtmqm: exit status 71
    2022-11-11T10:57:19.083Z DEBUG: --- Start Diagnostics ---
    2022-11-11T10:57:19.093Z DEBUG: /mnt/:
    total 12
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-data
    drwxrwxr-x. 3 nobody 65531 4096 Nov 11 10:57 mqm-log
    
    2022-11-11T10:57:19.132Z DEBUG: /mnt/mqm:
    total 4
    drwxrwsr-x. 13 1000650000 nobody 4096 Nov 11 10:57 data
    
    2022-11-11T10:57:19.237Z DEBUG: /mnt/mqm/data:
    total 52
    drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 config
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 conv
    drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 errors
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 exits64
    drwxrwsr-x. 2 1000650000 nobody 4096 Nov 11 10:57 log
    -rw-rw-r--. 1 1000650000 nobody  571 Nov 11 10:57 mqclient.ini
    drwxrwsr-x. 5 1000650000 nobody 4096 Nov 11 10:57 mqft
    -rw-rw-r--. 1 1000650000 nobody    0 Nov 11 10:57 mqs.ini
    -rw-rw-r--. 1 1000650000 nobody    0 Nov 11 10:57 mqs.ini.tmp
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 qmgrs
    -rw-rw-r--. 1 1000650000 nobody 1941 Nov 11 10:57 service.env
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 sockets
    drwxrwsrwx. 2 1000650000 nobody 4096 Nov 11 10:57 trace
    drwxrwsr-x. 3 1000650000 nobody 4096 Nov 11 10:57 web
    
    2022-11-11T10:57:19.250Z DEBUG: /mnt/mqm-log/log:
    total 0
    
    2022-11-11T10:57:19.254Z DEBUG: /mnt/mqm-data/qmgrs:
    total 0
    
    2022-11-11T10:57:19.261Z DEBUG: /var/mqm:
    lrwxrwxrwx. 1 root root 13 Jun 15 13:18 /var/mqm -> /mnt/mqm/data
    
    2022-11-11T10:57:19.508Z DEBUG: /var/mqm/errors:
    total 100
    -rw-rw----. 1 1000650000 nobody 89309 Nov 11 10:57 AMQ28.0.FDC
    -rw-rw-r--. 1 1000650000 nobody  2735 Nov 11 10:57 AMQERR01.LOG
    -rw-rw-r--. 1 1000650000 nobody  1808 Nov 11 10:57 AMQERR01.json
    -rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR02.LOG
    -rw-rw-r--. 1 1000650000 nobody     0 Nov 11 10:57 AMQERR03.LOG
    
    2022-11-11T10:57:19.518Z DEBUG: /etc/mqm:
    total 32
    -rw-r-----. 1 1000650000 root  742 Nov 11 10:57 15-tls.mqsc
    -rw-r--r--. 1       1001 root  745 Jun 15 13:07 15-tls.mqsc.tpl
    drwxr-sr-x. 1       1001 root 4096 Jun 16 10:35 MQOpenTracing
    -rw-r--r--. 1       1001 root  591 Jun 15 13:07 native-ha.ini.tpl
    drwxr-sr-x. 4 root       root 4096 Nov 11 10:57 pki
    drwxrwsr-x. 1       1001 root 4096 Jun 15 13:19 web
    
    2022-11-11T10:57:20.232Z DEBUG: ffstsummary:
     AMQ28.0.FDC 2022/11/11 10:57:11.302805 Installation1 crtmqdir 28 1 XY019009 xufOpenIniEdit xecU_W_INI_ACCESS_DENIED OK
    
    2022-11-11T10:57:20.232Z DEBUG: ---  End Diagnostics  ---​


    ------------------------------
    Nicolas Montesi
    ------------------------------


  • 2.  RE: Permission denied on filesystem - CP4I MQ 9.3.0.0-r1 MultiInstance - IBM Cloud ROKS 4.10

    Posted Mon November 14, 2022 09:50 AM

    Adding GID 65534 to the supplementalGroups list resolve the problem. The worker nodes have RHEL 8 as underlying OS. According to Red Hat changelog:

    The nobody user replaces nfsnobody
    
    In Red Hat Enterprise Linux 7, there was:
    
    -     the nobody user and group pair with the ID of 99, and
    -     the nfsnobody user and group pair with the ID of 65534, which is the default kernel overflow ID, too. 
    
    Both of these have been merged into the nobody user and group pair, which uses the 65534 ID in Red Hat Enterprise Linux 8. New installations no longer create the nfsnobody pair.
    
    This change reduces the confusion about files that are owned by nobody but have nothing to do with NFS. 
    


    ------------------------------
    Nicolas Montesi
    ------------------------------