Cloud Pak for Security

 View Only
  • 1.  New CP4S 1.4 common services installation issue

    IBM Champion
    Posted Mon October 19, 2020 10:04 AM

    Hello,

     

    We try to setup a lab/demo of the  CP4S 1.4 on a baremetal environnement to understand the onboarding and installation process.

    We successfully install OpenShift version: 4.4.17 (Update channel : stable-4.4)

    The installation contains 3 masters, 3 workers, a bootstrap and a service machine.

     

    [root@ocp-svc new]# oc get nodes

    NAME                   STATUS   ROLES    AGE    VERSION

    ocp-cp-1.lab.ocp.lan   Ready    master   13d    v1.17.1+20ba474

    ocp-cp-2.lab.ocp.lan   Ready    master   13d    v1.17.1+20ba474

    ocp-cp-3.lab.ocp.lan   Ready    master   13d    v1.17.1+20ba474

    ocp-w-1.lab.ocp.lan    Ready    worker   7d4h   v1.17.1+20ba474

    ocp-w-2.lab.ocp.lan    Ready    worker   7d4h   v1.17.1+20ba474

    ocp-w-3.lab.ocp.lan    Ready    worker   7d2h   v1.17.1+20ba474

     

    Following the documentation of Cloud Pak For Security 1.4, we prepare the environment with the prerequisite mentioned for an online installation.

    We create a volume with a PVC and storageclass as mentioned in the documentation (RO – 1 Tb - bloc).

    We complete the values.conf using our setup including the storageclass created before and certificates (public, private, CA from our lab PKI).

    Lauching the magic command line, works fine :

    [root@ocp-svc new]# cloudctl case launch --case ibm-cp-security --namespace cp4s  --inventory installProduct --action install --args "--license accept --helm3 /usr/local/bin/helm3 --inputDir /root/new/" --tolerance 1

     

    Prerequisite                                                                                Result

    Cluster Kubernetes version must be >=1.16.2                                                 true

    openshift Kubernetes version must be >=1.16.2                                               true

    Client oc CLI must meet the following regex: 4.[3-9]*.[0-9]*                                true

    Client cloudctl CLI must meet the following regex: Client Version: v3.4.[1-9]*              true

    Client helm3 CLI must meet the following regex: version.BuildInfo{Version:"v3.[1-9].[1-9]*  true

    Client docker CLI must meet the following regex: Docker version 1[8-9].d*.d*                true

    Client podman CLI must meet the following regex: version 1.([4-9]|[1-8][0-9]|9[0-9]).       True

     

    Prerequisite are OK.

    At the end, we got the following error :

    INFO - Waiting Common Services Pods initialization

    INFO - Waiting Common Services Pods initialization

    [ERROR]  Error on Common Services Pods Startup.

    [ERROR] Common Services Validation has failed has failed

    Launch script failed due to: exit status 1

    FAILED

     

    Looking (command line) :

     

    [root@ocp-svc new]# kubectl get clusterserviceversion -n ibm-common-services

    NAME                                          DISPLAY                                VERSION   REPLACES   PHASE

    ibm-catalog-ui-operator.v3.6.1                IBM Catalog UI Operator                3.6.1                Succeeded

    ibm-cert-manager-operator.v3.6.3              IBM Cert Manager Operator              3.6.3                Succeeded

    ibm-common-service-operator.v3.4.3            IBM Common Service Operator            3.4.3                Succeeded

    ibm-commonui-operator.v1.2.4                  Ibm Common UI Operator                 1.2.4                Succeeded

    ibm-healthcheck-operator.v3.6.1               IBM Health Check Operator              3.6.1                Succeeded

    ibm-helm-api-operator.v3.6.1                  IBM Helm API Operator                  3.6.1                Succeeded

    ibm-helm-repo-operator.v3.6.2                 IBM Helm Repo Operator                 3.6.2                Succeeded

    ibm-iam-operator.v3.6.5                       IBM IAM Operator                       3.6.5                Succeeded

    ibm-ingress-nginx-operator.v1.2.3             IBM Ingress Nginx Operator             1.2.3                Succeeded

    ibm-management-ingress-operator.v1.2.1        Management Ingress Operator            1.2.1                Succeeded

    ibm-metering-operator.v3.6.3                  IBM Metering Operator                  3.6.3                Succeeded

    ibm-mongodb-operator.v1.1.3                   IBM Mongodb Operator                   1.1.3                Succeeded

    ibm-platform-api-operator.v3.6.2              IBM Platform API Operator              3.6.2                Succeeded

    operand-deployment-lifecycle-manager.v1.2.3   Operand Deployment Lifecycle Manager   1.2.3                Succeeded

     

    When looking deeper, seems that a few pods are not launched (14 pending PODS) :

     

    [root@ocp-svc new]# kubectl -n ibm-common-services get pods -o wide

    NAME                                               READY   STATUS                       RESTARTS   AGE     IP             NODE                  NOMINATED NODE   READINESS GATES

    auth-idp-6c79c75c45-7ctrd                          0/4     Init:0/1                     0          9m9s    <none>         ocp-w-3.lab.ocp.lan   <none>           <none>

    auth-pap-77dd56bdbf-z92xk                          0/2     ContainerCreating            0          90m     <none>         ocp-w-1.lab.ocp.lan   <none>           <none>

    auth-pdp-84d9f69b4b-gt9xn                          0/2     Init:0/1                     0          13m     <none>         ocp-w-2.lab.ocp.lan   <none>           <none>

    catalog-ui-557787bffd-wbbfw                        1/1     Running                      0          93m     10.128.2.109   ocp-w-2.lab.ocp.lan   <none>           <none>

    cert-manager-cainjector-54774d6bf5-fpwzl           1/1     Running                      0          93m     10.131.0.111   ocp-w-1.lab.ocp.lan   <none>           <none>

    cert-manager-controller-5dffff5cb7-qqbrt           1/1     Running                      0          93m     10.131.0.109   ocp-w-1.lab.ocp.lan   <none>           <none>

    cert-manager-webhook-8b6d4bbcd-mxfv8               1/1     Running                      2          93m     10.131.0.112   ocp-w-1.lab.ocp.lan   <none>           <none>

    common-web-ui-9rtjp                                1/1     Running                      0          91m     10.128.2.115   ocp-w-2.lab.ocp.lan   <none>           <none>

    common-web-ui-cvs2g                                1/1     Running                      0          91m     10.129.3.25    ocp-w-3.lab.ocp.lan   <none>           <none>

    common-web-ui-ktlc9                                1/1     Running                      0          91m     10.131.0.116   ocp-w-1.lab.ocp.lan   <none>           <none>

    configmap-watcher-fc55ff478-hqf6c                  1/1     Running                      0          93m     10.131.0.110   ocp-w-1.lab.ocp.lan   <none>           <none>

    default-http-backend-cff9967f6-vpn7n               1/1     Running                      0          93m     10.129.3.21    ocp-w-3.lab.ocp.lan   <none>           <none>

    helm-api-7479f46b94-g5gwf                          0/2     ContainerCreating            0          6m54s   <none>         ocp-w-1.lab.ocp.lan   <none>           <none>

    helm-api-777b986fd7-598bv                          0/2     ContainerCreating            0          7m10s   <none>         ocp-w-2.lab.ocp.lan   <none>           <none>

    helm-repo-6fdf8999fb-sdwwt                         0/1     ContainerCreating            0          7m38s   <none>         ocp-w-3.lab.ocp.lan   <none>           <none>

    iam-onboarding-jw7wj                               0/1     Init:0/5                     0          5m59s   <none>         ocp-w-2.lab.ocp.lan   <none>           <none>

    iam-policy-controller-7bf656d5c6-s229v             1/1     Running                      2          91m     10.128.2.117   ocp-w-2.lab.ocp.lan   <none>           <none>

    ibm-catalog-ui-operator-5648b77c9d-xq9q7           1/1     Running                      0          95m     10.129.3.4     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-cert-manager-operator-7bcf948c6b-pmjls         1/1     Running                      0          95m     10.129.3.9     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-common-service-operator-77f9fd8b6-v7kz8        1/1     Running                      0          102m    10.129.2.255   ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-common-service-webhook-7d94655cd4-6n9dg        1/1     Running                      1          102m    10.129.3.0     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-commonui-operator-5bb7774f44-5mhwb             1/1     Running                      0          94m     10.129.3.13    ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-healthcheck-operator-84d6fd888-75rfg           1/1     Running                      0          95m     10.129.3.11    ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-helm-api-operator-5c79cb5d5-6jg5g              1/1     Running                      0          95m     10.129.3.5     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-helm-repo-operator-69468865fc-z7xj7            1/1     Running                      0          95m     10.129.3.10    ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-iam-operator-56d9b5f57b-jq84d                  1/1     Running                      0          92m     10.129.3.23    ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-ingress-nginx-operator-7cddb64fd6-b57n2        1/1     Running                      0          95m     10.129.3.6     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-management-ingress-operator-85b6475b9b-qs5xc   1/1     Running                      0          95m     10.129.3.7     ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-metering-operator-6fb44b5f56-gfwkq             1/1     Running                      0          94m     10.129.3.12    ocp-w-3.lab.ocp.lan   <none>           <none>

    ibm-mongodb-operator-85bdb7b56-cjg8x               1/1     Running                      0          29m     10.128.2.120   ocp-w-2.lab.ocp.lan   <none>           <none>

    ibm-platform-api-operator-bcfc9b88d-pvqm4          1/1     Running                      0          95m     10.129.3.8     ocp-w-3.lab.ocp.lan   <none>           <none>

    icp-memcached-7f5589d655-xjl5d                     1/1     Running                      0          94m     10.129.3.14    ocp-w-3.lab.ocp.lan   <none>           <none>

    management-ingress-5fc7cc9d47-5ct8f                1/1     Running                      0          90m     10.131.0.117   ocp-w-1.lab.ocp.lan   <none>           <none>

    metering-dm-59b56849bc-xtwkn                       0/1     Init:0/2                     0          94m     10.129.3.16    ocp-w-3.lab.ocp.lan   <none>           <none>

    metering-reader-5bdf844bb5-vgjx4                   0/1     Init:0/2                     0          94m     10.129.3.17    ocp-w-3.lab.ocp.lan   <none>           <none>

    metering-reader-7f55f4f95b-btgpb                   0/1     Init:0/2                     0          91m     10.128.2.114   ocp-w-2.lab.ocp.lan   <none>           <none>

    metering-report-6799d4f485-2ncgj                   1/1     Running                      0          94m     10.129.3.18    ocp-w-3.lab.ocp.lan   <none>           <none>

    metering-ui-69cd56dbb-r4njm                        0/1     Init:0/2                     0          94m     10.129.3.19    ocp-w-3.lab.ocp.lan   <none>           <none>

    metering-ui-6fcfc6854f-qdmj6                       0/1     Init:0/2                     0          91m     10.131.0.114   ocp-w-1.lab.ocp.lan   <none>           <none>

    nginx-ingress-controller-786c58dfbf-6x7tt          1/1     Running                      0          93m     10.129.3.22    ocp-w-3.lab.ocp.lan   <none>           <none>

    oidc-client-registration-8xrzr                     1/1     Running                      0          90m     10.128.2.119   ocp-w-2.lab.ocp.lan   <none>           <none>

    oidcclient-watcher-b99fdf987-8gr5v                 1/1     Running                      0          91m     10.129.3.27    ocp-w-3.lab.ocp.lan   <none>           <none>

    platform-api-798d8bfd89-ns4nt                      2/2     Running                      0          91m     10.128.2.116   ocp-w-2.lab.ocp.lan   <none>           <none>

    secret-watcher-84fb694fc7-kvc4m                    0/1     CreateContainerConfigError   0          26m     10.131.0.118   ocp-w-1.lab.ocp.lan   <none>           <none>

    secretshare-5b6dd4c5df-97lwh                       2/2     Running                      0          102m    10.129.3.1     ocp-w-3.lab.ocp.lan   <none>           <none>

    security-onboarding-vf2v4                          0/1     Init:0/1                     0          5m22s   10.128.2.121   ocp-w-2.lab.ocp.lan   <none>           <none>

    system-healthcheck-service-84ff95b7cf-2j6hk        1/1     Running                      0          94m     10.129.3.15    ocp-w-3.lab.ocp.lan   <none>           <none>

    tiller-deploy-66d8df58d7-ldbwv                     1/1     Running                      0          91m     10.131.0.115   ocp-w-1.lab.ocp.lan   <none>           <none>

      

    Looking at the OpenShift Web interface, we see a lot of errors linked to certificates problem / Mongo :

     

    iam-onboarding-8w9v6

    MountVolume.SetUp failed for volume "mongodb-client-cert" : secret "icp-mongodb-client-cert" not found

     

    (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[mongodb-ca-cert mongodb-client-cert], unattached volumes=[journal logrotate shared mongodb-ca-cert cluster-ca logrotate-conf ibm-iam-operand-privileged-token-2bd5c pap-cert mongodb-client-cert]: timed out waiting for the condition

    But the services seems to be running :

    [root@ocp-svc new]# kubectl -n ibm-common-services get pods -o wide  | grep mongo

    ibm-mongodb-operator-85bdb7b56-cjg8x               1/1     Running                      0          74m    10.128.2.120   ocp-w-2.lab.ocp.lan   <none>           <none>

     Is it related to the ibm-cert-manager-operator issue on github ?

    Any idea  ?



    ------------------------------
    Best regards,

    Zoldax



    ------------------------------


  • 2.  RE: New CP4S 1.4 common services installation issue

    Posted Fri October 23, 2020 03:21 AM
    Hi,

    as far as i know this can come from 2 things:

    1- not synchronized time between the bastion (if you use one)/HAproxy and the nodes
    2- sometime some pods take too much time to be pulled which create issues like this one

    thank you

    ------------------------------
    Amine HAMIDA
    ------------------------------



  • 3.  RE: New CP4S 1.4 common services installation issue

    IBM Champion
    Posted Sat October 31, 2020 05:56 AM
    Edited by Pascal Weber Mon November 02, 2020 12:42 PM
    Hello Amine,

    Thank you for your reply, sorry for my delay, I was on other subjects... :)

    I found the issue, I think this was related to some role and grant on the 1 TB disk I used shared across the machines, this causes problem with creatio/allocation of mongodb.

    Lets share how I fixed this :
    Recreating my yaml definitions files (rbac, class, deployment) for my shared disk and creating correct role and grant resolved the problem. (oc create role, oc adm policy add-role-to-user, oc scale deploy).

    The installation of the ibm common services works after that :)

    NAME: ibm-security-foundations
    LAST DEPLOYED: Mon Oct 26 17:12:45 2020
    NAMESPACE: cp4s
    STATUS: deployed
    REVISION: 1
    NOTES:
    NOTES:
    ################################################################################
    # #
    # Congratulations on installing IBM Security Foundations! #
    # #
    ################################################################################

    NAME: ibm-security-solutions
    LAST DEPLOYED: Mon Oct 26 17:19:40 2020
    NAMESPACE: cp4s
    STATUS: deployed
    REVISION: 1
    NOTES:
    ################################################################################
    # #
    # Congratulations on installing CP4S Security Solutions ! #
    # #
    ################################################################################

    See you on next episode :)

    CP4S 1.4 Success login zoldax



    Regards,
    Zoldax




    ------------------------------
    @zoldax

    https://www.youracclaim.com/users/pascal-weber.029e134d/badges
    ------------------------------



  • 4.  RE: New CP4S 1.4 common services installation issue

    Posted Mon December 14, 2020 01:32 PM

    Hi Zoldax,

    I am facing the same issue. Can you please explain the solution in detail?

    Thanks,



    ------------------------------
    Lata Sagnali
    ------------------------------