Hello,
We try to setup a lab/demo of the CP4S 1.4 on a baremetal environnement to understand the onboarding and installation process.
We successfully install OpenShift version: 4.4.17 (Update channel : stable-4.4)
The installation contains 3 masters, 3 workers, a bootstrap and a service machine.
[root@ocp-svc new]# oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-cp-1.lab.ocp.lan Ready master 13d v1.17.1+20ba474
ocp-cp-2.lab.ocp.lan Ready master 13d v1.17.1+20ba474
ocp-cp-3.lab.ocp.lan Ready master 13d v1.17.1+20ba474
ocp-w-1.lab.ocp.lan Ready worker 7d4h v1.17.1+20ba474
ocp-w-2.lab.ocp.lan Ready worker 7d4h v1.17.1+20ba474
ocp-w-3.lab.ocp.lan Ready worker 7d2h v1.17.1+20ba474
Following the documentation of Cloud Pak For Security 1.4, we prepare the environment with the prerequisite mentioned for an online installation.
We create a volume with a PVC and storageclass as mentioned in the documentation (RO – 1 Tb - bloc).
We complete the values.conf using our setup including the storageclass created before and certificates (public, private, CA from our lab PKI).
Lauching the magic command line, works fine :
[root@ocp-svc new]# cloudctl case launch --case ibm-cp-security --namespace cp4s --inventory installProduct --action install --args "--license accept --helm3 /usr/local/bin/helm3 --inputDir /root/new/" --tolerance 1
Prerequisite Result
Cluster Kubernetes version must be >=1.16.2 true
openshift Kubernetes version must be >=1.16.2 true
Client oc CLI must meet the following regex: 4.[3-9]*.[0-9]* true
Client cloudctl CLI must meet the following regex: Client Version: v3.4.[1-9]* true
Client helm3 CLI must meet the following regex: version.BuildInfo{Version:"v3.[1-9].[1-9]* true
Client docker CLI must meet the following regex: Docker version 1[8-9].d*.d* true
Client podman CLI must meet the following regex: version 1.([4-9]|[1-8][0-9]|9[0-9]). True
Prerequisite are OK.
…
…
…
At the end, we got the following error :
INFO - Waiting Common Services Pods initialization
INFO - Waiting Common Services Pods initialization
[ERROR] Error on Common Services Pods Startup.
[ERROR] Common Services Validation has failed has failed
Launch script failed due to: exit status 1
FAILED
Looking (command line) :
[root@ocp-svc new]# kubectl get clusterserviceversion -n ibm-common-services
NAME DISPLAY VERSION REPLACES PHASE
ibm-catalog-ui-operator.v3.6.1 IBM Catalog UI Operator 3.6.1 Succeeded
ibm-cert-manager-operator.v3.6.3 IBM Cert Manager Operator 3.6.3 Succeeded
ibm-common-service-operator.v3.4.3 IBM Common Service Operator 3.4.3 Succeeded
ibm-commonui-operator.v1.2.4 Ibm Common UI Operator 1.2.4 Succeeded
ibm-healthcheck-operator.v3.6.1 IBM Health Check Operator 3.6.1 Succeeded
ibm-helm-api-operator.v3.6.1 IBM Helm API Operator 3.6.1 Succeeded
ibm-helm-repo-operator.v3.6.2 IBM Helm Repo Operator 3.6.2 Succeeded
ibm-iam-operator.v3.6.5 IBM IAM Operator 3.6.5 Succeeded
ibm-ingress-nginx-operator.v1.2.3 IBM Ingress Nginx Operator 1.2.3 Succeeded
ibm-management-ingress-operator.v1.2.1 Management Ingress Operator 1.2.1 Succeeded
ibm-metering-operator.v3.6.3 IBM Metering Operator 3.6.3 Succeeded
ibm-mongodb-operator.v1.1.3 IBM Mongodb Operator 1.1.3 Succeeded
ibm-platform-api-operator.v3.6.2 IBM Platform API Operator 3.6.2 Succeeded
operand-deployment-lifecycle-manager.v1.2.3 Operand Deployment Lifecycle Manager 1.2.3 Succeeded
When looking deeper, seems that a few pods are not launched (14 pending PODS) :
[root@ocp-svc new]# kubectl -n ibm-common-services get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
auth-idp-6c79c75c45-7ctrd 0/4 Init:0/1 0 9m9s <none> ocp-w-3.lab.ocp.lan <none> <none>
auth-pap-77dd56bdbf-z92xk 0/2 ContainerCreating 0 90m <none> ocp-w-1.lab.ocp.lan <none> <none>
auth-pdp-84d9f69b4b-gt9xn 0/2 Init:0/1 0 13m <none> ocp-w-2.lab.ocp.lan <none> <none>
catalog-ui-557787bffd-wbbfw 1/1 Running 0 93m 10.128.2.109 ocp-w-2.lab.ocp.lan <none> <none>
cert-manager-cainjector-54774d6bf5-fpwzl 1/1 Running 0 93m 10.131.0.111 ocp-w-1.lab.ocp.lan <none> <none>
cert-manager-controller-5dffff5cb7-qqbrt 1/1 Running 0 93m 10.131.0.109 ocp-w-1.lab.ocp.lan <none> <none>
cert-manager-webhook-8b6d4bbcd-mxfv8 1/1 Running 2 93m 10.131.0.112 ocp-w-1.lab.ocp.lan <none> <none>
common-web-ui-9rtjp 1/1 Running 0 91m 10.128.2.115 ocp-w-2.lab.ocp.lan <none> <none>
common-web-ui-cvs2g 1/1 Running 0 91m 10.129.3.25 ocp-w-3.lab.ocp.lan <none> <none>
common-web-ui-ktlc9 1/1 Running 0 91m 10.131.0.116 ocp-w-1.lab.ocp.lan <none> <none>
configmap-watcher-fc55ff478-hqf6c 1/1 Running 0 93m 10.131.0.110 ocp-w-1.lab.ocp.lan <none> <none>
default-http-backend-cff9967f6-vpn7n 1/1 Running 0 93m 10.129.3.21 ocp-w-3.lab.ocp.lan <none> <none>
helm-api-7479f46b94-g5gwf 0/2 ContainerCreating 0 6m54s <none> ocp-w-1.lab.ocp.lan <none> <none>
helm-api-777b986fd7-598bv 0/2 ContainerCreating 0 7m10s <none> ocp-w-2.lab.ocp.lan <none> <none>
helm-repo-6fdf8999fb-sdwwt 0/1 ContainerCreating 0 7m38s <none> ocp-w-3.lab.ocp.lan <none> <none>
iam-onboarding-jw7wj 0/1 Init:0/5 0 5m59s <none> ocp-w-2.lab.ocp.lan <none> <none>
iam-policy-controller-7bf656d5c6-s229v 1/1 Running 2 91m 10.128.2.117 ocp-w-2.lab.ocp.lan <none> <none>
ibm-catalog-ui-operator-5648b77c9d-xq9q7 1/1 Running 0 95m 10.129.3.4 ocp-w-3.lab.ocp.lan <none> <none>
ibm-cert-manager-operator-7bcf948c6b-pmjls 1/1 Running 0 95m 10.129.3.9 ocp-w-3.lab.ocp.lan <none> <none>
ibm-common-service-operator-77f9fd8b6-v7kz8 1/1 Running 0 102m 10.129.2.255 ocp-w-3.lab.ocp.lan <none> <none>
ibm-common-service-webhook-7d94655cd4-6n9dg 1/1 Running 1 102m 10.129.3.0 ocp-w-3.lab.ocp.lan <none> <none>
ibm-commonui-operator-5bb7774f44-5mhwb 1/1 Running 0 94m 10.129.3.13 ocp-w-3.lab.ocp.lan <none> <none>
ibm-healthcheck-operator-84d6fd888-75rfg 1/1 Running 0 95m 10.129.3.11 ocp-w-3.lab.ocp.lan <none> <none>
ibm-helm-api-operator-5c79cb5d5-6jg5g 1/1 Running 0 95m 10.129.3.5 ocp-w-3.lab.ocp.lan <none> <none>
ibm-helm-repo-operator-69468865fc-z7xj7 1/1 Running 0 95m 10.129.3.10 ocp-w-3.lab.ocp.lan <none> <none>
ibm-iam-operator-56d9b5f57b-jq84d 1/1 Running 0 92m 10.129.3.23 ocp-w-3.lab.ocp.lan <none> <none>
ibm-ingress-nginx-operator-7cddb64fd6-b57n2 1/1 Running 0 95m 10.129.3.6 ocp-w-3.lab.ocp.lan <none> <none>
ibm-management-ingress-operator-85b6475b9b-qs5xc 1/1 Running 0 95m 10.129.3.7 ocp-w-3.lab.ocp.lan <none> <none>
ibm-metering-operator-6fb44b5f56-gfwkq 1/1 Running 0 94m 10.129.3.12 ocp-w-3.lab.ocp.lan <none> <none>
ibm-mongodb-operator-85bdb7b56-cjg8x 1/1 Running 0 29m 10.128.2.120 ocp-w-2.lab.ocp.lan <none> <none>
ibm-platform-api-operator-bcfc9b88d-pvqm4 1/1 Running 0 95m 10.129.3.8 ocp-w-3.lab.ocp.lan <none> <none>
icp-memcached-7f5589d655-xjl5d 1/1 Running 0 94m 10.129.3.14 ocp-w-3.lab.ocp.lan <none> <none>
management-ingress-5fc7cc9d47-5ct8f 1/1 Running 0 90m 10.131.0.117 ocp-w-1.lab.ocp.lan <none> <none>
metering-dm-59b56849bc-xtwkn 0/1 Init:0/2 0 94m 10.129.3.16 ocp-w-3.lab.ocp.lan <none> <none>
metering-reader-5bdf844bb5-vgjx4 0/1 Init:0/2 0 94m 10.129.3.17 ocp-w-3.lab.ocp.lan <none> <none>
metering-reader-7f55f4f95b-btgpb 0/1 Init:0/2 0 91m 10.128.2.114 ocp-w-2.lab.ocp.lan <none> <none>
metering-report-6799d4f485-2ncgj 1/1 Running 0 94m 10.129.3.18 ocp-w-3.lab.ocp.lan <none> <none>
metering-ui-69cd56dbb-r4njm 0/1 Init:0/2 0 94m 10.129.3.19 ocp-w-3.lab.ocp.lan <none> <none>
metering-ui-6fcfc6854f-qdmj6 0/1 Init:0/2 0 91m 10.131.0.114 ocp-w-1.lab.ocp.lan <none> <none>
nginx-ingress-controller-786c58dfbf-6x7tt 1/1 Running 0 93m 10.129.3.22 ocp-w-3.lab.ocp.lan <none> <none>
oidc-client-registration-8xrzr 1/1 Running 0 90m 10.128.2.119 ocp-w-2.lab.ocp.lan <none> <none>
oidcclient-watcher-b99fdf987-8gr5v 1/1 Running 0 91m 10.129.3.27 ocp-w-3.lab.ocp.lan <none> <none>
platform-api-798d8bfd89-ns4nt 2/2 Running 0 91m 10.128.2.116 ocp-w-2.lab.ocp.lan <none> <none>
secret-watcher-84fb694fc7-kvc4m 0/1 CreateContainerConfigError 0 26m 10.131.0.118 ocp-w-1.lab.ocp.lan <none> <none>
secretshare-5b6dd4c5df-97lwh 2/2 Running 0 102m 10.129.3.1 ocp-w-3.lab.ocp.lan <none> <none>
security-onboarding-vf2v4 0/1 Init:0/1 0 5m22s 10.128.2.121 ocp-w-2.lab.ocp.lan <none> <none>
system-healthcheck-service-84ff95b7cf-2j6hk 1/1 Running 0 94m 10.129.3.15 ocp-w-3.lab.ocp.lan <none> <none>
tiller-deploy-66d8df58d7-ldbwv 1/1 Running 0 91m 10.131.0.115 ocp-w-1.lab.ocp.lan <none> <none>
Looking at the OpenShift Web interface, we see a lot of errors linked to certificates problem / Mongo :
iam-onboarding-8w9v6
MountVolume.SetUp failed for volume "mongodb-client-cert" : secret "icp-mongodb-client-cert" not found
(combined from similar events): Unable to attach or mount volumes: unmounted volumes=[mongodb-ca-cert mongodb-client-cert], unattached volumes=[journal logrotate shared mongodb-ca-cert cluster-ca logrotate-conf ibm-iam-operand-privileged-token-2bd5c pap-cert mongodb-client-cert]: timed out waiting for the condition
But the services seems to be running :
[root@ocp-svc new]# kubectl -n ibm-common-services get pods -o wide | grep mongo
ibm-mongodb-operator-85bdb7b56-cjg8x 1/1 Running 0 74m 10.128.2.120 ocp-w-2.lab.ocp.lan <none> <none>
Is it related to the ibm-cert-manager-operator issue on github ?
Any idea ?
------------------------------
Best regards,
Zoldax
------------------------------