Cloud Pak for Data

 View Only
Expand all | Collapse all

WKC and WS Installation failures

  • 1.  WKC and WS Installation failures

    Posted Mon April 06, 2020 03:07 PM
    Both WKC and WS fails to install with the following errors on CP4D 2.5 when following the documentation that is available


    time="2020-04-06T19:00:45Z" level=info msg="Values override output: elasticsearch:\n  fsGroup: \"\"\nglobal:\n  docker_registry_prefix: cp.icr.io/cp/cpd\n  dockerRegistryPrefix: cp.icr.io/cp/cpd\n  ha: false\n  ibmProduct: zen\n  persistence:\n    storageClassName: null\n    useDynamicProvisioning: false\n  replicaCount: 1\n  storageClassName: null\n  storageclass_name: null\nredis-ha:\n  rbac:\n    create: false\n  securityContext:\n    fsGroup: \"\"\n    runAsNonRoot: \"\"\n    runAsUser: \"\"\nwdp-rabbitmq:\n  rbac:\n    create: false\n  securityContext:\n    fsGroup: \"\"\n    runAsUser: \"\"\n  serviceAccount:\n    create: false\n    name: cpd-viewer-sa\n"

    time="2020-04-06T19:00:45Z" level=info msg="Installing release 0022-wkc-base-prereqs"

    time="2020-04-06T19:00:45Z" level=error msg="rpc error: code = Unknown desc = render error in \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\": template: wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml:21:13: executing \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\" at <required \"persistentVolume.path must be specified\" .Values.persistentVolume.path>: error calling required: persistentVolume.path must be specified"

    time="2020-04-06T19:00:45Z" level=error msg="Error installing 0022-wkc-base-prereqs-3.0.263.tgz. Error: rpc error: code = Unknown desc = render error in \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\": template: wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml:21:13: executing \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\" at <required \"persistentVolume.path must be specified\" .Values.persistentVolume.path>: error calling required: persistentVolume.path must be specified"



    ------------------------------
    Giri Badanahatti
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 03:58 AM

    Hi,

    Can you provide the exact installation command you execute and the override file ?

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 3.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 09:43 AM
    Thanks Tomasz. 

    1. We prepared the cluster as documented in: https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_current/wsj/install/preinstall-wkc.html
    2. We executed cpd-linux adm --repo <repo location> --assembly wkc --namespace <project name> --apply
    3. We ran the installation cpd-linux --repo <repo location> --assembly wkc --namespace <project name> --storageclass <storage class name>.

    NOTE:  We did not have an override.yaml as the storage was not a portworx storage.

    ------------------------------
    Giri Badanahatti
    ------------------------------



  • 4.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 04:29 PM
    Can you please provide the exact command for 3? (please replace any confidential information with XXXX)

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 5.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 05:08 PM
    Hi Tomasz:

    That was close to the exact command that was executed. To be very precise, this was the command that was exectuted:

    ./cpd-linux --repo ../repo.yaml --assembly wsl --namespace cloudpak --storageclass ocs-storagecluster-cephfs

    This is the command that is documented in the documentation for non-portworx, non-airgapped clusters:

    ./cpd-Operating_System --repo ./repo.yaml \ --assembly wsl \ --namespace Project \ --storageclass Storage_class_name \ --transfer-image-to Registry_location \ --cluster-pull-prefix Registry_from_cluster \ --ask-push-registry-credentials

    Thanks for all your help!



    ------------------------------
    Giri Badanahatti
    ------------------------------



  • 6.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 05:22 PM
    Hi,

    Couple of points:

    1)  ocs-storagecluster-cephfs is not supported and most likely will not work - please use NFS or Portworx.
    storageClassName: null\n    useDynamicProvisioning: false\n from the log suggests that the storageclass wasn't provided at all or was provided incorrectly.

    2) I do not see transfer-image-to value (which really is mandatory).
    The first post suggests that you have used cp.icr.io/cp/cpd for this parameter, which is incorrect.
    It should the output of 'oc registry info' or the corresponding route.

    My suggestion would be to delete the namespace and start again with correct values.

    Thanks


    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 7.  RE: WKC and WS Installation failures

    Posted Tue April 07, 2020 10:24 PM
    Thanks Tomasz!

    For 1: As you can see that I have provided the storage class name on the command line and if the installer still thinks that there is no storage class then, there is an issue somewhere. I tried with NFS and it still fails.

    For 2: Not using the parameter --transfer-image-to, the logs clearly indicate that the installer will use IBM external registries. So I am not sure why this is needed.

    *** Transferring all required images ***

    [INFO] [2020-04-07 21:37:58-0000] Customer target registry is not specified, using IBM external registries

    I have tried to use the internal registry as well but without any success. I executed the install with the following:

    ./cpd-linux --repo ../repo.yaml --assembly wsl --namespace cloudpak --storageclass managed-nfs-storage --transfer-image-to image-registry.openshift-image-registry.svc:5000/cloudpak --target-registry-username kubeadmin --target-registry-password xxxxxxx

    it still fails with the following:

    ERROR] [2020-04-07 21:45:49-0131] Error initializing source docker://cp.icr.io/cp/cpd/zen-meta-couchdb:v2.5.0.0-210: unable to retrieve auth token: invalid username/password

    [ERROR] [2020-04-07 21:45:49-0132] Error in handling image request regarding zen-meta-couchdb:v2.5.0.0-210 from registry docker://cp.icr.io/cp/cpd/zen-meta-couchdb:v2.5.0.0-210 after 4 retries. Abort

    [ERROR] [2020-04-07 21:45:49-0132] Exiting due to error. Please check /root/cpd/bin/cpd-linux-workspace/Logs/CPD-2020-04-07T21-45-15.log for details

    Can you send me an example that works? 

    Also, I cannot delete the namespace as I have control plane and IAE already installed on this namespace.



    ------------------------------
    Giri Badanahatti
    ------------------------------



  • 8.  RE: WKC and WS Installation failures

    Posted Wed April 08, 2020 04:02 AM
    Hi,

    1) Do you mean it fails the same way with NFS ?

    2) You need to transfer the images, otherwise some add-ons  will never work - this is a requirement.

    3) target-registry-password - this needs to be the output of (oc whoami -T), not your password

    4) Are you using OCP 4 ? (based on your registry route) - this is not supported for 2.5

    5) The error:
    ERROR] [2020-04-07 21:45:49-0131] Error initializing source docker://cp.icr.io/cp/cpd/zen-meta-couchdb:v2.5.0.0-210: unable to retrieve auth token: invalid username/password

    suggest issues with authenticating to our external repo. Where did you get the token from ?

    can you do

    # docker login cp.icr.io/cp/cpd
    Username (iamapikey): cp
    Password:
    Login Succeeded

    # docker pull cp.icr.io/cp/cpd/zen-meta-couchdb:v2.5.0.0-210
    Trying to pull repository cp.icr.io/cp/cpd/zen-meta-couchdb ...
    v2.5.0.0-210: Pulling from cp.icr.io/cp/cpd/zen-meta-couchdb
    bdb299bbb0dd: Downloading [===> ] 2.21 MB/31.49 MB

    if you do not have docker, use podman.

    6) Why can you not reinstall the lite/ae ?

    Thank you





    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 9.  RE: WKC and WS Installation failures

    Posted Wed April 08, 2020 11:27 AM
    Edited by System Fri January 20, 2023 04:17 PM
    Hi Tomasz:

    Yes, I am using the OCP4.3. I was told that there is nothing technically stopping CP4D to work on it. I still get the same errors even after using the openshift internalregistry and NFS storage.

    ime="2020-04-08T15:19:19Z" level=info msg="Installing release 0022-wkc-base-prereqs"

    time="2020-04-08T15:19:19Z" level=error msg="rpc error: code = Unknown desc = render error in \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\": template: wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml:21:13: executing \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\" at <required \"persistentVolume.path must be specified\" .Values.persistentVolume.path>: error calling required: persistentVolume.path must be specified"

    time="2020-04-08T15:19:19Z" level=error msg="Error installing 0022-wkc-base-prereqs-3.0.263.tgz. Error: rpc error: code = Unknown desc = render error in \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\": template: wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml:21:13: executing \"wkc-base-prereqs/charts/wdp-rabbitmq/templates/pv.yaml\" at <required \"persistentVolume.path must be specified\" .Values.persistentVolume.path>: error calling required: persistentVolume.path must be specified"

    The exact command that was executed was:

    ./cpd-linux --repo ../repo.yaml --assembly wsl --namespace cloudpak --storageclass managed-nfs-storage --transfer-image-to default-route-openshift-image-registry.apps.aitocs.ibmcosait.com/cloudpak --target-registry-username kubeadmin --target-registry-password XXX --insecure-skip-tls-verify

    The downloads to the registry was all successful. The reason why I don't want to delete the namespace is that I had to jump thorough several hoops to get it IAE functional.

     Also, contents of the registry:

    curl -k -u unused:$(oc whoami -t) https://default-route-openshift-image-registry.apps.aitocs.ibmcosait.com/v2/_catalog?n=100

    {"repositories":["cloudpak/asset-files-api","cloudpak/catalog_master","cloudpak/couchdb3","cloudpak/cp4d-tiller","cloudpak/cp4d-ws-secrets-job","cloudpak/cpd-operator-init","cloudpak/cpdoperator","cloudpak/dap-dashboards-api","cloudpak/dc-main","cloudpak/dsx-requisite","cloudpak/elasticsearch","cloudpak/environments","cloudpak/environments-init-job","cloudpak/haproxy","cloudpak/icp4d-api","cloudpak/icp4data-nginx-repo","cloudpak/icpd-requisite","cloudpak/influxdb","cloudpak/job-trigger","cloudpak/jobs-api","cloudpak/jobs-ui","cloudpak/jupyter-notebooks-converter","cloudpak/model_viewer","cloudpak/ngp-projects-api","cloudpak/nlq-search","cloudpak/nlq-search-extractor","cloudpak/notebooks","cloudpak/portal-catalog","cloudpak/portal-common-api","cloudpak/portal-dashboards","cloudpak/portal-job-manager","cloudpak/portal-main","cloudpak/portal-ml-dl","cloudpak/portal-notifications","cloudpak/privatecloud-dsx-hi-proxy","cloudpak/privatecloud-redis-repo","cloudpak/privatecloud-spawner-api-k8s","cloudpak/privatecloud-usermgmt","cloudpak/privatecloud-utils-api","cloudpak/rabbitmq","cloudpak/redis","cloudpak/shaper-preinstall","cloudpak/shaper-server","cloudpak/wdp-connect-connection","cloudpak/wdp-connect-connector","cloudpak/wdp-dataprep","cloudpak/wdp-shaper","cloudpak/wkc-init-container","cloudpak/wkc-search_master","cloudpak/wslocal-x86-runtime-python36","cloudpak/zen-admin","cloudpak/zen-content","cloudpak/zen-core","cloudpak/zen-core-api","cloudpak/zen-data-sorcerer","cloudpak/zen-meta-couchdb","cloudpak/zen-metastoredb","cloudpak/zen-watchdog","openshift/apicast-gateway","openshift/apicurito-ui","openshift/cli","openshift/cli-artifacts","openshift/dotnet","openshift/dotnet-runtime","openshift/eap-cd-openshift","openshift/fis-java-openshift","openshift/fis-karaf-openshift","openshift/fuse-apicurito-generator","openshift/fuse7-console","openshift/fuse7-eap-openshift","openshift/fuse7-java-openshift","openshift/fuse7-karaf-openshift","openshift/golang","openshift/httpd","openshift/installer","openshift/installer-artifacts","openshift/java","openshift/jboss-amq-62","openshift/jboss-amq-63","openshift/jboss-datagrid65-client-openshift","openshift/jboss-datagrid65-openshift","openshift/jboss-datagrid71-client-openshift","openshift/jboss-datagrid71-openshift","openshift/jboss-datagrid72-openshift","openshift/jboss-datagrid73-openshift","openshift/jboss-datavirt64-driver-openshift","openshift/jboss-datavirt64-openshift","openshift/jboss-decisionserver64-openshift","openshift/jboss-eap64-openshift","openshift/jboss-eap70-openshift","openshift/jboss-eap71-openshift","openshift/jboss-eap72-openshift","openshift/jboss-fuse70-console","openshift/jboss-fuse70-eap-openshift","openshift/jboss-fuse70-java-openshift","openshift/jboss-fuse70-karaf-openshift","openshift/jboss-processserver64-openshift","openshift/jboss-webserver30-tomcat7-openshift","openshift/jboss-webserver30-tomcat8-openshift","openshift/jboss-webserver31-tomcat7-openshift"]}

     


    ------------------------------
    Giri Badanahatti
    ------------------------------



  • 10.  RE: WKC and WS Installation failures

    Posted Thu April 09, 2020 06:45 AM
    Hi,


    "Yes, I am using the OCP4.3. I was told that there is nothing technically stopping CP4D to work on it."
    That's not accurate - there are some issues you will face, with nginx configuration, etc.
    We definitely do not recommend  and don't support using 4.3 for CP4D 2.5.

    You may not recover from this without removing the namespace, as the previous installation attempts have modified the operator configuration, and the invalid parameters are still present.

    Please reach out to me directly to discuss.

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 11.  RE: WKC and WS Installation failures

    Posted Tue March 23, 2021 03:24 PM
    Hi,
      is this issue resolved? I hit the same issue with OCP4.3 installing CP4d. 
    the podman works:
    podman login cp.icr.io/cp/cpd
    Username: cp
    Password:
    Login Succeeded!
    [root@cdl-net-ocp-inf cloudpack]# podman pull cp.icr.io/cp/cpd/zen-meta-couchdb:v3.0.1.0-x86_64-45
    Trying to pull cp.icr.io/cp/cpd/zen-meta-couchdb:v3.0.1.0-x86_64-45...
    Getting image source signatures
    Copying blob 9b376e51483d done
    Copying blob b26afdf22be4 done
    Copying blob 8875c0c11206 done
    Copying blob 87eafd2107bf done
    Copying blob 634d1b3c7b4c done
    Copying blob 218f593046ab done

    and the base-registry is good:
    curl -k -u unused:$(oc whoami -t) https://default-route-openshift-image-registry.apps.cdl-net-ocp.os.fyre.ibm.com/v2/_catalog?n=100
    {"repositories":["openshift/apicast-gateway","openshift/apicurito-ui","openshift/cli","openshift/cli-artifacts","openshift/dotnet","openshift/dotnet-runtime","openshift/eap-cd-openshift","openshift/fis-java-openshift","openshift/fis-karaf-openshift","openshift/fuse-apicurito-generator","openshift/fuse7-console","openshift/fuse7-eap-openshift","openshift/fuse7-java-openshift","openshift/fuse7-karaf-openshift","openshift/golang","openshift/httpd","openshift/installer","openshift/installer-artifacts","openshift/java","openshift/jboss-amq-62","openshift/jboss-amq-63","openshift/jboss-datagrid65-client-openshift","openshift/jboss-datagrid65-openshift","openshift/jboss-datagrid71-client-openshift","openshift/jboss-datagrid71-openshift","openshift/jboss-datagrid72-openshift","openshift/jboss-datagrid73-openshift","openshift/jboss-datavirt64-driver-openshift","openshift/jboss-datavirt64-openshift","openshift/jboss-decisionserver64-openshift","openshift/jboss-eap64-openshift","openshift/jboss-eap70-openshift","openshift/jboss-eap71-openshift","openshift/jboss-eap72-openshift","openshift/jboss-fuse70-console","openshift/jboss-fuse70-eap-openshift","openshift/jboss-fuse70-java-openshift","openshift/jboss-fuse70-karaf-openshift","openshift/jboss-processserver64-openshift","openshift/jboss-webserver30-tomcat7-openshift","openshift/jboss-webserver30-tomcat8-openshift","openshift/jboss-webserver31-tomcat7-openshift","openshift/jboss-webserver31-tomcat8-openshift","openshift/jboss-webserver50-tomcat9-openshift","openshift/jenkins","openshift/jenkins-agent-maven","openshift/jenkins-agent-nodejs","openshift/mariadb","openshift/modern-webapp","openshift/mongodb","openshift/must-gather","openshift/mysql","openshift/nginx","openshift/nodejs","openshift/oauth-proxy","openshift/openjdk-11-rhel7","openshift/perl","openshift/php","openshift/postgresql","openshift/python","openshift/redhat-openjdk18-openshift","openshift/redhat-sso70-openshift","openshift/redhat-sso71-openshift","openshift/redhat-sso72-openshift","openshift/redhat-sso73-openshift","openshift/redis","openshift/rhdm74-decisioncentral-openshift","openshift/rhdm74-kieserver-openshift","openshift/rhdm74-optaweb-employee-rostering-openshift","openshift/rhpam74-businesscentral-monitoring-openshift","openshift/rhpam74-businesscentral-openshift","openshift/rhpam74-kieserver-openshift","openshift/rhpam74-smartrouter-openshift","openshift/ruby","openshift/tests"]}

    the same error:
    Error trying to reuse blob sha256:b26afdf22be4e9c30220796780a297b91549a3b3041b6fdcbda71bf48a6912e7 at destination: unable to retrieve auth token: invalid username/password: unauthorized: authentication required
    [ERROR] [2021-03-22 20:11:56-0309] Error in handling image request regarding zen-meta-couchdb:v3.0.1.0-x86_64-45 from registry docker://cp.icr.io/cp/cpd/zen-meta-couchdb:v3.0.1.0-x86_64-45 after 4 retries. Abort

    CLI run:
    bin/cpd-linux --repo ./repo.yaml \ --assembly lite \ --namespace $NAMESPACE \ --storageclass ibmc-file-gold-gid \ --transfer-image-to $(oc get route/default-route -n openshift-image-registry --template='{{ .spec.host }}')/$NAMESPACE \ --cluster-pull-prefix image-registry.openshift-image-registry.svc:5000/$NAMESPACE \ --target-registry-username $(oc whoami) \ --target-registry-password $(oc whoami -t) \ --accept-all-licenses \ --insecure-skip-tls-verify

    ------------------------------
    Zhi Hui Huang
    ------------------------------



  • 12.  RE: WKC and WS Installation failures

    Posted Mon March 29, 2021 08:34 PM

    Try using ocadmin with ROKs vs oc whoami.

    https://tjmcmanus.github.io/IBMPartnerDemo/Data35.html#installing-the-cloud-pak-for-data-control-plane-by-command-line



    ------------------------------
    Tom McManus
    ------------------------------