Cloud Pak for Data

Expand all | Collapse all

IBM Watson Discovery add-on installation on Cloud Pak for Data

  • 1.  IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 11 days ago
    Hello,
    I have a Cloud Pak for Data V2.1 installation and I would like to install the Watson Discovery V2.0.0 add-on.
    I have followed the instructions found at https://www.ibm.com/support/knowledgecenter/en/SSQNUZ_2.1.0/com.ibm.icpdata.doc/watson/discovery-install.html
    I followed each step of the installation procedure. Everything was OK, until the Helm installation command.
    Here is the command I ran :
    helm install --name ${name} zenrepos/ibm-watson-discovery-prod --tls --set global.license=accept --set core.common.persistence.storageClassName=oketi-nfs --set core.ingestion.mount.storageClassName=oketi-nfs --set elastic.persistence.storageClassName=oketi-nfs --set etcd.persistence.storageClassName=oketi-nfs --set hdp.persistence.storageClassName=oketi-nfs --set minio.persistence.storageClass=oketi-nfs --set postgresql.persistence.storageClassName=oketi-nfs --set elastic.backup.persistence.storageClassName=oketi-nfs --set global.icp.masterHostname=icp4bsc.nice.icp --set global.icp.masterIP=192.168.66.150
    (I initialized the name variable to discovery)
    Here is the error I get : Error: timed out waiting for the condition
    Would you have an idea on how to avoid this error ?
    I could not find any information related to this error message in the documentation.
    Many thanks in advance for any help on this !

    ------------------------------
    Valerie Le Roy
    ------------------------------


  • 2.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 11 days ago
    Hi Valerie,

    Could you please check if there is any pod, deployment and job created for that chart
    kubectl get --namespace=<my-namespace> configmaps,jobs,pods,statefulsets,deployments,roles,rolebindings,secrets,serviceaccounts --selector=release=<my-release>

    I am curious if it's just a time-out issue where the install just took too long and so the post-install jobs never ran. So, for that we can add --timeout 1200 in helm install command.

    So first clean up the previous install by running the following commands in order:

    helm delete --purge --tls <my-release>

    kubectl delete --namespace=<my-namespace> configmaps,jobs,pods,statefulsets,deployments,roles,rolebindings,secrets,serviceaccounts --selector=release=<my-release>

    kubectl delete --namespace=<my-namespace> persistentvolumeclaims --selector=release=<my-release>
     Remove persistent volumes ,

    kubectl delete persistentvolumes $(kubectl get persistentvolumes \

      --output=jsonpath='{range .items[*]}{@.metadata.name}:{@.status.phase}:{@.spec.claimRef.name}{"\n"}{end}' \

      | grep ":Released:" \

      | grep "<my-release>-" \

      | cut -d ':' -f 1)



    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 3.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 10 days ago
    Hi Sidrah,

    Thanks a lot for your answer !
    I confirm that some resources were created after the helm install command. So I deleted them by using the commands you provided.
    Then I ran again the install command with the timeout parameter:
    helm install --name ${name} zenrepos/ibm-watson-discovery-prod --tls --set global.license=accept --set core.common.persistence.storageClassName=oketi-nfs --set core.ingestion.mount.storageClassName=oketi-nfs --set elastic.persistence.storageClassName=oketi-nfs --set etcd.persistence.storageClassName=oketi-nfs --set hdp.persistence.storageClassName=oketi-nfs --set minio.persistence.storageClass=oketi-nfs --set postgresql.persistence.storageClassName=oketi-nfs --set elastic.backup.persistence.storageClassName=oketi-nfs --set global.icp.masterHostname=icp4bsc.nice.icp --set global.icp.masterIP=192.168.66.150 --timeout 1200
    I got the following error message : Error: watch closed before Until timeout
    Please note that I have a high-availability configuration, with 3 master nodes. So the global.icp.masterIP parameter is set to the IP address of the cluster VIP. And in the /etc/hosts file, I have added the following entry (as described in the installation procedure) :
    192.168.66.150    mycluster.icp

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 4.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 10 days ago
    Edited by Sidrah Junaid 10 days ago
    Hi Valerie,

    MasterHostname should be mycluster.icp. masterIP should be the master hostIP (not the load balancer).

    In addition to that, could you please run
    kubectl get pods -l release=${name}​
     using the same value for name as you used in that install command and share the result .

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 5.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 10 days ago
    Edited by Valerie Le Roy 9 days ago
    Hi Sidrah,
    Again, thanks a lot for your answer !
    Please find hereafter the results of the kubectl get pods command:
    Thanks for your support !

    I also ran again the command by using mycluster.icp for the hostname and IP address of the master (after the appropriate modifications in the /etc/hosts file). I got again the Error: watch closed before Until timeout error message.

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 6.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 9 days ago
    Hi Valerie,

    Can you please share the description of one of the Pending pod?

    kubectl describe pods <pod-name> -n <namespace>

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 7.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 7 days ago
    Hi Sidrah,

    Here is the description of one of the pending pods :
    [root@icp4bsc-master3 InstallPackage]# kubectl describe pods discovery-watson-discovery-elastic-0
    Name: discovery-watson-discovery-elastic-0
    Namespace: default
    Priority: 0
    PriorityClassName: <none>
    Node: <none>
    Labels: app=watson-discovery
    chart=elastic
    component=elastic
    controller-revision-hash=discovery-watson-discovery-elastic-dbdd68546
    heritage=Tiller
    release=discovery
    statefulset.kubernetes.io/pod-name=discovery-watson-discovery-elastic-0
    Annotations: kubernetes.io/psp: ibm-privileged-psp
    productID: ICP4D-addon-5802bdd0bc424e2488698c0a8f82cfb8-discovery
    productName: IBM Watson Discovery for IBM Cloud Private for Data
    productVersion: 2.0.0
    Status: Pending
    IP:
    Controlled By: StatefulSet/discovery-watson-discovery-elastic
    Containers:
    elastic:
    Image: icp4bsc.nice.icp:8500/default/ibm-wex-ee:12.0.3.927
    Ports: 9200/TCP, 9300/TCP
    Host Ports: 0/TCP, 0/TCP
    Limits:
    cpu: 99
    memory: 4Gi
    Requests:
    cpu: 750m
    memory: 2Gi
    Liveness: exec [bash -c ${WEX_HOME}/sbin/healthcheck.sh -w -t 5] delay=210s timeout=5s period=15s #success=1 #failure=3
    Readiness: exec [bash -c ${WEX_HOME}/sbin/healthcheck.sh -w -t 5] delay=30s timeout=5s period=15s #success=1 #failure=12
    Environment:
    WEX_ALL_COMPONENTS: elastic
    ELASTIC_REPLICAS: 1
    ELASTIC_BACKUP_DIR: /backup-data
    ELASTIC_ENDPOINT: https://localhost:9200
    ZEN_MARK_TLS: true
    ELASTIC_CERT_FILE: /opt/tls/elastic/tls.crt
    ELASTIC_KEY_FILE: /opt/tls/elastic/tls.key
    ELASTIC_CACERT: /opt/tls/elastic/ca.crt
    ELASTIC_USER: <set to the key 'username' in secret 'discovery-watson-discovery-elastic-secret'> Optional: false
    ELASTIC_PASSWORD: <set to the key 'password' in secret 'discovery-watson-discovery-elastic-secret'> Optional: false
    MANAGEMENT_SERVER_PORT: 9080
    MANAGEMENT_SERVER_SECURE_PORT: 9443
    MANAGEMENT_SERVER_ENDPOINT: http://discovery-watson-discovery-gateway:$(MANAGEMENT_SERVER_PORT)/wex/api/v1
    MANAGEMENT_SERVER_SECURE_ENDPOINT: https://discovery-watson-discovery-gateway:$(MANAGEMENT_SERVER_SECURE_PORT)/wex/api/v1
    RANKER_MODEL_MESH: discovery-watson-discovery-serve-ranker.default.svc.cluster.local:8033
    TRAINING_DATA_CRUD: discovery-watson-discovery-training-data-crud.default.svc.cluster.local:50051
    Mounts:
    /backup-data from backup-data (rw)
    /opt/tls/elastic from certificate (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from discovery-watson-discovery-shared-svc-acc-token-48g2k (ro)
    /wexdata from data (rw)
    Conditions:
    Type Status
    PodScheduled False
    Volumes:
    data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: data-discovery-watson-discovery-elastic-0
    ReadOnly: false
    certificate:
    Type: Secret (a volume populated by a Secret)
    SecretName: discovery-watson-discovery-tls-secret
    Optional: false
    backup-data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: discovery-watson-discovery-elastic-backup-data
    ReadOnly: false
    discovery-watson-discovery-shared-svc-acc-token-48g2k:
    Type: Secret (a volume populated by a Secret)
    SecretName: discovery-watson-discovery-shared-svc-acc-token-48g2k
    Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
    node.kubernetes.io/not-ready:NoExecute for 300s
    node.kubernetes.io/unreachable:NoExecute for 300s
    Events: <none>
    [root@icp4bsc-master3 InstallPackage]#

    Thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 8.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 5 days ago
    Hi ,

    I would suggest to use v2.0.1 as it has lots of improvements and you need to run deploy.sh instead of helm install.

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 9.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 5 days ago
    Hi Sidrah,

    I already started the installation of version 2.0.1 of Discovery, following the steps described in the installation procedure. I will post the results of the installation in the community.
    Thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 10.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 4 days ago
    Unfortunately, the 2.0.1 version installation with deploy.sh also failed.
    I have the following log messages :
    [kube] 2019/09/11 09:37:18 zen-discovery201-watson-discovery-elastic-init-plugin-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
    [tiller] 2019/09/11 09:42:28 warning: Release zen-discovery201 post-install ibm-watson-discovery-prod/charts/elastic/templates/init-plugin-job.yaml could not complete: watch closed before Until timeout
    [tiller] 2019/09/11 09:42:28 warning: Release "zen-discovery201" failed post-install: watch closed before Until timeout
    [storage] 2019/09/11 09:42:28 updating release "zen-discovery201.v1"
    [tiller] 2019/09/11 09:42:28 failed install perform step: watch closed before Until timeout

    Any idea ?

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 11.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 3 days ago
    Hi,
    I could solve my previous installation problem by entering the sysctl -w vm.max_map_count=262144  command on my cluster nodes to increase the system's virtual memory max map count.
    Anyway, I still have a pod in an init state : zen-discov201-watson-discovery-orchestrator-0 with the following error message in the logs :
    Error from server (BadRequest): container "orchestrator" in pod "zen-discov201-watson-discovery-orchestrator-0" is waiting to start: PodInitializing

    Many thanks in advance for any help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 12.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 3 days ago
    Hi,

    That's great!.
    Regarding pod in init state, can you check the description (especially the Event section) of that pod.

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 13.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 3 days ago
    Hi Sidrah,

    Here are the messages in the Event section :
    Events:
    Type          Reason          Age      From                                         Message
    ----             ------              ----      ----                                            -------
    Normal     Scheduled    43s      default-scheduler                 Successfully assigned zen/zen-discov201-watson-discovery-orchestrator-0 to 192.168.66.143
    Normal     Pulling           42s       kubelet, 192.168.66.143  pulling image "icp4bsc.nice.icp:8500/zen/ibm-wex-ee:12.0.3.1055"
    Normal     Pulled            42s       kubelet, 192.168.66.143  Successfully pulled image "icp4bsc.nice.icp:8500/zen/ibm-wex-ee:12.0.3.1055"
    Normal     Created         42s       kubelet, 192.168.66.143  Created container
    Normal     Started          41s       kubelet, 192.168.66.143  Started container

    And here are more details on the pod :
    Annotations:
    kubernetes.io/psp: ibm-discovery-prod-psp
    productID: ICP4D-addon-5802bdd0bc424e2488698c0a8f82cfb8-discovery
    productName: IBM Watson Discovery for IBM Cloud Pak for Data
    productVersion: 2.0.0
    seccomp.security.alpha.kubernetes.io/pod: docker/default

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 14.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 3 days ago
      |   view attached
    Here is the full Pod description :

    ------------------------------
    Valerie Le Roy
    ------------------------------

    Attachment(s)

    txt
    PodDescription.txt   6K 1 version


  • 15.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted 2 days ago
    I investigated the problem and I saw that one of the PersistentVolumeClaim (watson-discovery-ingestion-userdata) is not bound :

    It seems that the claim selector is not supported by my StorageClass (NFS).
    Here is the selector :
    "selector": {
    "matchExpressions": [
    {
    "key": "assign-to",
    "operator": "In",
    "values": [
    "wex-userdata"
    ]
    }
    ]
    },

    Does this mean that I need to create a persistent volume corresponding to this selector before installing the add-on ?

    Many thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------