Cloud Pak for Data Group

Expand all | Collapse all

IBM Watson Discovery add-on installation on Cloud Pak for Data

  • 1.  IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Wed September 04, 2019 10:17 AM
    Hello,
    I have a Cloud Pak for Data V2.1 installation and I would like to install the Watson Discovery V2.0.0 add-on.
    I have followed the instructions found at https://www.ibm.com/support/knowledgecenter/en/SSQNUZ_2.1.0/com.ibm.icpdata.doc/watson/discovery-install.html
    I followed each step of the installation procedure. Everything was OK, until the Helm installation command.
    Here is the command I ran :
    helm install --name ${name} zenrepos/ibm-watson-discovery-prod --tls --set global.license=accept --set core.common.persistence.storageClassName=oketi-nfs --set core.ingestion.mount.storageClassName=oketi-nfs --set elastic.persistence.storageClassName=oketi-nfs --set etcd.persistence.storageClassName=oketi-nfs --set hdp.persistence.storageClassName=oketi-nfs --set minio.persistence.storageClass=oketi-nfs --set postgresql.persistence.storageClassName=oketi-nfs --set elastic.backup.persistence.storageClassName=oketi-nfs --set global.icp.masterHostname=icp4bsc.nice.icp --set global.icp.masterIP=192.168.66.150
    (I initialized the name variable to discovery)
    Here is the error I get : Error: timed out waiting for the condition
    Would you have an idea on how to avoid this error ?
    I could not find any information related to this error message in the documentation.
    Many thanks in advance for any help on this !

    ------------------------------
    Valerie Le Roy
    ------------------------------


  • 2.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Wed September 04, 2019 11:08 AM
    Hi Valerie,

    Could you please check if there is any pod, deployment and job created for that chart
    kubectl get --namespace=<my-namespace> configmaps,jobs,pods,statefulsets,deployments,roles,rolebindings,secrets,serviceaccounts --selector=release=<my-release>

    I am curious if it's just a time-out issue where the install just took too long and so the post-install jobs never ran. So, for that we can add --timeout 1200 in helm install command.

    So first clean up the previous install by running the following commands in order:

    helm delete --purge --tls <my-release>

    kubectl delete --namespace=<my-namespace> configmaps,jobs,pods,statefulsets,deployments,roles,rolebindings,secrets,serviceaccounts --selector=release=<my-release>

    kubectl delete --namespace=<my-namespace> persistentvolumeclaims --selector=release=<my-release>
     Remove persistent volumes ,

    kubectl delete persistentvolumes $(kubectl get persistentvolumes \

      --output=jsonpath='{range .items[*]}{@.metadata.name}:{@.status.phase}:{@.spec.claimRef.name}{"\n"}{end}' \

      | grep ":Released:" \

      | grep "<my-release>-" \

      | cut -d ':' -f 1)



    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 3.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Thu September 05, 2019 08:52 AM
    Hi Sidrah,

    Thanks a lot for your answer !
    I confirm that some resources were created after the helm install command. So I deleted them by using the commands you provided.
    Then I ran again the install command with the timeout parameter:
    helm install --name ${name} zenrepos/ibm-watson-discovery-prod --tls --set global.license=accept --set core.common.persistence.storageClassName=oketi-nfs --set core.ingestion.mount.storageClassName=oketi-nfs --set elastic.persistence.storageClassName=oketi-nfs --set etcd.persistence.storageClassName=oketi-nfs --set hdp.persistence.storageClassName=oketi-nfs --set minio.persistence.storageClass=oketi-nfs --set postgresql.persistence.storageClassName=oketi-nfs --set elastic.backup.persistence.storageClassName=oketi-nfs --set global.icp.masterHostname=icp4bsc.nice.icp --set global.icp.masterIP=192.168.66.150 --timeout 1200
    I got the following error message : Error: watch closed before Until timeout
    Please note that I have a high-availability configuration, with 3 master nodes. So the global.icp.masterIP parameter is set to the IP address of the cluster VIP. And in the /etc/hosts file, I have added the following entry (as described in the installation procedure) :
    192.168.66.150    mycluster.icp

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 4.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Thu September 05, 2019 11:43 AM
    Edited by Sidrah Junaid Thu September 05, 2019 11:43 AM
    Hi Valerie,

    MasterHostname should be mycluster.icp. masterIP should be the master hostIP (not the load balancer).

    In addition to that, could you please run
    kubectl get pods -l release=${name}​
     using the same value for name as you used in that install command and share the result .

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 5.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Fri September 06, 2019 02:54 AM
    Edited by Valerie Le Roy Fri September 06, 2019 08:53 AM
    Hi Sidrah,
    Again, thanks a lot for your answer !
    Please find hereafter the results of the kubectl get pods command:
    Thanks for your support !

    I also ran again the command by using mycluster.icp for the hostname and IP address of the master (after the appropriate modifications in the /etc/hosts file). I got again the Error: watch closed before Until timeout error message.

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 6.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Fri September 06, 2019 10:57 AM
    Hi Valerie,

    Can you please share the description of one of the Pending pod?

    kubectl describe pods <pod-name> -n <namespace>

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 7.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Mon September 09, 2019 05:25 AM
    Hi Sidrah,

    Here is the description of one of the pending pods :
    [root@icp4bsc-master3 InstallPackage]# kubectl describe pods discovery-watson-discovery-elastic-0
    Name: discovery-watson-discovery-elastic-0
    Namespace: default
    Priority: 0
    PriorityClassName: <none>
    Node: <none>
    Labels: app=watson-discovery
    chart=elastic
    component=elastic
    controller-revision-hash=discovery-watson-discovery-elastic-dbdd68546
    heritage=Tiller
    release=discovery
    statefulset.kubernetes.io/pod-name=discovery-watson-discovery-elastic-0
    Annotations: kubernetes.io/psp: ibm-privileged-psp
    productID: ICP4D-addon-5802bdd0bc424e2488698c0a8f82cfb8-discovery
    productName: IBM Watson Discovery for IBM Cloud Private for Data
    productVersion: 2.0.0
    Status: Pending
    IP:
    Controlled By: StatefulSet/discovery-watson-discovery-elastic
    Containers:
    elastic:
    Image: icp4bsc.nice.icp:8500/default/ibm-wex-ee:12.0.3.927
    Ports: 9200/TCP, 9300/TCP
    Host Ports: 0/TCP, 0/TCP
    Limits:
    cpu: 99
    memory: 4Gi
    Requests:
    cpu: 750m
    memory: 2Gi
    Liveness: exec [bash -c ${WEX_HOME}/sbin/healthcheck.sh -w -t 5] delay=210s timeout=5s period=15s #success=1 #failure=3
    Readiness: exec [bash -c ${WEX_HOME}/sbin/healthcheck.sh -w -t 5] delay=30s timeout=5s period=15s #success=1 #failure=12
    Environment:
    WEX_ALL_COMPONENTS: elastic
    ELASTIC_REPLICAS: 1
    ELASTIC_BACKUP_DIR: /backup-data
    ELASTIC_ENDPOINT: https://localhost:9200
    ZEN_MARK_TLS: true
    ELASTIC_CERT_FILE: /opt/tls/elastic/tls.crt
    ELASTIC_KEY_FILE: /opt/tls/elastic/tls.key
    ELASTIC_CACERT: /opt/tls/elastic/ca.crt
    ELASTIC_USER: <set to the key 'username' in secret 'discovery-watson-discovery-elastic-secret'> Optional: false
    ELASTIC_PASSWORD: <set to the key 'password' in secret 'discovery-watson-discovery-elastic-secret'> Optional: false
    MANAGEMENT_SERVER_PORT: 9080
    MANAGEMENT_SERVER_SECURE_PORT: 9443
    MANAGEMENT_SERVER_ENDPOINT: http://discovery-watson-discovery-gateway:$(MANAGEMENT_SERVER_PORT)/wex/api/v1
    MANAGEMENT_SERVER_SECURE_ENDPOINT: https://discovery-watson-discovery-gateway:$(MANAGEMENT_SERVER_SECURE_PORT)/wex/api/v1
    RANKER_MODEL_MESH: discovery-watson-discovery-serve-ranker.default.svc.cluster.local:8033
    TRAINING_DATA_CRUD: discovery-watson-discovery-training-data-crud.default.svc.cluster.local:50051
    Mounts:
    /backup-data from backup-data (rw)
    /opt/tls/elastic from certificate (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from discovery-watson-discovery-shared-svc-acc-token-48g2k (ro)
    /wexdata from data (rw)
    Conditions:
    Type Status
    PodScheduled False
    Volumes:
    data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: data-discovery-watson-discovery-elastic-0
    ReadOnly: false
    certificate:
    Type: Secret (a volume populated by a Secret)
    SecretName: discovery-watson-discovery-tls-secret
    Optional: false
    backup-data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: discovery-watson-discovery-elastic-backup-data
    ReadOnly: false
    discovery-watson-discovery-shared-svc-acc-token-48g2k:
    Type: Secret (a volume populated by a Secret)
    SecretName: discovery-watson-discovery-shared-svc-acc-token-48g2k
    Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
    node.kubernetes.io/not-ready:NoExecute for 300s
    node.kubernetes.io/unreachable:NoExecute for 300s
    Events: <none>
    [root@icp4bsc-master3 InstallPackage]#

    Thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 8.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Tue September 10, 2019 11:37 AM
    Hi ,

    I would suggest to use v2.0.1 as it has lots of improvements and you need to run deploy.sh instead of helm install.

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 9.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Wed September 11, 2019 03:21 AM
    Hi Sidrah,

    I already started the installation of version 2.0.1 of Discovery, following the steps described in the installation procedure. I will post the results of the installation in the community.
    Thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 10.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Wed September 11, 2019 09:45 AM
    Unfortunately, the 2.0.1 version installation with deploy.sh also failed.
    I have the following log messages :
    [kube] 2019/09/11 09:37:18 zen-discovery201-watson-discovery-elastic-init-plugin-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
    [tiller] 2019/09/11 09:42:28 warning: Release zen-discovery201 post-install ibm-watson-discovery-prod/charts/elastic/templates/init-plugin-job.yaml could not complete: watch closed before Until timeout
    [tiller] 2019/09/11 09:42:28 warning: Release "zen-discovery201" failed post-install: watch closed before Until timeout
    [storage] 2019/09/11 09:42:28 updating release "zen-discovery201.v1"
    [tiller] 2019/09/11 09:42:28 failed install perform step: watch closed before Until timeout

    Any idea ?

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 11.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Thu September 12, 2019 11:25 AM
    Hi,
    I could solve my previous installation problem by entering the sysctl -w vm.max_map_count=262144  command on my cluster nodes to increase the system's virtual memory max map count.
    Anyway, I still have a pod in an init state : zen-discov201-watson-discovery-orchestrator-0 with the following error message in the logs :
    Error from server (BadRequest): container "orchestrator" in pod "zen-discov201-watson-discovery-orchestrator-0" is waiting to start: PodInitializing

    Many thanks in advance for any help !

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 12.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Thu September 12, 2019 11:40 AM
    Hi,

    That's great!.
    Regarding pod in init state, can you check the description (especially the Event section) of that pod.

    ------------------------------
    Sidrah Junaid
    ------------------------------



  • 13.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Fri September 13, 2019 05:09 AM
    Hi Sidrah,

    Here are the messages in the Event section :
    Events:
    Type          Reason          Age      From                                         Message
    ----             ------              ----      ----                                            -------
    Normal     Scheduled    43s      default-scheduler                 Successfully assigned zen/zen-discov201-watson-discovery-orchestrator-0 to 192.168.66.143
    Normal     Pulling           42s       kubelet, 192.168.66.143  pulling image "icp4bsc.nice.icp:8500/zen/ibm-wex-ee:12.0.3.1055"
    Normal     Pulled            42s       kubelet, 192.168.66.143  Successfully pulled image "icp4bsc.nice.icp:8500/zen/ibm-wex-ee:12.0.3.1055"
    Normal     Created         42s       kubelet, 192.168.66.143  Created container
    Normal     Started          41s       kubelet, 192.168.66.143  Started container

    And here are more details on the pod :
    Annotations:
    kubernetes.io/psp: ibm-discovery-prod-psp
    productID: ICP4D-addon-5802bdd0bc424e2488698c0a8f82cfb8-discovery
    productName: IBM Watson Discovery for IBM Cloud Pak for Data
    productVersion: 2.0.0
    seccomp.security.alpha.kubernetes.io/pod: docker/default

    ------------------------------
    Valerie Le Roy
    ------------------------------



  • 14.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Fri September 13, 2019 05:52 AM
      |   view attached
    Here is the full Pod description :

    ------------------------------
    Valerie Le Roy
    ------------------------------

    Attachment(s)

    txt
    PodDescription.txt   6K 1 version


  • 15.  RE: IBM Watson Discovery add-on installation on Cloud Pak for Data

    Posted Fri September 13, 2019 11:32 AM
    I investigated the problem and I saw that one of the PersistentVolumeClaim (watson-discovery-ingestion-userdata) is not bound :

    It seems that the claim selector is not supported by my StorageClass (NFS).
    Here is the selector :
    "selector": {
    "matchExpressions": [
    {
    "key": "assign-to",
    "operator": "In",
    "values": [
    "wex-userdata"
    ]
    }
    ]
    },

    Does this mean that I need to create a persistent volume corresponding to this selector before installing the add-on ?

    Many thanks for your help !

    ------------------------------
    Valerie Le Roy
    ------------------------------