Cloud Pak for Data

 View Only
Expand all | Collapse all

Data Virtualization Provisioning failure

Xin Lu

Xin LuWed July 08, 2020 09:24 AM

Xin Lu

Xin LuWed July 08, 2020 09:24 AM

  • 1.  Data Virtualization Provisioning failure

    Posted Tue July 07, 2020 11:09 PM

    I am trying to provision Data Virtualization on my cluster but it is stuck at 0%. I am guessing the reason to be resource limitation on the cluster. How do I provision  DV by scaling up my nodes or how do I install it on another node?

    I have also checked the logs for dv-related pod and this is what I see but I don't know how to interpret them:

    [INFO] The hostname is dv-addon.workshop-assets.svc.cluster.local
    [INFO] The namespace is workshop-assets
    [INFO] The domain is cluster.local
    time="2020-07-07T06:03:56Z" level=info msg="Curr wrk dir " cwd=/content/bin/authproxy
    time="2020-07-07T06:03:56Z" level=info msg="processing .. " dsx_auth_config_file=/content/etc/config.json
    time="2020-07-07T06:03:56Z" level=info msg=" AuthProxy HTTP Service starting " httpsPort=":8443" port=":8080"
    time="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/static
    time="2020-07-07T06:03:56Z" level=info msg=" static handle -  creating serving route" currAlias=/static dir=/content/resources
    time="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/add-ons
    time="2020-07-07T06:03:56Z" level=info msg=" static handle -  creating serving route" currAlias=/add-ons dir=/content/add-ons
    time="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/charts
    time="2020-07-07T06:03:56Z" level=info msg=" static handle -  creating serving route" currAlias=/charts dir=/content/charts
    time="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/webpage
    time="2020-07-07T06:03:56Z" level=info msg=" static handle -  creating serving route" currAlias=/webpage dir=/content/webpage


    ------------------------------
    Shivam Solanki
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 03:57 AM
    Hi,

    The db pods are most likely not starting, that's why you are stuck at 100%.

    Please run:
    kubectl get po --no-headers --all-namespaces -o wide| grep -Ev '([[:digit:]])/\1.*R' | grep -v 'Completed'

    check if there are pods stuck at Pending, and describe them if you can.

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 3.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:10 PM

    Hi Tomasz,

    The output of the kubectl command is shown below and I don't see `dv` pod pending here

    workshop-assets                     cassandra-0                                                                          0/1       Pending     0         1d        <none>           <none>          <none>
    workshop-assets                     iis-services-5bf5c47c46-xrr29                                                        0/1       Init:0/2    0         1d        172.30.93.114    10.185.199.3    <none>
    workshop-assets                     iis-xmetarepo-5996849f67-7rp8s                                                       0/1       Pending     0         1d        <none>           <none>          <none>
    workshop-assets                     is-en-conductor-0                                                                    0/1       Init:1/2    0         1d        172.30.211.255   10.185.199.53   <none>
    workshop-assets                     kafka-0                                                                              0/1       Pending     0         1d        <none>           <none>          <none>
    workshop-assets                     solr-0                                                                               0/1       Pending     0         1d        <none>           <none>          <none>
    workshop-assets                     solr-configset-collection-8vnfh                                                      0/1       Pending     0         1d        <none>           <none>          <none>
    workshop-assets                     wdp-couchdb-setup-job-bh66h                                                          0/1       Error       0         1d        172.30.93.97     10.185.199.3    <none>
    workshop-assets                     wdp-couchdb-setup-job-sphg9                                                          0/1       Error       0         1d        172.30.211.225   10.185.199.53   <none>
    workshop-assets                     zookeeper-0                                                                          0/1       Pending     0         1d        <none>           <none>          <none>
    


    ------------------------------
    Shivam Solanki
    ------------------------------



  • 4.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:10 PM

    Also, I check the dv pods and they appear to be running:

    dv-addon-7cbbdf5bc8-pv5pl                                                            1/1     Running     0          19h
    dv-service-provider-58566695b8-cbmx9                                                 1/1     Running     0          1d


    ------------------------------
    Shivam Solanki
    ------------------------------



  • 5.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:36 PM
    Hi,

    in fact these are only the initial DV pods, not the instance you created from UI.

    they are most likely stuck at ReplicaSet level, due to missing kernel settings, etc.

    the fact that you already have other pods pending suggests that your cluster is missing resources.

    Please check for dv related rs:

    oc get rs | grep dv

    describe the rs to check why they fail.

    thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 6.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:51 PM

    Is there a way to upgrade the existing worker nodes to make sure there are enough available resources. I checked the memory status:

    oc describe nodes | grep memory
    
      MemoryPressure   False   Wed, 08 Jul 2020 13:38:45 -0500   Tue, 07 Jul 2020 11:29:46 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
      memory:         32774920Ki
      memory:         28987144Ki
      memory             440850Ki (1%)  658800640 (2%)
      MemoryPressure   False   Wed, 08 Jul 2020 13:38:47 -0500   Mon, 06 Jul 2020 17:41:53 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
      memory:         65785812Ki
      memory:         60017620Ki
      memory             27400722Ki (45%)  71255228416 (115%)
      MemoryPressure   False   Wed, 08 Jul 2020 13:38:43 -0500   Mon, 06 Jul 2020 17:38:36 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
      memory:         65785812Ki
      memory:         60017620Ki
      memory             28320274Ki (47%)  82100512Ki (136%)
      MemoryPressure   False   Wed, 08 Jul 2020 13:38:46 -0500   Tue, 07 Jul 2020 13:56:28 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
      memory:         32774920Ki
      memory:         28987144Ki
      memory             338450Ki (1%)  553943040 (1%)
      MemoryPressure   False   Wed, 08 Jul 2020 13:38:47 -0500   Mon, 06 Jul 2020 17:41:24 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
      memory:         65785812Ki
      memory:         60017620Ki
      memory             27675154Ki (46%)  98635644928 (160%)


    ------------------------------
    Shivam Solanki
    ------------------------------



  • 7.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:52 PM

    And describing rs:

    Shivams-MBP:cloudpak4data-ste-3.0.1 shivamsolanki$ oc describe rs dv-addon-7cbbdf5bc8
    Name:           dv-addon-7cbbdf5bc8
    Namespace:      workshop-assets
    Selector:       app.kubernetes.io/component=addon,app.kubernetes.io/instance=dv,app.kubernetes.io/managed-by=Tiller,app.kubernetes.io/name=dv,helm.sh/chart=ibm-dv-addon,icpdata_addon=true,pod-template-hash=3766891674,release=dv
    Labels:         app.kubernetes.io/component=addon
                    app.kubernetes.io/instance=dv
                    app.kubernetes.io/managed-by=Tiller
                    app.kubernetes.io/name=dv
                    helm.sh/chart=ibm-dv-addon
                    icpdata_addon=true
                    pod-template-hash=3766891674
                    release=dv
    Annotations:    deployment.kubernetes.io/desired-replicas: 1
                    deployment.kubernetes.io/max-replicas: 2
                    deployment.kubernetes.io/revision: 1
    Controlled By:  Deployment/dv-addon
    Replicas:       1 current / 1 desired
    Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
    Pod Template:
      Labels:       app.kubernetes.io/component=addon
                    app.kubernetes.io/instance=dv
                    app.kubernetes.io/managed-by=Tiller
                    app.kubernetes.io/name=dv
                    helm.sh/chart=ibm-dv-addon
                    icpdata_addon=true
                    pod-template-hash=3766891674
                    release=dv
      Annotations:  productID: ICP4D-IBMDataVirtualizationv1300_00000
                    productName: IBM Data Virtualization
                    productVersion: 1.3.0.0
      Containers:
       dv-addon:
        Image:       docker-registry.default.svc:5000/workshop-assets/dv-addon:v1.3.0.0-301
        Ports:       8080/TCP, 8443/TCP
        Host Ports:  0/TCP, 0/TCP
        Limits:
          cpu:     100m
          memory:  100Mi
        Requests:
          cpu:        100m
          memory:     100Mi
        Liveness:     http-get https://:https/add-ons/meta/add-ons.json delay=0s timeout=1s period=10s #success=1 #failure=3
        Readiness:    http-get https://:https/add-ons/meta/add-ons.json delay=0s timeout=1s period=10s #success=1 #failure=3
        Environment:  <none>
        Mounts:       <none>
      Volumes:        <none>
    Events:           <none>
    


    ------------------------------
    Shivam Solanki
    ------------------------------



  • 8.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 03:04 PM
    This is not the correct resource. Please check for sts, as this has changed in the recent version.
    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 9.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 03:18 PM
    Could you please tell me what sts is and how to fix this issue by matching the correct resource?
    Thanks

    ------------------------------
    Shivam Solanki
    ------------------------------



  • 10.  RE: Data Virtualization Provisioning failure

    Posted Thu July 09, 2020 03:17 AM

    Hi,

    Looking at the thread on Slack, I see that you are facing issues much earlier:

    cannot get namespaces in the namespace \"undefined\": no RBAC policy matched

    that's why we don't see any resources.

    Please open a support ticket via Salesforce, as I believe we need to do webex/thorough troubleshooting here.

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 11.  RE: Data Virtualization Provisioning failure

    Posted Mon July 13, 2020 12:34 AM

    Hi Tomasz,

    Do you have the link for opening ticket via Salesforce?

    I need to get this thing working as I have to conduct a CP4D workshop for clients/partners. Therefore, I need to understand how to fix installation issues.

    Thanks



    ------------------------------
    Shivam Solanki
    ------------------------------



  • 12.  RE: Data Virtualization Provisioning failure

    Posted Mon July 13, 2020 06:09 AM
    Hi,

    https://www.ibm.com/mysupport

    This is not a typical issue, that's why I'm asking you to open a ticket.

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 13.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 09:24 AM
    Edited by System Fri January 20, 2023 04:22 PM


  • 14.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 02:13 PM
    My cluster is on Red Hat Openshift and I dont see an option of upgrading the nodes. There are options for adding worker pools and then worker nodes on the new worker pools but no option to upgrade the existing worker node. 


    ------------------------------
    Shivam Solanki
    ------------------------------



  • 15.  RE: Data Virtualization Provisioning failure

    Posted Wed July 08, 2020 09:24 AM
    Edited by System Fri January 20, 2023 04:33 PM


  • 16.  RE: Data Virtualization Provisioning failure

    Posted Tue July 05, 2022 09:09 AM
    How to get same data from Salesforce report (Task and Events) - Activity in Tableau


    Please, somebody knows why the external storage is mentioned into that file? I don't see a permission request from this library to access there, but in that case, Why it mention external storage into that file? I couln't follow the code to understand it.


    There is a report built in Salesforce from Tasks and Events report type.

    Columns in the report pulled from 4 below fields that were autogenerated.

    enter image description here
    I need to recreate same report in Tableau + visualizations. It would be easy to accomplish if there would be exactly same data in Tableau when I connect to Salesforce.

    I don't see Tasks and Events table in the Standard Connection.

    ------------------------------
    pauls pavan
    ------------------------------