This is not a typical issue, that's why I'm asking you to open a ticket.
Original Message:
Sent: Mon July 13, 2020 12:33 AM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
Hi Tomasz,
Do you have the link for opening ticket via Salesforce?
I need to get this thing working as I have to conduct a CP4D workshop for clients/partners. Therefore, I need to understand how to fix installation issues.
Thanks
------------------------------
Shivam Solanki
Original Message:
Sent: Thu July 09, 2020 03:16 AM
From: TOMASZ HANUSIAK
Subject: Data Virtualization Provisioning failure
Hi,
Looking at the thread on Slack, I see that you are facing issues much earlier:
cannot get namespaces in the namespace \"undefined\": no RBAC policy matched
that's why we don't see any resources.
Please open a support ticket via Salesforce, as I believe we need to do webex/thorough troubleshooting here.
Thanks
------------------------------
TOMASZ HANUSIAK
Original Message:
Sent: Wed July 08, 2020 03:17 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
Could you please tell me what sts is and how to fix this issue by matching the correct resource?
Thanks
------------------------------
Shivam Solanki
Original Message:
Sent: Wed July 08, 2020 03:04 PM
From: TOMASZ HANUSIAK
Subject: Data Virtualization Provisioning failure
This is not the correct resource. Please check for sts, as this has changed in the recent version.
Thanks
------------------------------
TOMASZ HANUSIAK
Original Message:
Sent: Wed July 08, 2020 02:52 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
And describing rs:
Shivams-MBP:cloudpak4data-ste-3.0.1 shivamsolanki$ oc describe rs dv-addon-7cbbdf5bc8Name: dv-addon-7cbbdf5bc8Namespace: workshop-assetsSelector: app.kubernetes.io/component=addon,app.kubernetes.io/instance=dv,app.kubernetes.io/managed-by=Tiller,app.kubernetes.io/name=dv,helm.sh/chart=ibm-dv-addon,icpdata_addon=true,pod-template-hash=3766891674,release=dvLabels: app.kubernetes.io/component=addon app.kubernetes.io/instance=dv app.kubernetes.io/managed-by=Tiller app.kubernetes.io/name=dv helm.sh/chart=ibm-dv-addon icpdata_addon=true pod-template-hash=3766891674 release=dvAnnotations: deployment.kubernetes.io/desired-replicas: 1 deployment.kubernetes.io/max-replicas: 2 deployment.kubernetes.io/revision: 1Controlled By: Deployment/dv-addonReplicas: 1 current / 1 desiredPods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 FailedPod Template: Labels: app.kubernetes.io/component=addon app.kubernetes.io/instance=dv app.kubernetes.io/managed-by=Tiller app.kubernetes.io/name=dv helm.sh/chart=ibm-dv-addon icpdata_addon=true pod-template-hash=3766891674 release=dv Annotations: productID: ICP4D-IBMDataVirtualizationv1300_00000 productName: IBM Data Virtualization productVersion: 1.3.0.0 Containers: dv-addon: Image: docker-registry.default.svc:5000/workshop-assets/dv-addon:v1.3.0.0-301 Ports: 8080/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP Limits: cpu: 100m memory: 100Mi Requests: cpu: 100m memory: 100Mi Liveness: http-get https://:https/add-ons/meta/add-ons.json delay=0s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get https://:https/add-ons/meta/add-ons.json delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: <none> Volumes: <none>Events: <none>
------------------------------
Shivam Solanki
Original Message:
Sent: Wed July 08, 2020 02:51 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
Is there a way to upgrade the existing worker nodes to make sure there are enough available resources. I checked the memory status:
oc describe nodes | grep memory MemoryPressure False Wed, 08 Jul 2020 13:38:45 -0500 Tue, 07 Jul 2020 11:29:46 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available memory: 32774920Ki memory: 28987144Ki memory 440850Ki (1%) 658800640 (2%) MemoryPressure False Wed, 08 Jul 2020 13:38:47 -0500 Mon, 06 Jul 2020 17:41:53 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available memory: 65785812Ki memory: 60017620Ki memory 27400722Ki (45%) 71255228416 (115%) MemoryPressure False Wed, 08 Jul 2020 13:38:43 -0500 Mon, 06 Jul 2020 17:38:36 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available memory: 65785812Ki memory: 60017620Ki memory 28320274Ki (47%) 82100512Ki (136%) MemoryPressure False Wed, 08 Jul 2020 13:38:46 -0500 Tue, 07 Jul 2020 13:56:28 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available memory: 32774920Ki memory: 28987144Ki memory 338450Ki (1%) 553943040 (1%) MemoryPressure False Wed, 08 Jul 2020 13:38:47 -0500 Mon, 06 Jul 2020 17:41:24 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available memory: 65785812Ki memory: 60017620Ki memory 27675154Ki (46%) 98635644928 (160%)
------------------------------
Shivam Solanki
Original Message:
Sent: Wed July 08, 2020 02:35 PM
From: TOMASZ HANUSIAK
Subject: Data Virtualization Provisioning failure
Hi,
in fact these are only the initial DV pods, not the instance you created from UI.
they are most likely stuck at ReplicaSet level, due to missing kernel settings, etc.
the fact that you already have other pods pending suggests that your cluster is missing resources.
Please check for dv related rs:
oc get rs | grep dv
describe the rs to check why they fail.
thanks
------------------------------
TOMASZ HANUSIAK
Original Message:
Sent: Wed July 08, 2020 02:10 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
Also, I check the dv pods and they appear to be running:
dv-addon-7cbbdf5bc8-pv5pl 1/1 Running 0 19hdv-service-provider-58566695b8-cbmx9 1/1 Running 0 1d
------------------------------
Shivam Solanki
Original Message:
Sent: Wed July 08, 2020 02:09 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
Hi Tomasz,
The output of the kubectl command is shown below and I don't see `dv` pod pending here
workshop-assets cassandra-0 0/1 Pending 0 1d <none> <none> <none>workshop-assets iis-services-5bf5c47c46-xrr29 0/1 Init:0/2 0 1d 172.30.93.114 10.185.199.3 <none>workshop-assets iis-xmetarepo-5996849f67-7rp8s 0/1 Pending 0 1d <none> <none> <none>workshop-assets is-en-conductor-0 0/1 Init:1/2 0 1d 172.30.211.255 10.185.199.53 <none>workshop-assets kafka-0 0/1 Pending 0 1d <none> <none> <none>workshop-assets solr-0 0/1 Pending 0 1d <none> <none> <none>workshop-assets solr-configset-collection-8vnfh 0/1 Pending 0 1d <none> <none> <none>workshop-assets wdp-couchdb-setup-job-bh66h 0/1 Error 0 1d 172.30.93.97 10.185.199.3 <none>workshop-assets wdp-couchdb-setup-job-sphg9 0/1 Error 0 1d 172.30.211.225 10.185.199.53 <none>workshop-assets zookeeper-0 0/1 Pending 0 1d <none> <none> <none>
------------------------------
Shivam Solanki
Original Message:
Sent: Wed July 08, 2020 03:57 AM
From: TOMASZ HANUSIAK
Subject: Data Virtualization Provisioning failure
Hi,
The db pods are most likely not starting, that's why you are stuck at 100%.
Please run:
kubectl get po --no-headers --all-namespaces -o wide| grep -Ev '([[:digit:]])/\1.*R' | grep -v 'Completed'
check if there are pods stuck at Pending, and describe them if you can.
Thanks
------------------------------
TOMASZ HANUSIAK
Original Message:
Sent: Tue July 07, 2020 11:08 PM
From: Shivam Solanki
Subject: Data Virtualization Provisioning failure
I am trying to provision Data Virtualization on my cluster but it is stuck at 0%. I am guessing the reason to be resource limitation on the cluster. How do I provision DV by scaling up my nodes or how do I install it on another node?
I have also checked the logs for dv-related pod and this is what I see but I don't know how to interpret them:
[INFO] The hostname is dv-addon.workshop-assets.svc.cluster.local[INFO] The namespace is workshop-assets[INFO] The domain is cluster.localtime="2020-07-07T06:03:56Z" level=info msg="Curr wrk dir " cwd=/content/bin/authproxytime="2020-07-07T06:03:56Z" level=info msg="processing .. " dsx_auth_config_file=/content/etc/config.jsontime="2020-07-07T06:03:56Z" level=info msg=" AuthProxy HTTP Service starting " httpsPort=":8443" port=":8080"time="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/statictime="2020-07-07T06:03:56Z" level=info msg=" static handle - creating serving route" currAlias=/static dir=/content/resourcestime="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/add-onstime="2020-07-07T06:03:56Z" level=info msg=" static handle - creating serving route" currAlias=/add-ons dir=/content/add-onstime="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/chartstime="2020-07-07T06:03:56Z" level=info msg=" static handle - creating serving route" currAlias=/charts dir=/content/chartstime="2020-07-07T06:03:56Z" level=info msg=Alias currAlias=/webpagetime="2020-07-07T06:03:56Z" level=info msg=" static handle - creating serving route" currAlias=/webpage dir=/content/webpage
------------------------------
Shivam Solanki
------------------------------
#CloudPakforDataGroup