Cloud Pak for Data

Cloud Pak for Data

Come for answers. Stay for best practices. All we’re missing is you.

 View Only

  • 1.  ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Tue September 03, 2019 09:14 AM
    Hi Team,

    I am currently working on ICP4D installation on Existing ICP environment which has 3 Master node, 1 Proxy node and 3 worker nodes. 
    # cloudctl version
    Client Version: 3.1.2-1203+81b254e18da556ae1d9b683a9702e8420896dae9
    Server Version: 3.1.2-1203+81b254e18da556ae1d9b683a9702e8420896dae9

    After run installer.x86_64.466, during the installation ,there is a error log and interupt:
    Error:
    2019-08-30 14:09:08 UTC - Running command: /ibm/InstallPackage/components/dpctl --config /ibm/InstallPackage/components/install.yaml helm waitChartReady -r zen-0010-infra -t 60
    2019-08-30 14:13:42 UTC - Installation failed for /ibm/InstallPackage/components/../base_modules/0010-infra..

    Cloud you give me some advices,thanks.

    check the helm/pod
    # helm list --tls
    NAME REVISION UPDATED STATUS CHART NAMESPACE
    zen-0005-boot 1 Fri Aug 30 10:08:34 2019 DEPLOYED 0005-boot-0.0.2 zen
    zen-0010-infra 1 Fri Aug 30 10:09:04 2019 DEPLOYED 0010-infra-0.0.2 zen
    # kubectl get pods -n zen
    NAME READY STATUS RESTARTS AGE
    cloudant-757fd747bc-88qkc 1/1 Running 0 2d19h
    dsx-influxdb-858cf54497-5lrw4 1/1 Running 0 2d19h
    dsx-influxdb-set-auth-rgbzh 0/1 Completed 1 2d19h
    redis-7b6949b49c-7wwcj 1/1 Running 0 2d19h
    redis-7b6949b49c-ztvdd 1/1 Running 0 2d19h
    usermgmt-5c9994d7d-45t5f 1/1 Running 0 2d19h
    usermgmt-5c9994d7d-ldknn 1/1 Running 0 2d19h
    utils-api-859bb59568-fs4d4 1/1 Running 0 2d19h
    zen-metastoredb-0 0/1 CrashLoopBackOff 5 50m
    zen-metastoredb-1 0/1 CrashLoopBackOff 4 26m
    zen-metastoredb-2 0/1 CrashLoopBackOff 793 2d19h
    zen-metastoredb-init-cfhv4 1/1 Running 0 2d19h

    # kubectl logs zen-metastoredb-0 -n zen
    ++ hostname
    + exec /cockroach/cockroach start --logtostderr --insecure --advertise-host zen-metastoredb-0.zen-metastoredb.zen.svc.cluster.local --http-host 0.0.0.0 --http-port 8080 --port 26257 --cache 25% --max-sql-memory 25% --join zen-metastoredb-0.zen-metastoredb.zen.svc.cluster.local:26257,zen-metastoredb-1.zen-metastoredb.zen.svc.cluster.local:26257,zen-metastoredb-2.zen-metastoredb.zen.svc.cluster.local:26257

    # kubectl describe pod zen-metastoredb-0 -n zen
    Name: zen-metastoredb-0
    Namespace: zen
    Priority: 0
    PriorityClassName: <none>
    Node: 9.x.x.102/9.x.x.102
    Start Time: Mon, 02 Sep 2019 04:34:13 -0400
    Labels: app=0010-infra
    chart=0010-infra
    component=zen-metastoredb
    controller-revision-hash=zen-metastoredb-6fc7b7c5d6
    heritage=Tiller
    release=zen-0010-infra
    statefulset.kubernetes.io/pod-name=zen-metastoredb-0
    Annotations: kubernetes.io/psp: ibm-privileged-psp
    productID: ICP4D-IBMCloudPrivateForData_1210_perpetual_00000
    productName: IBM Cloud Private For Data
    productVersion: 1.2.1.0
    Status: Running
    IP: 10.1.133.202
    Controlled By: StatefulSet/zen-metastoredb
    Containers:
    zen-metastoredb:
    Container ID: docker://3f6988f700f86c14ffd2fab19201d567986299d8367cccd804e0b0137406e01c
    Image: cnwbzp2074.cn.dst.ibm.com:5000/zen/zen-metastoredb:v2.1.0.0
    Image ID: docker-pullable://master01:5000/zen/zen-metastoredb@sha256:4038c1ad72aef8760c24534781f971699b8f93d063c4d1982435611cfe892f9a
    Ports: 26257/TCP, 8080/TCP
    Host Ports: 0/TCP, 0/TCP
    Command:
    /bin/bash
    -ecx
    exec /cockroach/cockroach start --logtostderr --insecure --advertise-host $(hostname).${STATEFULSET_FQDN} --http-host 0.0.0.0 --http-port 8080 --port 26257 --cache 25% --max-sql-memory 25% --join ${STATEFULSET_NAME}-0.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-1.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-2.${STATEFULSET_FQDN}:26257
    State: Waiting
    Reason: CrashLoopBackOff
    Last State: Terminated
    Reason: Error
    Exit Code: 132
    Started: Mon, 02 Sep 2019 05:32:55 -0400
    Finished: Mon, 02 Sep 2019 05:32:55 -0400
    Ready: False
    Restart Count: 7
    Limits:
    cpu: 500m
    memory: 1Gi
    Requests:
    cpu: 50m
    memory: 128Mi
    Liveness: http-get http://:http/health delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness: http-get http://:http/health%3Fready=1 delay=30s timeout=1s period=10s #success=1 #failure=2
    Environment:
    STATEFULSET_NAME: zen-metastoredb
    STATEFULSET_FQDN: zen-metastoredb.zen.svc.cluster.local
    COCKROACH_CHANNEL: kubernetes-helm
    Mounts:
    /cockroach/cockroach-data from datadir (rw)
    /var/run/secrets/kubernetes.io/serviceaccount from icpd-anyuid-sa-token-wrt9j (ro)
    Conditions:
    Type Status
    Initialized True
    Ready False
    ContainersReady False
    PodScheduled True
    Volumes:
    datadir:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: datadir-zen-metastoredb-0
    ReadOnly: false
    icpd-anyuid-sa-token-wrt9j:
    Type: Secret (a volume populated by a Secret)
    SecretName: icpd-anyuid-sa-token-wrt9j
    Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
    node.kubernetes.io/not-ready:NoExecute for 300s
    node.kubernetes.io/unreachable:NoExecute for 300s
    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal Scheduled 59m default-scheduler Successfully assigned zen/zen-metastoredb-0 to 9.x.x.102
    Normal Started 58m (x4 over 59m) kubelet, 9.x.x.102 Started container
    Normal Pulled 58m (x5 over 59m) kubelet, 9.x.x.102 Container image "cnwbzp2074.cn.dst.ibm.com:5000/zen/zen-metastoredb:v2.1.0.0" already present on machine
    Normal Created 58m (x5 over 59m) kubelet, 9.x.x.102 Created container
    Warning BackOff 4m35s (x268 over 59m) kubelet, 9.x.x.102 Back-off restarting failed container

    ------------------------------
    wei long
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Tue September 03, 2019 09:29 AM

    Hi,

    Do you know if you have modified the default cluster domain name for ICP (cluster.local) ?

    Can you please  collect the logs for the previous instance of zen-metastoredb-0:
    kubectl logs zen-metastoredb-0 -n zen -p

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 3.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Tue September 03, 2019 09:36 PM
    hi , TOMASZ
    Thanks for your reply.

    # kubectl logs zen-metastoredb-0 -n zen -p
    ++ hostname
    + exec /cockroach/cockroach start --logtostderr --insecure --advertise-host zen-metastoredb-0.zen-metastoredb.zen.svc.cluster.local --http-host 0.0.0.0 --http-port 8080 --port 26257 --cache 25% --max-sql-memory 25% --join zen-metastoredb-0.zen-metastoredb.zen.svc.cluster.local:26257,zen-metastoredb-1.zen-metastoredb.zen.svc.cluster.local:26257,zen-metastoredb-2.zen-metastoredb.zen.svc.cluster.local:26257

    # ss -tunlp|grep 26257|wc -l
    0
    # ss -tunlp|grep 8080|wc -l
    0



    ​​

    ------------------------------
    wei long
    ------------------------------

    Original Message


  • 4.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Wed September 04, 2019 09:22 AM

    Hi,

    Can you check if all of your nodes have AVX feature enabled?:

    lscpu | grep -i avx

    can you explain why 2.1.0.0 is being installed instead of 2.1.0.2 ?

    I'm aware of a defect with Cocroachdb in 2.1.0.0 blocking deployment on certain virtualization, it was resolved in later versions.

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 5.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Wed September 04, 2019 10:02 AM
    Hi TOMASZ,
    Good day!

    I think zen-metastoredb v2.1.0.0 is  included in ICP4D  installer.x86_64.466, installed by default.
    and lscpu info  as follow，thanks.

    # lscpu | grep -i avx
    Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts

    ------------------------------
    wei long
    ------------------------------

    Original Message


  • 6.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Wed September 04, 2019 10:22 AM
    Hi,

    Yes, but there is a newer installer  -> installer.x86_64.520 which is from 2.1.0.2 release.
    It seems that you are in fact missing a AVX2 feature

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 7.  RE: ICP4D installation failed on Existing ICP 3.1.2 enviornment

    Posted Thu September 05, 2019 07:24 AM

    I will download the 2.1.0.2 for re-install，thanks  TOMASZ.

    ------------------------------
    wei long
    ------------------------------

    Original Message


Related Content