IBM Cloud Private for Data

Expand all | Collapse all

Issues with Admin dashboard on ICP4D (dsx-admin)

  • 1.  Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted Mon January 28, 2019 11:26 AM
    Hi, we installed ICP4D over ICP and we have problem with admin pages (workers and monitoring)

    On Admin Dashboarda site there is 502 (image below). From K8s, ngnix:
    kubectl logs ibm-nginx-98bb8fddc-6l44p –f
    2019/01/28 14:37:26 [error] 579#579: *49581 upstream prematurely closed connection while reading response header from upstream, client: 10.1.219.64, server: localhost, request: "GET /api/v1/metrics/getNetworkAndMetricInfo HTTP/1.1", upstream: "http://10.0.129.117:3010/metrics/getNetworkAndMetricInfo", host: "172.17.255.240:31843", referrer: "https://172.17.255.240:31843/dsx-admin/"

    Na podstawie IP 10.0.129.117 znalazłem serwis dash-back, co doprowadziło mnie do tego poda :
    kubectl logs -n kube-system dash-back-deploy-5f9f97c7f4-269z2
    TypeError: Cannot read property 'transmitBytes' of undefined
    at /usr/src/app/package/lib/metrics-calculation.js:133:25
    at /usr/src/app/package/node_modules/mongoose/lib/model.js:4092:16
    at /usr/src/app/package/node_modules/mongoose/lib/aggregate.js:769:13
    at /usr/src/app/package/node_modules/kareem/index.js:131:16
    at args.(anonymous function) (/usr/local/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)
    2019-01-28T14:44:42: PM2 log: App [server:0] exited with code [1] via signal [SIGINT]
    2019-01-28T14:44:42: PM2 log: App [server:0] starting in -fork mode-
    2019-01-28T14:44:42: PM2 log: App [server:0] online
    (node:4751) DeprecationWarning: `openSet()` is deprecated in mongoose >= 4.11.0, use `openUri()` instead, or set the `useMongoClient` option if using `connect()` or `createConnection()`. See http://mongoosejs.com/docs/4.x/docs/connections.html#use-mongo-client



    On "Workers" sub-page we see only one from 150 workers but with no info
     

    Could you help with this?


    Thanks,
    Marcin

    ------------------------------
    Marcin Lewiarz
    ------------------------------


  • 2.  RE: Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted Fri February 01, 2019 01:52 AM
    Hi Marcin,
    Could you please execute the following command and then get the result posted here?
    kubectl get po --all-namespaces -o wide| grep -Ev '1/1 .* R|2/2 .* R|3/3 .* R|4/4 .* R' | grep -v 'Completed'

    ------------------------------
    Hong Wei Jia
    ------------------------------



  • 3.  RE: Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted Mon February 04, 2019 04:24 AM

    Hi, thanks for reply. Below is output of command:

    [root@icp-master01 dv]# kubectl get po --all-namespaces -o wide| grep -Ev '1/1 .* R|2/2 .* R|3/3 .* R|4/4 .* R' | grep -v 'Completed'
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    istio-system   grafana-797c6b6df5-2bs6b                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-4rvmm                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-6gpls                                          0/1       Evicted     0          71d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-6qrpg                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-74bnj                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-7qc6d                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-858b8                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9hphs                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9lrc5                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9tdg4                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-c5pmc                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-f9krp                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-fhvgr                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-ftb4m                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-h295m                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-h7qcb                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-hnc49                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-j9jd5                                          0/1       Evicted     0          65d       <none>           icp-management03   <none>
    istio-system   grafana-797c6b6df5-lfgkc                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-mttnl                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-qhbms                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-rpgmd                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-sdvw6                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-v6s9b                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-v8qdq                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-vvc89                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-wp7zm                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-xf298                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-zhx6p                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   kiali-7b67fc75f-rxnw7                                             0/1       Evicted     0          71d       <none>           icp-management02   <none>
    istio-system   prometheus-7dbd76ff7c-mlq74                                       0/1       Evicted     0          62d       <none>           icp-management02   <none>
    istio-system   servicegraph-c6645f9b8-dgjgl                                      0/1       Evicted     0          71d       <none>           icp-management03   <none>
    istio-system   servicegraph-c6645f9b8-xdpbd                                      0/1       Evicted     0          65d       <none>           icp-management02   <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    kube-system    key-management-crypto-6dbf76d447-4vdsh                            0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    key-management-lifecycle-5554cdf968-rs48h                         0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    key-management-persistence-67b568d7f7-g8gp2                       0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-client-7c9f8fb867-6ltcg                               0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-client-7c9f8fb867-h8sll                               0/2       Evicted     0          65d       <none>           icp-management03   <none>
    kube-system    logging-elk-kibana-6797485955-glbq5                               0/2       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-kibana-6797485955-x2lw8                               0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-logstash-85594c9dc6-j57j4                             0/1       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-logstash-85594c9dc6-qztlk                             0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-master-df8d75676-6czzt                                0/1       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-72kpz                                0/1       Evicted     0          64d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-78cf9                                0/1       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-ld6cs                                0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-master-df8d75676-rd5bx                                0/1       Evicted     0          71d       <none>           icp-management01   <none>
    kube-system    metrics-server-5d76677cdb-2l9kk                                   0/1       Evicted     0          71d       <none>           icp-management03   <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    zen            decision-tree-1548851441210-driver                                0/1       Error       0          4d        10.1.28.83       icp-worker146      <none>
    zen            decision-tree-gen-1548851426524-driver                            0/1       Error       0          4d        10.1.109.138     icp-worker148      <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    zen            kmeans-1548847710498-driver                                       0/1       Error       0          4d        10.1.86.210      icp-worker103      <none>
    zen            kmeans-1548848207873-driver                                       0/1       Error       0          4d        10.1.83.23       icp-worker122      <none>
    zen            kmeans-1548849563400-driver                                       0/1       Error       0          4d        10.1.86.221      icp-worker103      <none>
    zen            kmeans-1548849563754-driver                                       0/1       Error       0          4d        10.1.51.158      icp-worker66       <none>
    zen            kmeans-1548849563787-driver                                       0/1       Error       0          4d        10.1.51.159      icp-worker66       <none>
    zen            kmeans-1548849573843-driver                                       0/1       Error       0          4d        10.1.86.222      icp-worker103      <none>
    zen            kmeans-1548850301446-driver                                       0/1       Error       0          4d        10.1.86.225      icp-worker103      <none>
    zen            kmeans-gen-1548847698308-driver                                   0/1       Error       0          4d        10.1.86.209      icp-worker103      <none>
    zen            kmeans-gen-1548849547599-driver                                   0/1       Error       0          4d        10.1.86.219      icp-worker103      <none>
    zen            kmeans-gen-1548849547625-driver                                   0/1       Error       0          4d        10.1.83.28       icp-worker122      <none>
    zen            kmeans-gen-1548849547627-driver                                   0/1       Error       0          4d        10.1.86.218      icp-worker103      <none>
    zen            kmeans-gen-1548849547673-driver                                   0/1       Error       0          4d        10.1.86.220      icp-worker103      <none>
    zen            kmeans-gen-1548850287644-driver                                   0/1       Error       0          4d        10.1.86.224      icp-worker103      <none>
    zen            pregel-operation-1548851611287-driver                             0/1       Error       0          4d        10.1.18.117      icp-worker10       <none>
    zen            pregel-operation-gen-1548851602764-driver                         0/1       Error       0          4d        10.1.86.228      icp-worker103      <none>
    zen            scc-1548851578056-driver                                          0/1       Error       0          4d        10.1.214.22      icp-worker117      <none>
    zen            scc-gen-1548851563215-driver                                      0/1       Error       0          4d        10.1.5.85        icp-worker125      <none>
    


    Regards,
    Marcin



    ------------------------------
    Marcin Lewiarz
    ------------------------------