IBM Cloud Private for Data

Expand all | Collapse all

Issues with Admin dashboard on ICP4D (dsx-admin)

  • 1.  Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted 19 days ago
    Hi, we installed ICP4D over ICP and we have problem with admin pages (workers and monitoring)

    On Admin Dashboarda site there is 502 (image below). From K8s, ngnix:
    kubectl logs ibm-nginx-98bb8fddc-6l44p –f
    2019/01/28 14:37:26 [error] 579#579: *49581 upstream prematurely closed connection while reading response header from upstream, client: 10.1.219.64, server: localhost, request: "GET /api/v1/metrics/getNetworkAndMetricInfo HTTP/1.1", upstream: "http://10.0.129.117:3010/metrics/getNetworkAndMetricInfo", host: "172.17.255.240:31843", referrer: "https://172.17.255.240:31843/dsx-admin/"

    Na podstawie IP 10.0.129.117 znalazłem serwis dash-back, co doprowadziło mnie do tego poda :
    kubectl logs -n kube-system dash-back-deploy-5f9f97c7f4-269z2
    TypeError: Cannot read property 'transmitBytes' of undefined
    at /usr/src/app/package/lib/metrics-calculation.js:133:25
    at /usr/src/app/package/node_modules/mongoose/lib/model.js:4092:16
    at /usr/src/app/package/node_modules/mongoose/lib/aggregate.js:769:13
    at /usr/src/app/package/node_modules/kareem/index.js:131:16
    at args.(anonymous function) (/usr/local/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickCallback (internal/process/next_tick.js:104:9)
    2019-01-28T14:44:42: PM2 log: App [server:0] exited with code [1] via signal [SIGINT]
    2019-01-28T14:44:42: PM2 log: App [server:0] starting in -fork mode-
    2019-01-28T14:44:42: PM2 log: App [server:0] online
    (node:4751) DeprecationWarning: `openSet()` is deprecated in mongoose >= 4.11.0, use `openUri()` instead, or set the `useMongoClient` option if using `connect()` or `createConnection()`. See http://mongoosejs.com/docs/4.x/docs/connections.html#use-mongo-client



    On "Workers" sub-page we see only one from 150 workers but with no info
     

    Could you help with this?


    Thanks,
    Marcin

    ------------------------------
    Marcin Lewiarz
    ------------------------------


  • 2.  RE: Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted 16 days ago
    Hi Marcin,
    Could you please execute the following command and then get the result posted here?
    kubectl get po --all-namespaces -o wide| grep -Ev '1/1 .* R|2/2 .* R|3/3 .* R|4/4 .* R' | grep -v 'Completed'

    ------------------------------
    Hong Wei Jia
    ------------------------------



  • 3.  RE: Issues with Admin dashboard on ICP4D (dsx-admin)

    Posted 13 days ago

    Hi, thanks for reply. Below is output of command:

    [root@icp-master01 dv]# kubectl get po --all-namespaces -o wide| grep -Ev '1/1 .* R|2/2 .* R|3/3 .* R|4/4 .* R' | grep -v 'Completed'
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    istio-system   grafana-797c6b6df5-2bs6b                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-4rvmm                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-6gpls                                          0/1       Evicted     0          71d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-6qrpg                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-74bnj                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-7qc6d                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-858b8                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9hphs                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9lrc5                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-9tdg4                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-c5pmc                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-f9krp                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-fhvgr                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-ftb4m                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-h295m                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-h7qcb                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-hnc49                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-j9jd5                                          0/1       Evicted     0          65d       <none>           icp-management03   <none>
    istio-system   grafana-797c6b6df5-lfgkc                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-mttnl                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-qhbms                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-rpgmd                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-sdvw6                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-v6s9b                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-v8qdq                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-vvc89                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-wp7zm                                          0/1       Evicted     0          65d       <none>           icp-management02   <none>
    istio-system   grafana-797c6b6df5-xf298                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   grafana-797c6b6df5-zhx6p                                          0/1       Evicted     0          65d       <none>           icp-management01   <none>
    istio-system   kiali-7b67fc75f-rxnw7                                             0/1       Evicted     0          71d       <none>           icp-management02   <none>
    istio-system   prometheus-7dbd76ff7c-mlq74                                       0/1       Evicted     0          62d       <none>           icp-management02   <none>
    istio-system   servicegraph-c6645f9b8-dgjgl                                      0/1       Evicted     0          71d       <none>           icp-management03   <none>
    istio-system   servicegraph-c6645f9b8-xdpbd                                      0/1       Evicted     0          65d       <none>           icp-management02   <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    kube-system    key-management-crypto-6dbf76d447-4vdsh                            0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    key-management-lifecycle-5554cdf968-rs48h                         0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    key-management-persistence-67b568d7f7-g8gp2                       0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-client-7c9f8fb867-6ltcg                               0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-client-7c9f8fb867-h8sll                               0/2       Evicted     0          65d       <none>           icp-management03   <none>
    kube-system    logging-elk-kibana-6797485955-glbq5                               0/2       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-kibana-6797485955-x2lw8                               0/2       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-logstash-85594c9dc6-j57j4                             0/1       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-logstash-85594c9dc6-qztlk                             0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-master-df8d75676-6czzt                                0/1       Evicted     0          65d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-72kpz                                0/1       Evicted     0          64d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-78cf9                                0/1       Evicted     0          71d       <none>           icp-management02   <none>
    kube-system    logging-elk-master-df8d75676-ld6cs                                0/1       Evicted     0          71d       <none>           icp-management03   <none>
    kube-system    logging-elk-master-df8d75676-rd5bx                                0/1       Evicted     0          71d       <none>           icp-management01   <none>
    kube-system    metrics-server-5d76677cdb-2l9kk                                   0/1       Evicted     0          71d       <none>           icp-management03   <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    zen            decision-tree-1548851441210-driver                                0/1       Error       0          4d        10.1.28.83       icp-worker146      <none>
    zen            decision-tree-gen-1548851426524-driver                            0/1       Error       0          4d        10.1.109.138     icp-worker148      <none>
    NAMESPACE      NAME                                                              READY     STATUS      RESTARTS   AGE       IP               NODE               NOMINATED NODE
    zen            kmeans-1548847710498-driver                                       0/1       Error       0          4d        10.1.86.210      icp-worker103      <none>
    zen            kmeans-1548848207873-driver                                       0/1       Error       0          4d        10.1.83.23       icp-worker122      <none>
    zen            kmeans-1548849563400-driver                                       0/1       Error       0          4d        10.1.86.221      icp-worker103      <none>
    zen            kmeans-1548849563754-driver                                       0/1       Error       0          4d        10.1.51.158      icp-worker66       <none>
    zen            kmeans-1548849563787-driver                                       0/1       Error       0          4d        10.1.51.159      icp-worker66       <none>
    zen            kmeans-1548849573843-driver                                       0/1       Error       0          4d        10.1.86.222      icp-worker103      <none>
    zen            kmeans-1548850301446-driver                                       0/1       Error       0          4d        10.1.86.225      icp-worker103      <none>
    zen            kmeans-gen-1548847698308-driver                                   0/1       Error       0          4d        10.1.86.209      icp-worker103      <none>
    zen            kmeans-gen-1548849547599-driver                                   0/1       Error       0          4d        10.1.86.219      icp-worker103      <none>
    zen            kmeans-gen-1548849547625-driver                                   0/1       Error       0          4d        10.1.83.28       icp-worker122      <none>
    zen            kmeans-gen-1548849547627-driver                                   0/1       Error       0          4d        10.1.86.218      icp-worker103      <none>
    zen            kmeans-gen-1548849547673-driver                                   0/1       Error       0          4d        10.1.86.220      icp-worker103      <none>
    zen            kmeans-gen-1548850287644-driver                                   0/1       Error       0          4d        10.1.86.224      icp-worker103      <none>
    zen            pregel-operation-1548851611287-driver                             0/1       Error       0          4d        10.1.18.117      icp-worker10       <none>
    zen            pregel-operation-gen-1548851602764-driver                         0/1       Error       0          4d        10.1.86.228      icp-worker103      <none>
    zen            scc-1548851578056-driver                                          0/1       Error       0          4d        10.1.214.22      icp-worker117      <none>
    zen            scc-gen-1548851563215-driver                                      0/1       Error       0          4d        10.1.5.85        icp-worker125      <none>
    


    Regards,
    Marcin



    ------------------------------
    Marcin Lewiarz
    ------------------------------