Cloud Pak for Data

Cloud Pak for Data

Come for answers. Stay for best practices. All we’re missing is you.

 View Only

  • 1.  Resource Status

    Posted 03/31/20 02:09 AM
      |   view attached
    Hello

    Our Cloud Pak for Data web console is always loading status, so I reviewed its resource staus.
    Watching it on grafana, I am worried about Memory satulation showing the high rate.
    Each of colour in the attached photo shows each of master/worker nodes, and I don't use swap on all nodes.

    It seems that other resource status such as CPU utilisation and Disk IO utilisation is not problem.
    It is between 10 and 30 percent on average.

    What brings this status? Is there something I have to do to improve memory satulation more?

    Warm regards,

    Chris

    ------------------------------
    Chris
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: Resource Status

    Posted 04/01/20 05:08 AM
    Hi,

    Your saturation is very high (see my test cluster for comparison, look at the scale difference) , what is the configuration of your cluster?
    Can you please share the outputs of:

    kubectl top node

    kubectl describe node | sed -n -e '/Total/,/Events/p' -e 's/Name:/\n---:>&/p'

    Can you explain what you mean by 'Our Cloud Pak for Data web console is always loading status' - are you saying its not performing well ?



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------



  • 3.  RE: Resource Status

    Posted 04/01/20 11:57 PM
    Hello, 


    Thank you for your reply.

    Our cluster cannot use "kubectl top" command as the message shows "the server is currently unable to handle the request (get nodes.metrics.k8s.io)",
    so I put only the result of "kubectl describe node" below.
    ------
    [root@master01 ~]# kubectl describe node | sed -n -e '/Total/,/Events/p' -e 's/Name:/\n---:>&/p'

    ---:>Name: master01
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 1105m (13%) 285m (3%)
    memory 1406Mi (4%) 380Mi (1%)
    Events:

    ---:>Name: master02
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 1285m (16%) 345m (4%)
    memory 1942Mi (6%) 420Mi (1%)
    Events:

    ---:>Name: master03
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 775m (9%) 35m (0%)
    memory 866Mi (2%) 100Mi (0%)
    Events:

    ---:>Name: worker01
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 31235m (97%) 242845m (758%)
    memory 115916Mi (90%) 134996Mi (104%)
    Events:

    ---:>Name: worker02
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 31430m (98%) 263045m (822%)
    memory 103482Mi (80%) 195644Mi (152%)
    Events:

    ---:>Name: worker03
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 29585m (92%) 31045m (97%)
    memory 81930Mi (63%) 86412Mi (67%)
    Events:

    ---:>Name: worker04
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource Requests Limits
    -------- -------- ------
    cpu 31980m (99%) 955669m (2986%)
    memory 106157Mi (82%) 319668Mi (248%)
    Events:


    What should I do to improve the cp4d performance more? 

    Thank you,
    Chris

    ------------------------------
    Chris
    ------------------------------

    Original Message


  • 4.  RE: Resource Status

    Posted 04/02/20 07:04 AM

    Hi,

    To enable top commands, please do:


    1. git clone https://github.com/kubernetes-sigs/metrics-server.git
    2. cd metrics-server; kubectl create -f deploy/kubernetes/
    3. oc edit deployments metrics-server -n kube-system and modify "kubernetes.io/arch: amd64" to "beta.kubernetes.io/arch: amd64"
    4. Once done, ensure the metric server pod is running oc get pods -n kube-system | grep metrics
    5. oc adm top node, oc adm top pods, kubectl top pods, kubectl top nodes and Administrator->Manage Platform should work now


    We can then inspect which pods are contributing the most to the utilization.

    What addons are you running ?

    Thanks



    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message