High Performance Computing Group

 View Only
  • 1.  [LSF Explorer]Create sample dashboard in Kibana

    Posted Tue December 22, 2020 08:06 AM
    Edited by System Fri January 20, 2023 04:12 PM

    Description

    This document introduces the steps of how to generate useful dashboard in Kibana, based on the data collected by LSF Explorer.

    Modules of Kibana

    • Management
    • Visualization
    • Dashboard

    Kibana version:7.2.1

    The data collected by LSF Explore used in visualization

    Fields:

    Index

    Fields

    Description

    lsf_job_acct 

    max_mem

    Job maximum memory usage, in bytes

    lsf_job_acct

    resource_req.mem.min

    Memory resource requirement string describes the memory   resources a job needs

    lsf_job_acct

    cluster_name

    LSF cluster name

    lsf_job_acct

    queue_name

    Queue to which the job was submitted

    lsf_job_acct

    event_time

    ISO-formatted event time in the cluster local time zone

    Lsf_job_acct

    job_id

    LSF Job id

    Lsf_job_acct

    run_time

    Job run time

            * lsf_job_acct index - Information about done and exited LSF jobs (DONE, EXIT).

                                                   Type of event is  JOB_FINISH

    Chart 1:Memory outlier

    This section will introduce how to generate a chart which shows the ratio of the actually used memory to the requested memory.

    A scripted field named percentiles (mem used/mem request) will be created and its threshold will be set to 60%. When the job's percentiles are less than 60%, it is considered as memory outlier.

    Assuming that the Kibana is connected to Elasticsearch, and lsb_acct index is used as the data source.

    • Add scripted field

    1. Go to Management > Kibana > Index Patterns
    2. Select the index pattern(lsb_acct* index) that will be added a scripted field to
    3. Go to the pattern's Scripted fields
    4. Click Add scripted field
    5. Enter a name for the scripted field, for example 'percentiles'
    6. Enter the expression that will be used to compute a value on the fly from the index script as below

    ((doc['max_mem'].value==0)?0:((doc['max_mem'].value/1024)/((doc['resource_req.mem.min'].size()==0)?(doc['max_mem'].value/1024):doc['resource_req.mem.min'].value)))

    This script is to get a ratio of memory usage and request memory

    1. Click Create field.
    • Create visualization

    1. Open Visualize to show the overview page
    2. Click Create new visualization. You'll see all the visualization types in Kibana
    3. Click Vertical Bar
    4. In Choose a source, select the lsb_acct* index pattern
    5. Show the number of job id per percentile along the Y-axis
      1. In the Metrics pane, expand Y-axis
      2. Set Aggregation to Unique Count
      3. Set Field to job_id
      4. Set Custom label to '# of jobs'

    6. Show the percentiles along the X-axis
      1. In the Buckets pane, click Add>X-axis
      2. Set Aggregation to Range
      3. Set Field to percentiles which defined before as scripted field
      4. Set 'from' and 'to' as below

      5. Set custom label to 'Memory used/Memory Request'
      6. In the buckets pane, click Add->Split Series
      7. Set Aggregation to Range
      8. Set Field to percentiles which defined before as scripted field
      9. Set 'from' and 'to' as below

      10. Click Apply changes. Then, the visualization should look like this

        *Red stands for outlier; Green stands for OK

      11. Save the visualization with name 'Memory outlier'.

    Chart 2:Req_mem VS Mem_used

    This section will introduce how to generate a chart that the actual job memory usage comparing with the memory request.

    1. Click Create new visualization. You'll see all the visualization types in Kibana
    2. Click Vertical Bar
    3. In Choose a source, select the lsb_acct* index pattern
    4. Show Average of memory usage per event time along the Y-axis
      1. In the Metrics pane, expand Y-axis
      2. Set Aggregation to 'Average'
      3. Set Field to 'Max_mem'
      4. Set Custom label to 'Average Memory usage'
      5. Click Advanced and add. script '{ "script" : "doc['max_mem'].value /1024" }' as JSON input
      6. Click the Metrics & Axes tab
      7. Click Average of memory usage
      8. Set Chart type to 'area'
    5. Show Average of memory request per event time along the X-axis
      1. In the Metrics pane, click Add metrics and expand X-axis
      2. Set Aggregation to 'Average'
      3. Set Aggregation to 'resource_req.mem.min'
      4. Set Custom label to 'Average of memory request'
      5. Click the Metrics & Axes tab
      6. Click Average of memory usage
      7. Set Chart type to 'area'
    6. Show Max of memory usage per event time along the Y-axis
      1. In the Metrics pane, click Add metrics and expand Y-axis
      2. Set Aggregation to 'Max'
      3. Set Field to 'Max_mem'
      4. Set Custom label to 'Max of memory usage'.
      5. Click advanced and add script:{ "script" :"doc['max_mem'].value /1024" } as JSON input
    7. Show the event time along the X-axis
      1. In the Buckets pane, click Add>X-aixs
      2. Set Aggregation to 'Date Histogram'
      3. Set Field to 'event_time', and Minimum Interval to 'Auto'
    8. Click Apply changes. visualization should look like this

    9. Save the visualization with name 'Req_mem VS Mem_used'

    Chart 3:Job run time rank

    Statistic Number of jobs is based on run time rank.

    • Add scripted field

    1. Go to Management > Kibana > Index Patterns
    2. Select the index pattern(lsb_acct* index) to add a scripted field to
    3. Go to the pattern's Scripted fields
    4. Click Add scripted field.
    5. Enter a name for the scripted field, for example 'percentiles'
    6. Enter the expression that is used to compute a value on the fly from the index script as below

    if (doc['run_time'].value <=300){ return ' 0 m to 5 m ' }else if (doc['run_time'].value <=3600){ return '5m to 1 hours' }else if (doc['run_time'].value <=14400){ return '1 hours to 4 hours' }else if (doc['run_time'].value<=86400){ return '4 hours to 1 day' }else { return 'More than one day' }This script is for calculating job run time rank.

    • Create visualization

    1. Open Visualize to show the overview page
    2. Click Create new visualization. You'll see all the visualization types in Kibana
    3. Click Vertical Bar
    4. In Choose a source, select the lsb_acct* index pattern.
    5. Show the number of job id per percentile along the Y-axis
      1. In the Metrics pane, expand Y-axis.
      2. Set Aggregation to Unique Count
      3. Set Field to
      4. Set Custom label to '# of jobs'
    6. Show the job run time rank along the X-axis
      1. In the Buckets pane, click Add>X-axis
      2. Set Aggregation to Terms
      3. Set Field to job_runtime_rank which defined before as scripted field
      4. Set order by to 'metric: # Unique count of jobs'
      5. Click Apply changes. visualization should looks like this

      6. Save the visualization with name 'Job run time rank'.

    Chart 4:Summary

    This section introduces how to generate a data table. This table will show the job memory usage, job memory request, memory usage/memory application, and the number of jobs completed aggregated by the cluster name and queue name.

    1. Click Create new visualization. All the visualization types display in Kibana.
    2. Click Data Table.
    3. In Choose a source, select the lsb_acct* index pattern.
    4. Add Metrics into Data table.
      1. In the Metrics pane, click
      2. Set Aggregation to 'Unique Count'
      3. Set Field to 'job_id'.
      4. Set Custom label to '# of jobs'.
      5. Click Add metrics and click Metric.
      6. Set Aggregation to 'Average'.
      7. Set Field to 'Max_mem'.
      8. Set Custom label to 'Average of memory usage'
      9. Click advanced and add script: { "script" :"doc['max_mem'].value /1024" } as JSON input.
      10. Click Add metrics and click Metric.
      11. Set Aggregation to 'Average'.
      12. Set Field to 'resource_req.mem.min'
      13. Set Custom label to 'Average of memory request'.
      14. Click Add metrics and click Metric.
      15. Set Field to 'percentiles'
      16. Set Custom label to 'Average of (memory usage/memory request)'
    5. Add dimensions into Data Table
      1. In Buckets pane, click Split Rows
      2. Set Aggregation to 'Terms'.
      3. Set Field 'cluster_name'
      4. Set Order by to 'Alphabetical'
      5. Set Custom label to 'Cluster Name'.
      6. Click Add sub-buckets.
      7. Click Split Rows
      8. Set Field 'queue_name'
      9. Set sub aggregation to 'Terms'
      10. Set Order by 'Alphabetical'
      11. Set Custom label to 'Queue Name'.
    6. Click Apply changes. The graph is as below

    Create dashboard

    A dashboard is a collection of visualizations that can be arranged and shared. This section will create a dashboard that contains the visualizations that have been saved during above tutorial.

    1. Open Dashboard.
    2. On the Dashboard overview page, click Create new dashboard
    3. Set the time filter to proper time
    4. Click Add in the menu bar.
    5. Add Memory outlier, Req_mem VS Mem_used,Summary.
    6. Resize Summary chart. A resize control is on the lower right.
    7. Click Add Filter
      1. Set Field to 'cluster_name'.
      2. Set Operator to 'is one of'
      3. Set Value to $yourclustername
      4. Enable Create custom label and set a name for it
      5. Click Save.
      6. Repeat step1-5 to add projct_name and queue_name as filters

        Note: By default, the Search API returns the top 10 matching documents, which is why only 10 items can be seen in the drop-down list, even though there are more. Click 'Edit as query DSL' to edit the query to make more items be listed.

    The sample dashboard may look like below



    ------------------------------
    Edward Deng
    ------------------------------
    #SpectrumComputingGroup


  • 2.  RE: [LSF Explorer]Create sample dashboard in Kibana

    Posted Wed December 30, 2020 02:05 PM

    Hi Edward,

    This visualization Req_mem VS Mem_used is missing some parameters. I am not able to follow along. I am happy to report that memory outliner went well and I am moving on to 'Job Runtime Rank' now to see if I can get that visualization created using your instructions.

    Bob



    ------------------------------
    Robert White
    ------------------------------



  • 3.  RE: [LSF Explorer]Create sample dashboard in Kibana

    Posted Tue January 05, 2021 12:21 AM
    Hi Robert,
    Thanks for your findings. I've updated the post.

    ------------------------------
    Edward Deng
    ------------------------------