Description

This document introduces the steps of how to generate useful dashboard in Kibana, based on the data collected by LSF Explorer.

Modules of Kibana

• Management
• Visualization
• Dashboard

Kibana version:7.2.1

The data collected by LSF Explore used in visualization

Fields:

        * lsf_job_acct index - Information about done and exited LSF jobs (DONE, EXIT).

                                               Type of event is  JOB_FINISH

Chart 1：Memory outlier

This section will introduce how to generate a chart which shows the ratio of the actually used memory to the requested memory.

A scripted field named percentiles (mem used/mem request) will be created and its threshold will be set to 60%. When the job's percentiles are less than 60%, it is considered as memory outlier.

Assuming that the Kibana is connected to Elasticsearch, and lsb_acct index is used as the data source.

• Add scripted field

1. Go to Management > Kibana > Index Patterns
2. Select the index pattern(lsb_acct* index) that will be added a scripted field to
3. Go to the pattern's Scripted fields
4. Click Add scripted field
5. Enter a name for the scripted field, for example 'percentiles'
6. Enter the expression that will be used to compute a value on the fly from the index script as below

((doc['max_mem'].value==0)?0:((doc['max_mem'].value/1024)/((doc['resource_req.mem.min'].size()==0)?(doc['max_mem'].value/1024):doc['resource_req.mem.min'].value)))

This script is to get a ratio of memory usage and request memory

7. Click Create field.

• Create visualization

1. Open Visualize to show the overview page
2. Click Create new visualization. You'll see all the visualization types in Kibana
3. Click Vertical Bar
4. In Choose a source, select the lsb_acct* index pattern
5. Show the number of job id per percentile along the Y-axis
   1. In the Metrics pane, expand Y-axis
   2. Set Aggregation to Unique Count
   3. Set Field to job_id
   4. Set Custom label to '# of jobs'

      

6. Show the percentiles along the X-axis
   1. In the Buckets pane, click Add>X-axis
   2. Set Aggregation to Range
   3. Set Field to percentiles which defined before as scripted field
   4. Set 'from' and 'to' as below

      

   5. Set custom label to 'Memory used/Memory Request'
   6. In the buckets pane, click Add->Split Series
   7. Set Aggregation to Range
   8. Set Field to percentiles which defined before as scripted field
   9. Set 'from' and 'to' as below

      

   10. Click Apply changes. Then, the visualization should look like this

       

       *Red stands for outlier; Green stands for OK

   11. Save the visualization with name 'Memory outlier'.

Chart 2：Req_mem VS Mem_used

This section will introduce how to generate a chart that the actual job memory usage comparing with the memory request.

1. Click Create new visualization. You'll see all the visualization types in Kibana
2. Click Vertical Bar
3. In Choose a source, select the lsb_acct* index pattern
4. Show Average of memory usage per event time along the Y-axis
   1. In the Metrics pane, expand Y-axis
   2. Set Aggregation to
   3. Set Field to
   4. Set Custom label to 'Average Memory usage'
   5. Click Advanced and add. script '{ "script" : "doc['max_mem'].value /1024" }' as JSON input
   6. Click the Metrics & Axes tab
   7. Click Average of memory usage
   8. Set Chart type to 'area'
5. Show Average of memory request per event time along the X-axis
   1. In the Metrics pane, click Add metrics and expand X-axis
   2. Set Aggregation to 'Average'
   3. Set Aggregation to 'resource_req.mem.min'
   4. Set Custom label to 'Average of memory request'
   5. Click the Metrics & Axes tab
   6. Click Average of memory usage
   7. Set Chart type to 'area'
6. Show Max of memory usage per event time along the Y-axis
   1. In the Metrics pane, click Add metrics and expand Y-axis
   2. Set Aggregation to 'Max'
   3. Set Field to Max_mem
   4. Set Custom label to 'Max of memory usage'.
   5. Click advanced and add script:{ "script" :"doc['max_mem'].value /1024" } as JSON input
7. Show the event time along the X-axis
   1. In the Buckets pane, click Add>X-aixs
   2. Set Aggregation to Date Histogram.
   3. Set Field to event_time.
8. Click Apply changes. visualization should look like this

   

9. Save the visualization with name 'Req_mem VS Mem_used'

Chart 3：Job run time rank

Statistic Number of jobs is based on run time rank.

• Add scripted field

1. Go to Management > Kibana > Index Patterns
2. Select the index pattern(lsb_acct* index) to add a scripted field to
3. Go to the pattern's Scripted fields
4. Click Add scripted field.
5. Enter a name for the scripted field, for example 'percentiles'
6. Enter the expression that is used to compute a value on the fly from the index script as below

if (doc['run_time'].value <=300){ return ' 0 m to 5 m ' }else if (doc['run_time'].value <=3600){ return '5m to 1 hours' }else if (doc['run_time'].value <=14400){ return '1 hours to 4 hours' }else if (doc['run_time'].value<=86400){ return '4 hours to 1 day' }else { return 'More than one day' }This script is for calculating job run time rank.

• Create visualization

1. Open Visualize to show the overview page
2. Click Create new visualization. You'll see all the visualization types in Kibana
3. Click Vertical Bar
4. In Choose a source, select the lsb_acct* index pattern.
5. Show the number of job id per percentile along the Y-axis
   1. In the Metrics pane, expand Y-axis.
   2. Set Aggregation to Unique Count
   3. Set Field to
   4. Set Custom label to '# of jobs'
6. Show the job run time rank along the X-axis
   1. In the Buckets pane, click Add>X-axis
   2. Set Aggregation to Terms
   3. Set Field to job_runtime_rank which defined before as scripted field
   4. Set order by to 'metric: # Unique count of jobs'
   5. Click Apply changes. visualization should looks like this

      

   6. Save the visualization with name 'Job run time rank'.

Chart 4：Summary

This section introduces how to generate a data table. This table will show the job memory usage, job memory request, memory usage/memory application, and the number of jobs completed aggregated by the cluster name and queue name.

1. Click Create new visualization. All the visualization types display in Kibana.
2. Click Data Table.
3. In Choose a source, select the lsb_acct* index pattern.
4. Add Metrics into Data table.
   1. In the Metrics pane, click
   2. Set Aggregation to 'Unique Count'
   3. Set Field to 'job_id'.
   4. Set Custom label to '# of jobs'.
   5. Click Add metrics and click Metric.
   6. Set Aggregation to 'Average'.
   7. Set Field to 'Max_mem'.
   8. Set Custom label to 'Average of memory usage'
   9. Click advanced and add script: { "script" :"doc['max_mem'].value /1024" } as JSON input.
   10. Click Add metrics and click Metric.
   11. Set Aggregation to 'Average'.
   12. Set Field to 'resource_req.mem.min'
   13. Set Custom label to 'Average of memory request'.
   14. Click Add metrics and click Metric.
   15. Set Field to 'percentiles'
   16. Set Custom label to 'Average of (memory usage/memory request)'
5. Add dimensions into Data Table
   1. In Buckets pane, click Split Rows
   2. Set Aggregation to 'Terms'.
   3. Set Field 'cluster_name'
   4. Set Order by to 'Alphabetical'
   5. Set Custom label to 'Cluster Name'.
   6. Click Add sub-buckets.
   7. Click Split Rows
   8. Set Field 'queue_name'
   9. Set sub aggregation to 'Terms'
   10. Set Order by 'Alphabetical'
   11. Set Custom label to 'Queue Name'.
6. Click Apply changes. The graph is as below

   

Create dashboard

A dashboard is a collection of visualizations that can be arranged and shared. This section will create a dashboard that contains the visualizations that have been saved during above tutorial.

1. Open Dashboard.
2. On the Dashboard overview page, click Create new dashboard
3. Set the time filter to proper time
4. Click Add in the menu bar.
5. Add Memory outlier, Req_mem VS Mem_used,Summary.
6. Resize Summary chart. A resize control is on the lower right.
7. Click Add Filter
   1. Set Field to 'cluster_name'.
   2. Set Operator to 'is one of'
   3. Set Value to $yourclustername
   4. Enable Create custom label and set a name for it
   5. Click Save.
   6. Repeat step1-5 to add projct_name and queue_name as filters

      Note: By default, the Search API returns the top 10 matching documents, which is why only 10 items can be seen in the drop-down list, even though there are more. Click 'Edit as query DSL' to edit the query to make more items be listed.
      

The sample dashboard may look like below

Hi Edward,

This visualization Req_mem VS Mem_used is missing some parameters. I am not able to follow along. I am happy to report that memory outliner went well and I am moving on to 'Job Runtime Rank' now to see if I can get that visualization created using your instructions.

Bob

Description

This document introduces the steps of how to generate useful dashboard in Kibana, based on the data collected by LSF Explorer.

Modules of Kibana

• Management
• Visualization
• Dashboard

Kibana version:7.2.1

The data collected by LSF Explore used in visualization

Fields:

        * lsf_job_acct index - Information about done and exited LSF jobs (DONE, EXIT).

                                               Type of event is  JOB_FINISH

Chart 1：Memory outlier

This section will introduce how to generate a chart which shows the ratio of the actually used memory to the requested memory.

A scripted field named percentiles (mem used/mem request) will be created and its threshold will be set to 60%. When the job's percentiles are less than 60%, it is considered as memory outlier.

Assuming that the Kibana is connected to Elasticsearch, and lsb_acct index is used as the data source.

• Add scripted field

1. Go to Management > Kibana > Index Patterns
2. Select the index pattern(lsb_acct* index) that will be added a scripted field to
3. Go to the pattern's Scripted fields
4. Click Add scripted field
5. Enter a name for the scripted field, for example 'percentiles'
6. Enter the expression that will be used to compute a value on the fly from the index script as below

((doc['max_mem'].value==0)?0:((doc['max_mem'].value/1024)/((doc['resource_req.mem.min'].size()==0)?(doc['max_mem'].value/1024):doc['resource_req.mem.min'].value)))

This script is to get a ratio of memory usage and request memory

7. Click Create field.

• Create visualization

1. Open Visualize to show the overview page
2. Click Create new visualization. You'll see all the visualization types in Kibana
3. Click Vertical Bar
4. In Choose a source, select the lsb_acct* index pattern
5. Show the number of job id per percentile along the Y-axis
   1. In the Metrics pane, expand Y-axis
   2. Set Aggregation to Unique Count
   3. Set Field to job_id
   4. Set Custom label to '# of jobs'

      

6. Show the percentiles along the X-axis
   1. In the Buckets pane, click Add>X-axis
   2. Set Aggregation to Range
   3. Set Field to percentiles which defined before as scripted field
   4. Set 'from' and 'to' as below

      

   5. Set custom label to 'Memory used/Memory Request'
   6. In the buckets pane, click Add->Split Series
   7. Set Aggregation to Range
   8. Set Field to percentiles which defined before as scripted field
   9. Set 'from' and 'to' as below

      

   10. Click Apply changes. Then, the visualization should look like this

       

       *Red stands for outlier; Green stands for OK

   11. Save the visualization with name 'Memory outlier'.

Chart 2：Req_mem VS Mem_used

This section will introduce how to generate a chart that the actual job memory usage comparing with the memory request.

1. Click Create new visualization. You'll see all the visualization types in Kibana
2. Click Vertical Bar
3. In Choose a source, select the lsb_acct* index pattern
4. Show Average of memory usage per event time along the Y-axis
   1. In the Metrics pane, expand Y-axis
   2. Set Aggregation to
   3. Set Field to
   4. Set Custom label to 'Average Memory usage'
   5. Click Advanced and add. script '{ "script" : "doc['max_mem'].value /1024" }' as JSON input
   6. Click the Metrics & Axes tab
   7. Click Average of memory usage
   8. Set Chart type to 'area'
5. Show Average of memory request per event time along the X-axis
   1. In the Metrics pane, click Add metrics and expand X-axis
   2. Set Aggregation to 'Average'
   3. Set Aggregation to 'resource_req.mem.min'
   4. Set Custom label to 'Average of memory request'
   5. Click the Metrics & Axes tab
   6. Click Average of memory usage
   7. Set Chart type to 'area'
6. Show Max of memory usage per event time along the Y-axis
   1. In the Metrics pane, click Add metrics and expand Y-axis
   2. Set Aggregation to 'Max'
   3. Set Field to Max_mem
   4. Set Custom label to 'Max of memory usage'.
   5. Click advanced and add script:{ "script" :"doc['max_mem'].value /1024" } as JSON input
7. Show the event time along the X-axis
   1. In the Buckets pane, click Add>X-aixs
   2. Set Aggregation to Date Histogram.
   3. Set Field to event_time.
8. Click Apply changes. visualization should look like this

   

9. Save the visualization with name 'Req_mem VS Mem_used'

Chart 3：Job run time rank

Statistic Number of jobs is based on run time rank.

• Add scripted field

1. Go to Management > Kibana > Index Patterns
2. Select the index pattern(lsb_acct* index) to add a scripted field to
3. Go to the pattern's Scripted fields
4. Click Add scripted field.
5. Enter a name for the scripted field, for example 'percentiles'
6. Enter the expression that is used to compute a value on the fly from the index script as below

if (doc['run_time'].value <=300){ return ' 0 m to 5 m ' }else if (doc['run_time'].value <=3600){ return '5m to 1 hours' }else if (doc['run_time'].value <=14400){ return '1 hours to 4 hours' }else if (doc['run_time'].value<=86400){ return '4 hours to 1 day' }else { return 'More than one day' }This script is for calculating job run time rank.

• Create visualization

1. Open Visualize to show the overview page
2. Click Create new visualization. You'll see all the visualization types in Kibana
3. Click Vertical Bar
4. In Choose a source, select the lsb_acct* index pattern.
5. Show the number of job id per percentile along the Y-axis
   1. In the Metrics pane, expand Y-axis.
   2. Set Aggregation to Unique Count
   3. Set Field to
   4. Set Custom label to '# of jobs'
6. Show the job run time rank along the X-axis
   1. In the Buckets pane, click Add>X-axis
   2. Set Aggregation to Terms
   3. Set Field to job_runtime_rank which defined before as scripted field
   4. Set order by to 'metric: # Unique count of jobs'
   5. Click Apply changes. visualization should looks like this

      

   6. Save the visualization with name 'Job run time rank'.

Chart 4：Summary

This section introduces how to generate a data table. This table will show the job memory usage, job memory request, memory usage/memory application, and the number of jobs completed aggregated by the cluster name and queue name.

1. Click Create new visualization. All the visualization types display in Kibana.
2. Click Data Table.
3. In Choose a source, select the lsb_acct* index pattern.
4. Add Metrics into Data table.
   1. In the Metrics pane, click
   2. Set Aggregation to 'Unique Count'
   3. Set Field to 'job_id'.
   4. Set Custom label to '# of jobs'.
   5. Click Add metrics and click Metric.
   6. Set Aggregation to 'Average'.
   7. Set Field to 'Max_mem'.
   8. Set Custom label to 'Average of memory usage'
   9. Click advanced and add script: { "script" :"doc['max_mem'].value /1024" } as JSON input.
   10. Click Add metrics and click Metric.
   11. Set Aggregation to 'Average'.
   12. Set Field to 'resource_req.mem.min'
   13. Set Custom label to 'Average of memory request'.
   14. Click Add metrics and click Metric.
   15. Set Field to 'percentiles'
   16. Set Custom label to 'Average of (memory usage/memory request)'
5. Add dimensions into Data Table
   1. In Buckets pane, click Split Rows
   2. Set Aggregation to 'Terms'.
   3. Set Field 'cluster_name'
   4. Set Order by to 'Alphabetical'
   5. Set Custom label to 'Cluster Name'.
   6. Click Add sub-buckets.
   7. Click Split Rows
   8. Set Field 'queue_name'
   9. Set sub aggregation to 'Terms'
   10. Set Order by 'Alphabetical'
   11. Set Custom label to 'Queue Name'.
6. Click Apply changes. The graph is as below

   

Create dashboard

A dashboard is a collection of visualizations that can be arranged and shared. This section will create a dashboard that contains the visualizations that have been saved during above tutorial.

1. Open Dashboard.
2. On the Dashboard overview page, click Create new dashboard
3. Set the time filter to proper time
4. Click Add in the menu bar.
5. Add Memory outlier, Req_mem VS Mem_used,Summary.
6. Resize Summary chart. A resize control is on the lower right.
7. Click Add Filter
   1. Set Field to 'cluster_name'.
   2. Set Operator to 'is one of'
   3. Set Value to $yourclustername
   4. Enable Create custom label and set a name for it
   5. Click Save.
   6. Repeat step1-5 to add projct_name and queue_name as filters

      Note: By default, the Search API returns the top 10 matching documents, which is why only 10 items can be seen in the drop-down list, even though there are more. Click 'Edit as query DSL' to edit the query to make more items be listed.
      

The sample dashboard may look like below