Enterprise Linux

Authors: R Nageswara Sastry, Kajol Jain, Athira Rajeev, Anju T Sudhakar, Madhavan Srinivasan.

Introduction 

An IBM® POWER9™ processor has several features when compared to the previous generations of IBM POWER® processors. One such feature is the 24x7 performance monitor unit (PMU) that enables precise measurement of chip or socket-level resource utilization. These counter data are exposed via Linux perf/hv_24x7 interface to users. The 24×7 PMU in IBM® POWER9™ processor-based servers provide the facility to continuously collect large numbers of nest hardware performance events efficiently and accurately with a very low overload to memory. This article is intended to introduce the users to a framework to visualize IBM Power9 System socket level resource utilizations via Grafana using perf/hv_24x7 interface. Article explains the framework components and their installation/configuration steps. Framework consist of two components, Performance Co-pilot (PCP) and Grafana. 

Performance-Co-Pilot (PCP) 

Performance Co-Pilot (PCP) is a system performance analysis toolkit. PCP has performance collection daemons that are enabled to collect IBM Power9 System’s socket level resource utilization metrics. Reference this article on how to install and setup the Performance Co-Pilot packages and configure its daemons to collect Power9 Socket level performance monitoring unit (PMU) data. On the successful installation and setup, proceed to add the events needed to export via PCP.  There are two config files that are needed to be updated for PCP to export the values. One “perfevent.conf” and second is derived metric config files.  

Setup events in perfevent pmda: 

“perfevent.conf" will be used by the PCP to enable specific PMUs and its events to monitor and export the data. Here are the example memory access events which could be used to generate memory bandwidth utilisation metrics 

Steps to create derived configuration files: 

Change the directory to ‘/var/lib/pcp/config/derived’ and create required configuration file with the following content. 

After adding the above events in perfevent.conf, install the perfevent pmda (refer to  https://developer.ibm.com/articles/nest-infrastructure-powervm-perf-interface/  on steps to install pcp/pmda). After installing perfevent pmda, start the perfevent pmda daemons. 

Required services: 

Using systemctl start the services namely pmproxy, grafana-server and pmcd. Steps to start these services. 

Verification of the above: from the netstat –nap command output. 

Verify the setup: 

The above added event if configured correctly will show using the ‘pminfo’ command. 

# pminfo| grep MCS_RD_BW_CHIP_0 

MCS_RD_BW_CHIP_0 

This concludes the performance co-pilot setup needed to collect and export the Power9 Socket level performance monitoring counter data. Next will be to look at tools to visualizes  this data.

Grafana – installation and setup: 

 Install grafana-pcp rpm and start the ‘grafana-server’ using the systemctl command 

# systemctl start grafana-server 

Verification: grafana service run on port number ‘3000’, the same can be checked using ‘nestat’ command. 

# netstat -nap | grep grafana 

tcp 0 0.0.0.0:3000 0.0.0.0:* LISTEN 506923/grafana-servunix 3 [ ] STREAM CONNECTED 913571 506923/grafana-serv 

Open the grafana in web browser using the  

Url: http://ipaddress:3000 (Default credentials are admin/admin) 

To work with grafana we need to add ‘data source’ from which grafana has to read the data for display. We are going to use the Performance Co-Pilot (PCP) plug-in for this purpose. Click on Performance Co-Pilot, which is in the Right hand side lower pane. To enable PCP click on ‘Enable’ button.

With PCP enabled, Now switch to Dashboard. In the ‘Dashboard’ tab you can choose what kind of overview you want to see. Here choose ‘PCP Vector Host Overview’. 

 

After choosing needed overview, add the data source by that data will be pulled for showing the visualizations. Click on ‘Add data source’. 

 
From the list choose ‘PCP Vector’. 

Fill the details of the URL from which data has to be read. This URL contains the IP address as well as the port number of the pmproxy service i.e. 44322 and
click on ‘Save & Test’.  The following screen confirms that there is no issue with the data source and it’s working fine.

 

Steps for the visualization using PCP data source. In the PCP vector host overview create a new panel for your data and configure. Click on ‘+’ and then choose ‘Dash Board’. 

After creating the dash board and new panel will be created and user can ‘add query’ and ‘choose visualization’. 

Click on ‘Add Quey’. In the Query user can mention from what events to read the data. Here taken the event which was added to perfevent pmda. 

After adding the event, grafana service starts reading the data and default visualization will be shown as below. These visualizations can be customized 

Here is the grafana view for the 24x7 Nest counters presenting memory bandwidth data. Chart created using the above mentioned steps. 

Conclusion: 

This article provides an overview on framework to collect and visualize Power9 System level resources such as Memory Controller, SMP Links and so on via performance Co-pilot and Grafana/Vector.  


Contacting the Enterprise Linux on Power Team
Have questions for the Enterprise Linux on Power team or want to learn more? Follow our discussion group on IBM Community Discussions.