Authors: R Nageswara Sastry, Kajol Jain, Athira Rajeev, Anju T Sudhakar, Madhavan Srinivasan.
Introduction
An IBM® POWER9™ processor has several features when compared to the previous generations of IBM POWER® processors. One such feature is the 24x7 performance monitor unit (PMU) that enables precise measurement of chip or socket-level resource utilization. These counter data are exposed via Linux perf/hv_24x7 interface to users. The 24×7 PMU in IBM® POWER9™ processor-based servers provide the facility to continuously collect large numbers of nest hardware performance events efficiently and accurately with a very low overload to memory. This article is intended to introduce the users to a framework to visualize IBM Power9 System socket level resource utilizations via Grafana using perf/hv_24x7 interface. Article explains the framework components and their installation/configuration steps. Framework consist of two components, Performance Co-pilot (PCP) and Grafana.
Performance-Co-Pilot (PCP)
Performance Co-Pilot (PCP) is a system performance analysis toolkit. PCP has performance collection daemons that are enabled to collect IBM Power9 System’s socket level resource utilization metrics. Reference this article on how to install and setup the Performance Co-Pilot packages and configure its daemons to collect Power9 Socket level performance monitoring unit (PMU) data. On the successful installation and setup, proceed to add the events needed to export via PCP. There are two config files that are needed to be updated for PCP to export the values. One “perfevent.conf” and second is derived metric config files.
Setup events in perfevent pmda:
“perfevent.conf" will be used by the PCP to enable specific PMUs and its events to monitor and export the data. Here are the example memory access events which could be used to generate memory bandwidth utilisation metrics
# cd /var/lib/pcp/pmdas/perfevent/
# vi perfevent.conf
//add the following, please make sure the following are //tab seperated
hv_24x7.PM_MCS01_128B_RD_DISP_PORT01 0 chip:0
hv_24x7.PM_MCS01_128B_RD_DISP_PORT23 0 chip:0
hv_24x7.PM_MCS23_128B_RD_DISP_PORT01 0 chip:0
hv_24x7.PM_MCS23_128B_RD_DISP_PORT23 0 chip:0
Steps to create derived configuration files:
Change the directory to ‘/var/lib/pcp/config/derived’ and create required configuration file with the following content.
#cd /var/lib/pcp/config/derived
# cat hv_24x7-metric.conf
MCS_RD_BW_CHIP_0 = (((scalar(perfevent.hwcounters.hv_24x7.PM_MCS01_128B_RD_DISP_PORT01_chip_0.value)+
scalar(perfevent.hwcounters.hv_24x7.PM_MCS01_128B_RD_DISP_PORT23_chip_0.value)+
scalar(perfevent.hwcounters.hv_24x7.PM_MCS23_128B_RD_DISP_PORT01_chip_0.value)+
scalar(perfevent.hwcounters.hv_24x7.PM_MCS23_128B_RD_DISP_PORT23_chip_0.value)) * 64 * 256) / (1000*1000))
After adding the above events in perfevent.conf, install the perfevent pmda (refer to https://developer.ibm.com/articles/nest-infrastructure-powervm-perf-interface/ on steps to install pcp/pmda). After installing perfevent pmda, start the perfevent pmda daemons.
Required services:
Using systemctl start the services namely pmproxy, grafana-server and pmcd. Steps to start these services.
#systemctl start pmproxy
#systemctl start pmcd
#systemctl start grafana-server
Verification of the above: from the netstat –nap command output.
#netstat –nap
tcp 0 0 0.0.0.0:44321 0.0.0.0:* LISTEN 3273/pmcd
tcp 0 0 0.0.0.0:44322 0.0.0.0:* LISTEN 131476/pmproxy
tcp 0 0 0.0.0.0:44323 0.0.0.0:* LISTEN 131476/pmproxy
Verify the setup:
The above added event if configured correctly will show using the ‘pminfo’ command.
# pminfo| grep MCS_RD_BW_CHIP_0
MCS_RD_BW_CHIP_0
This concludes the performance co-pilot setup needed to collect and export the Power9 Socket level performance monitoring counter data. Next will be to look at tools to visualizes this data.
Grafana – installation and setup:
Install grafana-pcp rpm and start the ‘grafana-server’ using the systemctl command
# systemctl start grafana-server
Verification: grafana service run on port number ‘3000’, the same can be checked using ‘nestat’ command.
# netstat -nap | grep grafana
tcp 0 0.0.0.0:3000 0.0.0.0:* LISTEN 506923/grafana-servunix 3 [ ] STREAM CONNECTED 913571 506923/grafana-serv
Open the grafana in web browser using the
Url: http://ipaddress:3000 (Default credentials are admin/admin)
To work with grafana we need to add ‘data source’ from which grafana has to read the data for display. We are going to use the Performance Co-Pilot (PCP) plug-in for this purpose. Click on Performance Co-Pilot, which is in the Right hand side lower pane. To enable PCP click on ‘Enable’ button.