In the world of high-performance computing, we often obsess over CPU utilization. When a load instruction misses the local caches and has to fetch data from a distant NUMA node or main memory, the processor pipeline stalls.

This is the Memory Wall, and to climb it, you need to know exactly which instructions are causing the delay. To solve this, I developed loadlatency.

The Problem: The High Cost of a Load

Modern POWER processors are significantly faster than the DRAM they sit next to. A single load instruction can take:

• A few cycles if the data is in the L1 cache.

• Hundreds of cycles if it has to travel to another socket.

Standard profilers tell you which functions are hot, but they don't always tell you why. Is the function slow because it has too many calculations, or because one specific load instruction is consistently missing the cache?

The Solution: Hardware-Level Insight with PMU

My script, loadlatency, doesn't guess. It uses the Performance Monitoring Unit (PMU)—dedicated hardware inside the IBM Power chip—to sample load instructions that exceed a specific latency threshold.

By tapping into hardware events (like PM_LD_MISS_L1 or sampled instructional data), the script provides a surgical look at your binary's execution.

What loadlatency Reveals:

1. The Culprit Instruction: The exact instruction address (NIP) where the stall occurred.

2. Latency Quantified: The actual cycle count the hardware spent waiting for that specific data to return.

3. Data Source Mapping: It identifies if the data was eventually found in the L2, L3, or if it suffered the ultimate penalty of a Main Memory access.

Why This Matters for IBM Power Architecture

IBM Power processors are built for massive data throughput, featuring SMT-8 threading and high-bandwidth interfaces. However, these features can actually hide memory bottlenecks in standard tools.

Using loadlatency on Power allows you to:

• Optimize Data Layouts: Rearrange structures to ensure that data being loaded together actually sits together in memory (Spatial Locality).

• Debug NUMA Affinity: Instantly see if latency is high because threads are accessing memory on a remote processor socket.

How to Use It

The workflow leverages the power of perf combined with custom parsing logic:

# 1. Capture sampled load instructions with latency data
perf record --weight -e "{cpu/event=0x672C0101EC,thresh_cmp=10/}":u  -- ./your_high_performance_app

# 2. Parse the trace using loadlatency
perf script -i <path to perf.data file> -s ./llatency-script.py

The script then processes the raw PMU samples and outputs a readable report of the instructions that are "bottlenecking" your application.

Conclusion

By identifying the specific load instructions that take the longest, you can stop guessing and start optimizing the parts of your code that actually matter.