If you’ve ever managed high-utilization workloads on IBM Power Systems, you know that the Linux scheduler isn't just a traffic cop—it’s the conductor of a high-performance orchestra. On architectures like POWER9 or POWER10, where we deal with SMT-8 (eight threads per core) and complex L3 cache topologies, where and when a task runs determines the difference between millisecond responsiveness and costly latency.

However, the scheduler is often a "black box." To peek inside, we usually turn to perf, but the raw output of perf script can be an unreadable mountain of text.

To solve this, I developed sched-strategy: a specialized parser designed to transform raw tracepoints into a clear picture of scheduling behavior.

The Challenge: Why "Raw" Data Isn't Enough

The Linux kernel provides incredible instrumentation via tracepoints like sched:sched_switch. By running a simple command, you can capture every time a task is put on a CPU or taken off:

perf record -e sched:sched_switch -a sleep 5
perf script -i <path to perf.data> -s ./sched-strategy-script.py

The problem? A 5-second trace on a heavily loaded Power server can generate hundreds of thousands of lines. Manually calculating wake-up latency (the time between a task being ready and actually hitting a CPU) or identifying CPU migration patterns from this text is nearly impossible.

Introducing sched-strategy

My script acts as the "intelligence layer" for perf script. It parses the event stream to calculate critical performance metrics that help us understand the strategy the scheduler is employing.

Key Insights Provided:

• Task Latency: Exactly how long a task spent in the "runnable" state before the scheduler granted it CPU time.

• CPU Affinity & Residency: Tracks if tasks are staying on the same core (keeping caches "hot") or jumping across NUMA nodes.

• Switch Frequency: Identifies "noisy" tasks that are context-switching too often, which can degrade performance on Power's massive multithreading pipelines.

Why This Matters for IBM Power

IBM Power architecture thrives on high memory bandwidth and massive thread density. However, this hardware power is only fully realized if the scheduler makes smart decisions.

Wrapping Up

Understanding the scheduler is the final frontier of Linux performance tuning. By turning raw perf logs into structured strategy data, we can move away from "guessing" and start optimizing based on facts.