IBM Destination Z - Group home

Breaking Through CPU Roadblocks

By Destination Z posted Mon December 23, 2019 03:43 PM


As outlined in a previous article, z/OS uses the Warning Track capability of the zEnterprise EC12 (zEC12) to address the short-engine problem in which vertical low-priority processors can be denied access to CPU for extended periods of time. The problem is that the program dispatched to a vertical low processor might hold a latch or lock that higher-priority work needs to proceed. This can cause response-time goals to be missed and utilization to be suboptimal. However, the solution to a similar problem helps make z/OS the best OS for running critical application work.

A situation analogous to short engines can occur within z/OS systems—a task under z/OS can be denied CPU assess for a long time when the CPUs of the partition are so busy doing higher-priority work that the lowest-priority work doesn’t get access to them. If a job is stalled in this way while holding a lock, latch or ENQ, it can cause problems similar to those caused by short engines. That is, important work can be delayed waiting for resources that can’t be made available because the resource holder can’t get a CPU to run on.

For example, consider a discretionary batch job that accesses DB2. Being discretionary, it runs at a low priority. In this case, DB2 will be running under the job’s task at the job’s priority. If DB2 acquires one of its latches and the system suddenly becomes very busy, the job will lose its CPU and not run again until activity quiets down, which might be a long time. It cannot release the latch while it’s blocked out of access to CPU. Higher-priority transactional work can be suspended for an extended period while waiting for the low-priority job to get dispatched and free the latch. Obviously, this can cause some real response-time issues.

Simple Solution
As early as the 1970s, MVS solved this problem for the case of ENQ resources, called ENQ promotion. When contention for an ENQ resource occurs, z/OS promotes the job holding the resource to a priority high enough to get access to CPU so it can run and free up the resource so higher-priority work can get the resource and continue processing. This costs some CPU cycles and causes a short-term priority inversion, but the benefit to response time is well worth the cost. Similar processing is done for some of the z/OS system locks. The means of promotion isn’t quite the same but it achieves the same end.

Unfortunately, for resources z/OS doesn’t know about, like DB2 latches and other private resources, the promotion approach doesn’t work so well. The additional overhead of making z/OS aware of when private resources are held and released is prohibitively expensive in CPU cycles. Instead, an innovative, non-intuitive technique was devised by Bernie Pierce at IBM. (Those who have read my previous articles on interesting z/OS performance designs will recognize the name. He was certainly the z/OS development team’s most prolific source of solutions to performance problems like this.)

The basic idea is simple. The genius was recognizing that such a simple approach could do the job—and do it inexpensively. The objective, of course, is to never starve any job of CPU for a long time while it’s holding a resource, no matter how low its priority is. But, z/OS doesn’t know whether a job holds any private resources because it has no knowledge of these resources. Therefore, the approach gives a dollop of CPU access to every job that’s been stalled long enough to give it a chance to free up any resources it might hold. Of course, most of the time, a job doesn’t actually hold any resources needed by other work. Given that, this approach might appear wasteful, but in reality, it can be accomplished with just a few tenth of a percent of the total capacity of a partition. And, the capacity allocated for this process isn’t wasted. The low-priority jobs that benefit get productive work done. Although giving these jobs an instantaneous priority boost when higher-priority work is ready to run does cause momentary priority inversions, it’s so minor that there’s almost no impact to the higher-priority work’s performance.

Blocked Workload Support
The design, called Blocked Workload Support, was introduced in z/OS V1.9 and rolled back to z/OS V1.7. As is typical of z/OS, installation controls regulate when the priority boosts are given and limit the amount of capacity that can be consumed by giving boosts to stalled low-priority work. An installation can specify how long work can be starved for CPU before it will be given a bump. When it’s been stalled the specified amount of time, the job is given on the order of a half millisecond of execution time (depending on processor model) to keep it making forward progress. The original default duration was 20 seconds, but IBM has more recently recommended a value of 3 to 5 seconds. The installation can also cap the total amount of capacity used to provide the boosts. The percentage is specified in tenths of a percent with a default of 0.5 percent.

z/OS is recognized as an OS that applies resources to the most important work, as specified by the installation in a performance policy. However, this intent can be thwarted when less important work locks up resources needed by more important work. The Warning Track implementation on the zEC12 and the Blocked Workload Support in z/OS attack this problem at two different levels. They provide mechanisms to clear lower-priority work out of the way without violating z/OS priority dispatching to any great extent. Subtle solutions like this keep z/OS ahead of the pack when it comes to supporting mixed workload types with diverse performance objectives.

Bob Rogers worked on mainframe system software for 43 years at IBM before retiring as a Distinguished Engineer in 2012. He started with IBM as a computer operator in 1969. After receiving a B.A. in mathematics from Marist College two years later, he became a computer programmer at the IBM Poughkeepsie Programming Center, where he worked on the OS/360 OS. Rogers continued to work on mainframe OS development for his entire career at IBM. He contributed to the transitions to XA-370 and ESA/370, and was lead software designer for the transition to the 64-bit z/Architecture. More recently, he implemented the support for single z/OS images with more than 16 CPUs and was a lead designer of the z/OS support for the zAAP and zIIP specialty engines. He has been a popular and frequent speaker at SHARE for many years.