Originally posted by: SystemAdmin

Back in the early 1980s, identical jobs submitted on mainframes at different times might have varied elapsed times, but their CPU times were always very close (within say 5%), seemingly independent of the many other processes and hundreds of interactive users that were running simultaneously. These days, a single process run on a 4 processor machine with no other (user) processes running shows a much different CPU time than when the same process is run on the same machine with 1 other (user) process (both CPU intensive processes). I've seen cases where a (single threaded) process run on a dedicated 4 processor machine may have a CPU time half that reported for the same process run on the same machine, but with one to three other processes running simultaneously off and on over the time period it is running.

I don't know if other processes are competing for memory access cycles and this wait time is being charged as CPU time or what the real story might be, but it seems odd that we have to run on dedicated systems when checking CPU performance changes to applications. We see this on AIX and Linux and Solaris on various processor architectures, so it isn't limited to any one OS or processor architecture (although some seem to be more affected by workload than others). We expect elapsed times to be inconsistant, but CPU times should not be varying by 50% just because other work is running on the same system. I don't know what can be done about this, but I'm just curious if others have found a way around this issue other than to run on a dedicated system.

The other aspect of this that concerns me is that when a multi-threaded application runs, how much are the multiple threads affecting each other even when not trying to serialize access to shared resources? How much under-the-covers waiting for cache fills and memory - things that will be charged to CPU time - affect the efficiencies of the threads? It gets worse if your thread runs using hyperthreading or its equivalent on various CPU architectures. My multi-threaded application tries to use threads that keep all CPUs busy and throttles back on threads when other processes become active. It tracks CPU load and adjusts its thread use accordingly. However, I have not yet found a way to tell when a CPU is only half a CPU (via HT). I have to disable this on AIX to avoid seeing twice the number of processors that are really there.

Application developers seem to have a difficult task when trying to track and improve their application's performance characteristics because of so much system variability these days. In this sense I miss the old days of running applications on MVS on a System 370/80. Checking/monitoring/optimizing performance of applications that run on many different Unix/Linux platforms and processors seems a lot more difficult than it should be.

Any comments or suggestions?

Regards,
Brion