Mainframe Storage

Mainframe Storage

Enhancing performance, reliability, and security ensuring the availability of critical business workloads

 View Only

Why Not Miles Per Hour ?

By Tony Pearson posted Thu July 26, 2007 07:09 PM

  

Originally posted by: TonyPearson


Perhaps I wrapped up my exploration of disk system performance one day too early. (While it is Friday here in Malaysia, it is still only Thursday back home)

Barry Burke, EMC blogger (aka The Storage Anarchist) writes:

Aren't you mixing metrics here?

Miles per Gallon measures an effeciency ratio (amount of work done with a fixed amount of energy), not a speed ratio (distance traveled in a unit of time).

Given that IOPs and MB/s are the unit of "work" a storage array does, wouldn't the MPG equivalent for storage be more like IOPs per Watt or MB/s per Watt? Or maybe just simply Megabytes Stored per Watt (a typical "green" measurement)?

You appear to be intentionally avoiding the comparison of I/Os per Second and Megabytes per Second to Miles Per Hour?

May I ask why?

This is a fair question, Barry, so I will try to address it here.

It was not a typo, I did mean MPG (miles per gallon) and not MPH (miles per hour). It is always challenging to find an analogy that everyone can relate to explain concepts in Information Technology that might be harder to grasp. I chose MPG because it was closely related to IOPS and MB/s in four ways:

  • MPG applies to all instances of a particular make and model. Before Henry Ford and the assembly line, cars were made one at a time, by a small team of craftsmen, and so there could be variety from one instance to another. Today, vehicles and storage systems are mass-produced in a manner that provides consistent quality. You can test one vehicle, and safely assume that all similar instances of the same make and model will have the similar mileage. The same is true for disk systems, test one disk system and you can assume that all others of the same make and model will have similar performance.

  • MPG has a standardized measurement benchmark that is publicly available. The US Environmental Protection Agency (EPA) is an easy analogy for the Storage Performance Council, providing the results of various offerings to chose from.

  • MPG has usage-specific benchmarks to reflect real-world conditions.The EPA offers City MPG for the type of driving you do to get to work, and Highway MPG, to reflect the type ofdriving on a cross-country trip. These serve as a direct analogy to SPC having SPC-1 for Online transaction processing (OLTP) and SPC-2 for large file transfers, database queries and video streaming.

  • MPG can be used for cost/benefit analysis.For example, one could estimate the amount of business value (miles travelled) for the amount of dollar investment (cost to purchase gallons of gasoline, at an assumed gas price). The EPA does this as part of their analysis. This is similar to the way IOPS and MB/s can be divided by the cost of the storage system being tested on SPC benchmark results. The business value of IOPS or MB/s depends on the application, but could relate to the number of transactions processed per hour, the number of music downloads per hour, or number of customer queries handled per hour, all of which can be assigned a specific dollar amount for analysis.

It seemed that if I was going to explain why standardized benchmarks were relevant, I should find an analogy that has similar features to compare to. I thought about MPH, since it is based on time units like IOPS and MB/s, butdecided against it based on an earlier comment you made, Barry, about NASCAR:

Let's imagine that a Dodge Charger wins the overwhelming majority of NASCAR races. Would that prove that a stock Charger is the best car for driving to work, or for a cross-country trip?

Your comparison, Barry, to car-racing brings up three reasons why I felt MPH is a bad metric to use for an analogy:

  • Increasing MPH, and driving anywhere near the maximum rated MPH for a vehicle, can be reckless and dangerous,risking loss of human life and property damage. Even professional race car drivers will agree there are dangers involved. By contrast, processing I/O requests at maximum speed poses no additional risk to the data, nor possibledamage to any of the IT equipment involved.

  • While most vehicles have top speeds in excess of 100 miles per hour, most Federal, State and Local speed limits prevent anyone from taking advantage of those maximums. Race-car drivers in NASCAR may be able to take advantage of maximum MPH of a vehicle, the rest of us can't. The government limits speed of vehicles precisely because of the dangers mentioned in the previous bullet. In contrast, processing I/O requests at faster speeds poses no such dangers, so the government poses no limits.

  • Neither IOPS nor MB/s match MPH exactly.Earlier this week,I related IOPS to "Questions handled per hour" at the local public library, and MB/s to "Spoken words per minute" in those replies. If I tried to find a metric based on unit type to match the "per second" in IOPS and MB/s, then I would need to find a unit that equated to "I/O requests" or "MB transferred" rather than something related to "distance travelled".

    In terms of time-based units, the closest I could come up with for IOPS was acceleration rate of zero-to-sixty MPH in a certain number of seconds. Speeding up to 60MPH, then slamming the breaks, and then back up to 60MPH, start-stop, start-stop, and so on, would reflect what IOPS is doing on a requestby request basis, but nobody drives like this (except maybe the taxi cab drivers here in Malaysia!)

    Since vehicles are limited to speed limits in normal road conditions, the closest I could come up with for MB/s would be "passenger-miles per hour", such that high-occupancy vehicles like school buses could deliver more passengers than low-occupancy vehicles with only a few passengers.

    Neither start-stops nor passenger-miles per hour have standardized benchmarks, so they don't work well for comparisonbetween vehicles.If you or anyone can come up with a metric that will help explain the relevance of standardized benchmarks better than the MPG that I already used, I would be interested in it.

You also mention, Barry, the term "efficiency" but mileage is about "fuel economy".Wikipedia is quick to point out that the fuel efficiency of petroleum engines has improved markedly in recent decades, this does not necessarily translate into fuel economy of cars. The same can be said about the performance of internal bandwidth ofthe backplane between controllers and faster HDD does not necessarily translate to external performance of the disk system as a whole. You correctly point this out in your blog about the DMX-4:

Complementing the 4Gb FC and FICON front-end support added to the DMX-3 at the end of 2006, the new 4Gb back-end allows the DMX-4 to support the latest in 4Gb FC disk drives.

You may have noticed that there weren't any specific performance claims attributed to the new 4Gb FC back-end. This wasn't an oversight, it is in fact intentional. The reality is that when it comes to massive-cache storage architectures, there really isn't that much of a difference between 2Gb/s transfer speeds and 4Gb/s.

Oh, and yes, it's true - the DMX-4 is not the first high-end storage array to ship a 4Gb/s FC back-end. The USP-V, announced way back in May, has that honor (but only if it meets the promised first shipments in July 2007). DMX-4 will be in August '07, so I guess that leaves the DS8000 a distant 3rd.

This also explains why the IBM DS8000, with its clever "Adaptive Replacement Cache" algorithm, has such highSPC-1 benchmarks despite the fact that it still uses 2Gbps drives inside. Given that it doesn't matter between2Gbps and 4Gbps on the back-end, why would it matter which vendor came first, second or third, and why call it a "distant 3rd" for IBM? How soon would IBM need to announce similar back-end support for it to be a "close 3rd" in your mind?

I'll wrap up with you're excellent comment that Watts per GB is a typical "green" metric. I strongly support the whole"green initiative" and I used "Watts per GB" last month to explain about how tape is less energy-consumptive than paper.I see on your blog you have used it yourself here:

The DMX-3 requires less Watts/GB in an apples-to-apples comparison of capacity and ports against both the USP and the DS8000, using the same exact disk drives

It is not clear if "requires less" means "slightly less" or "substantially less" in this context, and have no facts from my own folks within IBM to confirm or deny it. Given that tape is orders of magnitude less energy-consumptive than anything EMC manufacturers today, the point is probably moot.

I find it refreshing, nonetheless, to have agreed-upon "energy consumption" metrics to make such apples-to-apples comparisons between products from different storage vendors. This is exactly what customers want to do with performance as well, without necessarily having to run their own benchmarks or work with specific storage vendors. Of course, Watts/GB consumption varies by workload, so to make such comparisons truly apples-to-apples, you would need to run the same workload against both systems. Why not use the SPC-1 or SPC-2 benchmarks to measure the Watts/GB consumption? That way, EMC can publish the DMX performance numbers at the same time as the energy consumption numbers, and then HDS can follow suit for its USP-V.

I'm on my way back to the USA soon, but wanted to post this now so I can relax on the plane.

technorati tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,

5 comments
12 views

Permalink

https://community.ibm.com/community/user/blogs/tony-pearson1/2007/07/26/why-not-miles-per-hour

Comments

Archive User
Tue July 31, 2007 05:12 PM

Perhaps a cars 0-60mph time or 1/4 mile time are good analogies to IOPs. Adding repeated starts and stops isn't necessarily relevent. The 2 tests are standard benchmarks that measure a cars performance. Both are generally repeatable "best possible" results just like SPC-1 IOPS. Of course 0-60 is dependent on traction, and the 1/4 mile less so but it also adds drag as a factor.
MB/sec could be like passengers transported/sec but that isn't a standard test. Performance per $ is occasionally reported but I don't recall the units. You would expect a Ferarri to be faster than a Civic but is it worth the price difference?

Archive User
Mon July 30, 2007 08:45 PM

21.5 MPG sounds reasonably close to 24MPG. If you were deciding between a car that is 24MPG and one that is 12MPG, and you know you get only 21.5MPG with the first, you may assume you will get 10.25MPG with the second. The point is that these estimates don't have to be exact, just proportional and in the same relative order. Even if SPC benchmarks were off by 20% of what customers actually get, they would still be valuable, relevant and useful in decision making in this regard.
I did point to the SPC website which explains why SPC-1 was reasonably close to certain set of OLTP workloads, and why SPC-2 was reasonably close to a different set of video transfer, file transfer and database query workloads, and tried to use the analogy of City MPG and Highway MPG to explain why they are different from each other, and why one benchmark may be closer to your workload than the other.
For the new EPA ratings, I will most likely be comparing 2008 model cars with other 2008 model cars, so as long as these are measured in a standardized consistent manner, than the new ratings are valuable and relevant to my decision making process. Only if I was comparing new cars to used cars would it matter, and at that point, differences less than 8-12% would probably be over-ridden by other decision factors.
And yes, as for lawyers, I am sure the ones with higher LSAT scores are better than the ones with lower LSAT scores, which is why these standardized tests are factored in the recruitment decisions of nearly every law firm.
Like lawyers and cars, storage is not chosen solely on a single criteria, just that it should be factored in to the decision making process when performance-critical workloads are involved.
Some day in the future, when enough people publish the results of their real-world performance, spelling out what application workloads they are running on what vendor gear they are running it on, and everyone about to make a purchase decision could find real-world performance results that appear to match what they are investigating, for their applications, for their level of workload, for the brands of storage they are interested in buying, then perhaps we wouldn't need standardized benchmarks at all. Until that day, there's SPC.

Archive User
Mon July 30, 2007 05:53 PM

It's been fun...but you're probably the only person I've ever heard that claims to get "reasonably close" to the rated MPG.
I'll go out on a limb and opine that our respective definitions of "reasonable" probably differ quite a bit (even without debating exactly how much better "uses less power" means).
Me - I just assume that the MPG ratings don't really mean all that much, and if I get better than the rating I can be artificially happy, but if I get worse, I probably oughta lighten up on the gas pedal more often...but that's just me (FYI - no matter how hard I try, I can't get any better than 21.5MPG highway, even though my car is rated at 24MPG).
Just curious - how are you going to rationalize the new EPA ratings when you buy your next car, adjusted downward 12% lower city, 8% lower highway (which will bring the ratings closer to my experience, but they'll probably be lower than yours)?
And it's your blog, so of course you can keep making the assertion that there is some mystical relationship between SPC and actual results of OLTP, database and file transfer workloads. But so far, you've still presented no evidence of that assertion.
As far as I can tell, SPC is no better an indicator of "good storage" than the LSAT is a predictor of a "good lawyer" (insert favorite lawyer joke here :> )

Archive User
Mon July 30, 2007 05:02 PM

BarryB,I enjoyed your response! And thank you for the kind words on my masterful use of English language.
Perhaps cars were a bad source for analogy, as we seem to have merely carried over our differences of opinions from the storage world over to the car world. I have owned five cars (serially, of course, not all at the same time), and all of them got reasonably close to the mileage that was advertised for City MPG and Highway MPG. While I agree these will never achieve five-decimal-point accuracy to actual results, they were close enough to help with my purchase decisions in the past, and I plan to continue to use them for future car purchase decisions.
And that's all I expect from SPC benchmarks, to be close enough to actual results of OLTP, database, and file transfer workloads to provide helpful guidance in purchase decisions.

Archive User
Mon July 30, 2007 01:55 PM

Well, I still think you're mixing metaphors, comparing fruit flies with potato peelers. I explain my perspective over on my own blog (follow the link below).
Thanks!