IBM FlashSystem

IBM FlashSystem

Find answers and share expertise on IBM FlashSystem

 View Only

Getting Under EMC Skin

By Tony Pearson posted Sat July 21, 2007 12:43 AM

  

Originally posted by: TonyPearson


For those in the US, a comedian named Carlos Mencia has a great TV show, Mind of Menciaand one of my favorite segments is "Why the @#$% is this news!" where he goes about showingblatantly obvious things that were reported in various channels.

So, when I saw that IBM once again, for the third year in a row, has the fastest disk system,the IBM System Storage SAN Volume Controller (SVC), based on widely-accepted industry benchmarksrepresenting typical business workloads, I thought, "Do I really want to blog about this,and sound like a broken record, repeating my various statements of the past of how great SVC is?" It's like reminding people that IBM hashad the most US patents than any other company, every year, for the past 14 years.

(Last year, I received comments fromWoody Hutsell, VP of Texas Memory Systems,because I pointed out that their "World's Fastest Storage"® cache-only system, was not as fast as IBM's SVC.You can ready my opinions, and the various comments that ensued, hereand here. )

That all changed when EMC uber-blogger Chuck Hollis forgot his own Lessons in Marketingwhen heposted his rantDoes Anyone Take The SPC Seriously?That's like asking "Does anyone take book and movie reviews seriously?" Of course they do!In fact, if a movie doesn't make a big deal of its "Two thumbs up!" rating, you know it did not sitwill with the reviewers. It's even more critical for books. I guess this latest news from SPC reallygot under EMC's skin.

For medium and large size businesses, storage is expensive, and customers want to do as much research as possible ahead of time to make informed decisions. A lot of money is at stake, and often, once you choose a product, you are stuckwith that vendor for many years to come, sometimes paying software renewals after only 90 days, and hardware maintenance renewals after only a year when the warranty runs out.

Customers shopping for storage like the idea of a standardized test that is representative, so they can compare one vendor's claims with another. The Storage Performance Council (SPC), much like the Transaction Processing Performance Council (TPC-C) for servers, requires full disclosure of the test environment so people can see what was measured and make their own judgement on whether or not it reflects their workloads. Chuck pours scorn on SPC but I think we should point to TPC-C as a great success story and ask why he thinks the same can't happen for storage? Server performance is also a complicatedsubject, but people compare TPC-C and TPC-H benchmarks all the time.



Note: This blog post has been updated. I am retracting comments that were unfair generalizations. The next two paragraphs are different than originally posted.

Chuck states that "Anyone is free, however, to download the SPC code, lash it up to their CLARiiON, and have at it." I encourage every customer to do this with whatever disk systems they already have installed. Judge for yourself how each benchmark compares to your experience with your application workload, and consider publishing the results for the benefit of others, or at least send me the results, so that I can understand better all of these"use cases" that Chuck talks about so often. I agree that real-world performance measurements using real applications and real data are always going to be more accurate and more relevant to that particular customer. Unfortunately, there are little or no such results made public. They are noticeably absent. With thousands of customers running with storage from all the major storage vendors, as well as storage from smaller start-up companies, I would expect more performance comparison data to be readily available.

In my opinion, customers would benefit by seeing the performance results obtained by others. SPC benchmarks help to fill this void, to provide customers who have not yet purchased the equipment, and are looking for guidance of which vendors to work with, and which products to put into their consideration set.

Truth is, benchmarks are just one of the many ways to evaluate storage vendors and their products. There are also customer references, industry awards, and corporate statements of a company's financial health, strategy and vision.Like anything, it is information to weigh against other factors when making expensive decisions. And I am sure the SPC would be glad to hear of any suggestions for a third SPC-3 benchmark, if the first two don't provide you enough guidance.

So, if you are not delighted with the performance you are getting from your storage now, or would benefit by having even faster I/O, consider improving its performance by adding SAN Volume Controller. SVC is like salt or soy sauce, it makes everything taste better. IBM would be glad to help you with a try-and-buy or proof-of-concept approach, and even help you compare the performance, before and after, with whatever gear you have now. You might just be surprised how much better life is with SVC. And if, for some reason, the performance boost you experience for your unique workload is only 10-30% better with SVC, you are free to tell the world about your disappointment.

technorati tags: , , , , , , , ,, , , , , , , , , , ,



15 comments
5 views

Permalink

https://community.ibm.com/community/user/blogs/tony-pearson1/2007/07/21/getting-under-emc-skin

Comments

Archive User
Mon July 23, 2007 01:45 PM

Apologies accepted. And thanks for the education on IBM's blogging policy.
In the spirit of IBM blogging guideline #10, I won't revive the issue we debated by re-posting my comments elsewhere.
And I'm happy with "fellow blogger" - in fact, I've respectfully added you to my blogroll with that sentiment in mind.
Thanks! Barry

Archive User
Sat July 21, 2007 06:47 PM

<60us is hardly noticeable in systems that have average 2ms write and 6ms read response times.
True though, maybe I should find a voice of my own on this blog-sphere malarky.

Archive User
Sat July 21, 2007 06:26 PM

Thanks BarryW - I'll get over it ("Mark" is better than the other m-word I was called :*)
And I might just take you up on that visit - it does sound like you're doing some pretty neat things there. At the end of the day, though, you're still introducing latency into the I/O path - while it's impressive how much you've been able to reduce it, you know you can't get it to zero with the SVC. Theoretically, InVista *should* be able to do better, since the switch is merely redirecting the frames instead of forwarding them. But only time will tell (don't blast me - I'm a Symm dude, remember...I don't know about all this SVC/Invista stuff).
Anyway, stepping away from the whole SPC *and* the SVC/Invista debate, I certainly respect good engineering.
Hey - maybe you should have your own blog?

Archive User
Sat July 21, 2007 05:52 PM

First of all, sorry BarryB for getting you mixed up with storagezilla (Mark).
I just wanted to clarify the "optimizing the next configuration" statement - what we did for 4.2.0 and 8G4 nodes was make full use of the strides in the Intel-hardware base. Dual core CPU's, much faster FSB speeds. So we could bind single - what we call - IO threads to single cores and FC ports and huge strides in removal of spin-locks, thus doubling the IO processing power in the nodes. Interally I have benchmarked pure workloads at >100% increase over 4.1.0 8F4 nodes and mixed workloads - due to some additional patent pending cache alogorithm enhancements - >150% increase (mixed I mean 70/30 all miss)
So I'd expect customers to see even more than SPC - and this was done on a much more basic configuration of backend storage - without the need to "optimization".
Anyway if you are ever down Winchester way sounds like we could have an interesting chat over a pint ot two :)

Archive User
Sat July 21, 2007 08:57 AM

Sorry, Mr. Whyte, I didn't realize your comments were aimed at me, since you addressed me as "Mark." But Tony has deleted my comment entries where I signed my full name, so I guess you could have forgotten we have the same given name (and no - I'm not hiding behind my pseudonym: I am Barry Burke, and I work at EMC).
I will agree with you on one point - from here, we can only go in circles. I could argue that the configuration you used is intentionally optimized for the benchmark and not really representative of general-purpose SVC deployments. And then you could retort that the configuration is indeed real-world, and that some umpty-ump customers use this same configuration every day. Then I'd say "Yup, but how many of them run SPC-2 as their application?", and then....well you get the picture.
So, Barry, let's just leave it where it is for now. I'm sure we'll get to have another round in the future - I'll try to think up another analogy while you work on optimizing the next configuration (nice job, by the way).
Tony - the "minion" thing really was uncalled for. Please reread the last line of the comment of mine that you deleted.
And remember, you are doing this blog under the umbrella of the IBM Logo and the banner of your title as Brand Manager. IMHO that makes this blog just like any other collateral I find on IBM.com - despite what your disclaimer says.
I've started a poll over at my blog site to see what other people think about whether Corporate Blogs are to be considered marketing collateral or not. Feel free to cast your vote.

Archive User
Fri July 20, 2007 06:24 AM

Mark,
The fact that "suped up Charger" != "street spec Charger" is the key. SPC'd SVC == customer SVC
Robert,
I think you have hit the nail squarely on the head and got to the real "external perception" of the issue. All arguing aside, a lack of published performance information leaves everyone open to 'speculation' as to why there is none - usually thinking the worst - which I'm not stating is the case, but the skeptic in us all does rear its head in this kind of situation.
I don't think we are going to get EMC to suddenly step up to the plate and perform an SPC benchmark - not after making such a song and dance about it being 'unrepresntative' - however its good to see that there is an understanding of where we inside IBM are coming from with respect to openness and the public airing of our products performance.
I don't really see where this thread can go from here, apart from round in circles, IBM will continue to use SPC as a way to externalize our storage products performance, and EMC will continue to tell the world that its pointless. So be it.

Archive User
Fri July 20, 2007 04:41 AM

Both the pro-SPC and anti-SPC groups have valid points in regards to what SPC does and does not provide.
But to a big extent that is all irrelevant as the real issue is EMC's and to a lesser extent HDS's desire to hide the facts from the public.
What facts? Well any meaningful performance numbers what so ever about the DMX product range for example. There is basically no way for someone to sit down and see if the DMX is a suitable fit for their requirements short of having to do a full blown test themselves.
While IBM and others may be incorrect in stating that there are no legal restrictions or contractual restrictions in a customer publishing performance results it is extremely odd that I am unable to find any published results anywhere from a customer. Why would this be?
EMC wishes to hide behind the marketing excuses such as it's not representational of real world applications or other trite excuses. Basically, EMC is stating that customers are too stupid to understand benchmark results without them standing next to them to explain why 10,000 IOP/s is really not slower than 15,000 IOP/s.
I like the EMC products and I respect the technical expertise of the people involved here in this argument but the simple fact of the matter is that if the EMC products out performed everyone else they would publish the results. You couldn't stop the marketing department from doing it. Anything other than doing the SPC benchmark is nothing more than a face saving fig leaf wearing marketing program.

Archive User
Thu July 19, 2007 05:28 PM

Barry Whyte -
(My attempt to return this debate to the point at hand, in a civil manner).
Would you consider a NASCAR race a "benchmark" of sorts?
Let's imagine that a Dodge Charger wins the overwhelming majority of NASCAR races. Would that prove that a stock Charger is the best car for driving to work, or for a cross-country trip?
I think not - it would only prove that a (souped up) Dodge Charger is better most of the time in NASCAR races. And nothing more.
And I *know* it wouldn't make my wife buy one!
SPC=NASCAR raceSVC=Dodge Charger
Your argument of "relative" comparisons doesn't change that one iota - without knowing how the SPC relates to any other workload makes it meaningless. In fact, just knowing that the DS8300 is 2x a DS4800 tells me nothing about which one I need. If my workload happens to look nothing like the SPC, it could be that a single DS4800 would suffice, or that even a single DS8300 won't be enough. I have no way of knowing, unless I know a lot about performance, the SPC and the workload characteristics of my own applications.
Which most people admittedly don't.
Benchmarks can indeed identify relationships along a fixed criteria, but if you can't correlate that criteria to your own requirements, benchmarks remain meaningless.
At least, to everyone but your Marketing Department (just teasing, Tony!)
Bottom line - I know my wife is smart enough not to buy a Charger just because it wins at NASCAR. I can only hope that most IT folks who watch the SPC results are at least as smart as my wife :*>
---DISCLAIMER: My wife doesn't watch NASCAR.

Archive User
Wed July 18, 2007 11:19 PM

Hi Tony
Someone showed me your blog today, and -- I have to say -- while I respect the fact that you're waving the IBM flag, I have to say I'm disappointed.
I've come to expect better from IBM -- even their storage guys.
Why do I say this? Take a look at what you've said in public for just a moment.
You've provided incorrect hearsay regarding legal matters.
Not good.
You've made representations that SPC testing is somehow relevant to customers' environments, but offered nothing more than platitudes in support of that statement.
Not good.
You've represented that SPC represents the "majority of vendors", while EMC, HDS, NetApp, Dell and numerous others refuse to participate on any level, and IBM/HP/Sun/Fujitsu only with portions of their product line.
Not good.
You are factually incorrect in some regards to EMC, and skillfully manipulative in others. Regardless of your emotions, you owe it to yourself and IBM to conduct yourself in a factual, honest manner to the best of your ability.
Not good at all.
Tony, I try to avoid any factual mispresentation regarding IBM or its products. Opinions are clearly labelled as such. And, if I do get something factually wrong, I try to correct it as soon as possible.
I'd trust you'd do the same.

Archive User
Wed July 18, 2007 04:08 PM

So watching this as an interested IBMer close to the ground on making the improvements to the latest SVC release it seems we are getting stuck on one issue. That is :
SPC doesn't provide a customer by customer performance benchmark.
Thats not the intention, and I agree with everything thats been said regarding specific testing for a specific workload - you can't beat it.
However that does not negate the results that the SPC publish. If nothing more it gives you a pretty good feel for what a system *could* achieve under a consistent workload when compared with other such systems **running the same workload**
For example it shows that a DS8300 running the same workload is capable of more than 2x a DS4800 (in the tested configurations). No matter what you may say, that is useful to customers. Its a mark on the wall that can help people to start on the minefield of buying their next controller, cutting through vendor specific spin about their read cache hit measurements.
Nowhere on the SPC site does it claim to be a one-test fits all - its a *benchmark*; to me a storage benchmark is a consistent workload aiming to grade or scale all participants.
Wiktionary defines the act of benchmarking:
'To measure the performance of an item relative to another similar item in an impartial scientific manner'
No Iron Pyrite here!

Archive User
Wed July 18, 2007 07:12 AM

But apparently you are admitting that you have no proof that such a policy has ever existed - thank you for setting the record straight. LIke I said, it's always better if you stick to the facts.
MORE IMPORTANTLY, however, I disagree wholeheartedly that standardized benchmarks are "the next best thing" as you claim. Just as standardized testing in our public schools ignores the unique capabilities of our children, tests like TPC and SPC merely homogenize the participants to a standard this is impossible to relate to the real world.
And where standardized testing in an incentive-based social infrastructure inevitably results in teaching to the test instead of unbounded curriculae, so too does standardized benchmarking inevitably result in unrealistic configurations optimized only to best the test.
As our alternative, EMC offers our customers and prospects the opportunity to test and compare our equipment in our own labs (or in theirs) under workloads that are mutually agreed to be representative of the customers actual use model. We have literally dozens of test-bench applications that can be dynamically configured to match customer application I/O workloads, as well as comprehensive traces of actual production environments used to model new applications or combinations of workloads that a customer might be planning to implement.
Most importantly, we have the people and expertise to help our customers understand how these benchmarks will relate to their own environments - perhaps the biggest gap in both standardized benchmarking and in standardized testing is that there is nobody to take responsibility for explaining what the results mean. As far as I can tell, SPC participants merely hide behind "it's a standard test", leaving the customer to decide applicability.
Given a world where you claim customers cannot afford the investment necessary to make a truly informed decision, I am sadly resigned to the fact that standardized testing will continue. I remain hopeful, however, that this will not "dumb down" our collective marketplace.
That only IBM participates so agressively in SPC, while both EMC and Hitachi (et al) do not, gives evidence that the majority of the market is not being misled by the homogenization of standardized testing.

Archive User
Wed July 18, 2007 07:09 AM

Tony -
Not that I don't believe you, but could you please send me the names of those customers you're hiding behind?
Seriously, my research indicates that this allegation is most likely nothin more urban folklore. I personally can find no evidence of such a clause ever existing in our standard P&S agreements going back at least 6 years, except in agreements covering equipment provided as loaners, for evaluation and/or to our partners as part of a CSA arrangement. I've looked pretty durn hard, and I can't find it.
So if a customer tells you their contract prohibits publishing benchmark results of EMC gear, it is entirely possible that they recall such a clause in a prior loaner/eval agreement. Or that they don't actually own the EMC system in question. Or perhaps that they have come to beleive the urban folkloare themselves.
Or maybe, you've just been mis-informed by over-zealous account teams - I don't know.
But any way you slice it, this is bad blogketing...I doubt any editor of any credible technology publication would permit such unfounded allegations to be printed or published. And I know the IBM I used to work for wouldn't approve of this tactic.
So again, I encourage you to retract or redact the assertion, and to focus instead on the facts.

Archive User
Tue July 17, 2007 10:53 PM

Barry, Mark, when we ask customers to run IBM vs. EMC side-by-side in realistic application workloads in their own data centers, so that everyone can benefit from publishing the results, the customers often tell us that EMC terms and conditions of their production equipment prevent them from doing so. I am not a lawyer either, but why would customers lie about this if it weren't true?
If EMC has recently stopped this practice worldwide, then perhaps customers are mistakenly referring to older terms and conditions of years past, and have not read their most recent purchase contracts, than that might offer an explanation. Perhaps I missed the EMC press release that indicated this practice has changed.
If EMC customers are now "free" to run side-by-side comparisons against competitive gear, then I look forward to the rush of requests to IBM to "bring it on!"
Until then, many customers don't have the budget to buy or lease equipment from all vendors, or the skills, to run their own workload comparisons among different makes and models, and so the next best thing are industry-standard benchmarks that are open to all storage vendors to participate.

Archive User
Tue July 17, 2007 10:37 PM

EMC Blogger Mark Twomey opines on this here:http://storagezilla.typepad.com/storagezilla/2007/07/spc-fes2.html

Archive User
Mon July 16, 2007 10:34 AM

Tony -
I think you may have gone a bit too far in your blogketing on this topic...
I'm not in EMC Legal, but I've been assured that there are no such restrictions as you describe in our standard purchase and sales agreements.
Perhaps you have mistaken the limitations that probably exist in our loaner, evaluation and/or Common Support Agreements? It seems reasonable that a company might restrict appropriate use of systems that are being provided without charge or for use in customer support and troubleshooting.
I'd appreciate it if you would back up this bit of blogketing with some hard facts, instead of heresay evidence from your so-called "clients." And if you can't provide some proof of the restrictions you claim you've "heard about," then I'd encourage you to retract your assertions.
As I've said elsewhere, this whole blogketing thing works much better if you stick to the facts.