IBM i Global

 View Only
Expand all | Collapse all

What's all the fuss about NMVe's?

  • 1.  What's all the fuss about NMVe's?

    IBM Champion
    Posted Mon March 22, 2021 07:27 AM
    Hi Guys

    I thought I'd take a break from the IBM P9 unboxing videos to do a little research into NMVe's and the sort of impact they could have on a typical Scale Out POWER9 S914 server.

    The video below is a link to the presentation I have at the i-UG.co.uk Hybrid event in my home town of Wolverhampton last week
    https://www.rowtonit.com/whats-all-the-fuss-about-nvme

    I thought I would share it on here in case any of you had wondered what sort of effect this new type of storage might have on IBM i?
    Spoiler Alert, they really are faster and cheaper than HDD's ;-)

    Cheers Brad

    PS Don't worry, no servers of techies we harmed in the making or delivery of this presentation ;-)


    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------


  • 2.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Tue March 23, 2021 08:53 AM

    Let's see if I grasped your presentation (After all we are separated by a common language!) 

    NVMe are faster.

    NVMe are cheaper.

    Question: Can I replace a failed NVMe without having to power down the server or restart the Operating system?

    Observation: I was rather surprised that RAID6 beat Mirror on the read and write tests. Given RAID-6 needs to do reads and many writes to update any bit of data while mirror simply goes about its two writes. Hmmm.

    Thank you for pointing out that it's much more than JUST the disk units. The SAS RAID controllers and backplane bits make up a very large part of the cost and those expansion drawers bring both purchase and recurring costs!  While it's clearly less of an impact I would expect the NVMe configuration to require fewer motivated electrons to keep them running and thus generate fewer BTUs of heat to dissipate as well. 

    One other thing, here in the U.S. we're not all driving Porsche's we all have great big heavy duty pickup trucks with massive V8 engines to carry us and our laptops and coffee cups, so the stack of gold ingots in the bed don't slow us down too much. ;-) 

    Excellent presentation, thanks for sharing!!!



    ------------------------------
    Larry Bolhuis
    Chief i-entist
    Frankeni Technology Consulting, LLC
    Middleville MI
    6162604746
    ------------------------------



  • 3.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Tue March 23, 2021 09:14 AM
    Hi Larry

    Great to hear from you, one of the main things missing from my session was you my friend.  As for you thinking it's just American's that don't understand me, no I'd say the majority of Brit's suffer from the same problem.

    My tests are not official benchmarks, they were not done in laboratory conditions, However, they were carefully considered, based on standard IBM i functionality, repeated numerous times, with an IPL between each iteration and the results were consistent.   In short, I'm pretty confident in my findings.

    You pose the question below (which knowing you is more for our audience that yourself), but it's a bloody important question, so let me answer it here.
    Question: Can I replace a failed NVMe without having to power down the server or restart the Operating system?

    Yes, you can replace a failed U.2 NVMe without having to power down and restart the Operating System.    
    No you can not replace a failed AIC NVMe without having to power down and restart the Operating System

    As for your ability to haul gold, I'll bear that in mind if I have a Lottery Win any time soon.

    Stay Safe My Friend

    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------



  • 4.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Tue March 23, 2021 10:01 AM
    Hi All

    NVMe just keeps getting better..... I've just learnt that you can replace the AIC NVMe without shutdown of the server, it does have a number of other considerations and pre-requites but never the less both the U.2 and AIC  NVMe can be concurrently maintained.  

    So now, it is just the older M.2 NVMe which needed downtime to swap out. 

    Hope that clears this up.


    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------



  • 5.  RE: What's all the fuss about NMVe's?

    Posted Tue March 23, 2021 07:42 PM
    Steve,

    I was listening to your presentation, and I have sold several small systems with NVMe's so I know it quite well.

    But I have asked Rochester and  with no answer so far:

    1. NVME's are so fast and we can on a P10 have pretty much data, and I would like to us VIOS to have e.g 2-3 LPAR's

    2. Rochester does not recommend VIOS, but I think they at least could  test how much the performance degradtion will be, because the alternative is to still sell and install external storage, and many customers don't like that.

    Any comment to this ?



        Mobil: +47 92292811
        E-mail: frank.johansen@kvikt.no




    Fra:        





  • 6.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Tue March 23, 2021 10:04 PM

    Frank,
      
        I would say that Rochester not recommending VIOS is perhaps a bit strong. 

        Certainly an alternative for IBM i customers is to use IBM i hosting in this environment.  It removes the need to know VIOS and focuses on the strength of IBM i. For customers who do not want external storage then VIOS isn't needed either. IBM i can share storage, provide networking, and share tape drives and more recently tape libraries. 

        Perhaps Steve will run his same I/O tests on a guest partition of IBM i on his servers to see what the numbers are in that configuration!

        



    ------------------------------
    Larry Bolhuis
    Chief i-entist
    Frankeni Technology Consulting, LLC
    Middleville MI
    6162604746
    ------------------------------



  • 7.  RE: What's all the fuss about NMVe's?

    Posted Wed March 24, 2021 06:06 AM
    IBM hostingi works fine, I have sold and installed that, but VIOS is a better solution.

    If someone could test the performance degration that would be fine.



        Mobil: +47 92292811
        E-mail: frank.johansen@kvikt.no




    Fra:        





  • 8.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed March 24, 2021 09:01 AM

    Frank,

        I would like to see your reasoning for why VIOS is a better solution IN THIS CASE.   

        I have an entire presentation that pits the pros and cons of VIOS hosting with IBM i hosting. It is not meant to prove one or the other is better but to use as a scorecard to determine in any given situation which of the two fits the bill better.  Areas where VOS excels include the ability to have dual servers for redundancy, provide connectivity to external storage, support partition mobility, partition suspend/resume, and active memory expansion.  It also does not consume IBM i license cores. Only this last is an advantage in the situation you are describing. 

        On the flip side, when viewed from the seat of an IBM i specialist VIOS is not as easy to maintain. It requires manual updating of device firmware, uses a command language foreign to many IBM i admins, uses completely different logic for slicing up hosted disk for client LPARs, and as noted in a thread on a thread over on midrange.com just this week, the process to replace a failed drive in VIOS is not at all as simple as it is for IBM i. And neither is backing it up.

        I would argue that there is no one best hosting solution for Power Systems, as we in the land of consulting love to say: "It Depends!"

         - Larry 



    ------------------------------
    Larry Bolhuis
    Chief i-entist
    Frankeni Technology Consulting, LLC
    Middleville MI
    6162604746
    ------------------------------



  • 9.  RE: What's all the fuss about NMVe's?

    Posted Wed March 24, 2021 10:54 AM
    IBM has stated following:



    And I have just asked, how much will the performance degradation be ?

    We have now extremely fast storage units with NVMe's and can have large capacities, up to 30-40 TB, and this will probably be increased in the future

    So, therefore I can see that in cases where we normally install external storage with VIOS, then it would be better for the customer to use NVMe's with VIOS with a System i environment only.

    I am not a deep technical person, but my understanding of  IBM Hosting i, then the client partitions are dependent of the main partition, and with VIOS we have VIOS with reduncancy, and that owns all the hardware, and more or less can see the LPAR's  as different physical systems.

    So VIOS has been used for larger System i systems, but as mentioned with the capacities and speed of NVMe's then I think it's
    neccessary to test and evaluate VIOS with such an environment.









        Mobil: +47 92292811
        E-mail: frank.johansen@kvikt.no




    Fra:        





  • 10.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed March 24, 2021 11:14 AM
    Frank,

       One of the key things you mention here is VIOS redundancy. We can all agree that redundancy is indeed a good thing. However if you are using internal disk then redundancy is effectively off the table because VIOS must 'own' the disk in order to share it.  In theory you could use two VIOS servers each with one NVME drive and let IBM i mirror at the O/S level but I have yet to find anyone who thinks this is a good solution. For one it would mean that a failed NVME drive means that one VIOS is completely lost. Yes you could add additional hardware on which to load the boot drives for VIOS to prevent that using slots 49 and 50 for example but you've also now doubled your maintenance workload for VIOS with minimal redundancy gain there. 

      When using external storage VIOS brings much more to the table than it does with internal disk. Couple that with the advantages of remote replication, flash copy, and simple migrations and that is where the combo of VIOS and external storage shine. 

       - Larry

    ------------------------------
    Larry Bolhuis
    Chief i-entist
    Frankeni Technology Consulting, LLC
    Middleville MI
    6162604746
    ------------------------------



  • 11.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed March 24, 2021 06:25 AM
    Hi Frank

    Thanks for your post, as you know, you have asked me this question in private and I gave you a detailed answer, so you know where I stand. 

    As this is in public, let me also repeat to you here what I said in private, I am not a VIOS expert but I can not find any evidence of Rochester "Not Recommending VIOS"

    If you want my opinion based on what you have shared, it would be that NVMe will work with VIOS but it does not seem like a good fit.  VIO Servers do not benefit from high performance storage for their Base Operating System (BOS), in the same way that IBM i does.  So having NVMe giving the VIOS BOS a higher performing I/O profile does not seem in and of itself to be that beneficial. 

    However, having high performance storage being passed to client LPARs would generally be a good thing, so I can see why you would want that.  I'm would be most interested to hear about how you plan to implement redundancy and failover in the storage you give to the VIOS clients?

    Like most things in life VIOS has it's pro's and con's and it's only when you have completed your system design that you can see whether or not it is appropriate to use VIOS, NMVe or a SAN. 

    In a previous reply,  Larry has pointed out, in many IBM i environments, "i hosting i" is a simple and reliable solution that IBM i SysAdmins can understand and maintain much more easily.  I personally run IBM i hosting IBM i on NVMe and have clients who do this and also host AIX and these all run great and considerably faster when hosted on NMVe.

    You don't give us much detail of your desired system design, you mention a P10 server, VIOS and 2-3 LPARs.   I would recommend your post a new thread in the IBM community forum to get a broader input on your question.  I would suggest in the PowerVM group, but where ever you put it you would get a better response if you give details of your desired system design, including:
    • number's of VIO Servers
    • number and type of guest LPARs
    • storage capacity requirements for each LPAR
    • storage performance requirements for each LPAR
    • storage RAS (Reliably, Availability & Serviceability) requirements for each LPAR
    • any advanced VIOS functions that you would like to use e.g. LPM 
    Then not just me but many system's engineers can offer the opinions and advice you seek, may of these people will know far more about VIOS than I do. So, I'm sure that together as a community we could help you.  

    I look forward to seeing you post.


    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------



  • 12.  RE: What's all the fuss about NMVe's?

    Posted Wed March 24, 2021 02:55 PM
    Hi Steve,

    Very informative video. Thanks. Glad to learn that that concurrent maintenance is available on the AIC cards. I noticed that the AIC cards have a DWPD of five vs the U.2's that have a DWPD of three. Five sounds better than three. Is there any compelling reason to use the U.2's over the AIC cards?

    Steve

       


    ------------------------------
    Stephen Cataldo
    President
    Software Concepts Inc
    North Billerica MA
    978-584-0400 x714
    ------------------------------



  • 13.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Mon March 29, 2021 05:46 AM
    Hi Steve

    Thanks for the question, it highlights just how fast things are moving in this space!  Things changed in the Drive Write Per Day (DWPD) rating when the Gen4 Add in Cards (AiCs) were released.

    The PCIe Gen3 AiC were 5 DWPD, the Gen4 AiC (is now the same as the U.2) at 3 DWPD.
    So why go Gen4 over Gen3?  Short answer, speed, Gen4 is potentially 2.5X faster. 


    Coming back to comparing AiC and U.2 form factors as was at the  hear of your question.

    AiC's have the edge over U.2 when it comes to both price and performance.

    So, why would one ever choose U.2?

    Firstly, even though you can perform concurrent maintenance on both an U.2 and AiC form factors, in most cases it is much easier and less risky to perform this maintenance on a U.2 as you do not have to take the top of the server. Which, in most cases means you do not have to move the server to it's service position in the rack.    Put another way, you are less likely to unplug a cable  from the back by accident.

    Secondly, you can have up to 4 x U.2 NVMe's in the front of a server without using up any of the internal PCIe slots, this leaves them free for other uses such as network, SAS and Fibre Channel.

    So, summing up my personal viewpoint: 
    Whilst there is no 100% right answer as to whether you should always choose AiC or U.2 NMVe, I'm still pretty confident you should always choose NMVe over HDD.

    Hope that helps.

    Brad

    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------



  • 14.  RE: What's all the fuss about NMVe's?

    Posted Thu May 20, 2021 10:18 AM
    Hey Steve,

    I noticed a message in the IBM configurator and the sales manual indicating that the use of the NVMe U.2 drive will make the machine run at it's noise level maximum's. Since you had setup a few of these NVME machine I was wondering if you noticed any unusual noise differences to HDD's. Does it sound like a jet trying to take off. Most of my installs are tower models in office \environments.

    I would have thought that these NVMe devices would make less noise. Go figure.

    Thanks
    Steve

    ------------------------------
    Stephen Cataldo
    President
    Software Concepts Inc
    North Billerica MA
    978-584-0400 x714
    ------------------------------



  • 15.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Thu May 20, 2021 10:26 AM
    Hi Stephen

    There is a firmware patch to address this, even without this patch it's not too bad, fine in a computer room.

    But this is yet another reason to stay up to date with your POWER firmware 

    Regards, Brad

    Steve Bradshaw








  • 16.  RE: What's all the fuss about NMVe's?

    Posted Thu May 20, 2021 10:49 AM

    Hello,
    Mr. Bradshaw is correct. There was a recent update to the firmware that will reduce the fan noise on the S914 (9009-41G) platform when systems are configured with 800GB or 1.6TB U.2 NVMe devices. This is applicable to both the rack mounted and tower based platforms.


    Regards,
    Douglas


    Douglas Gibbs
    IO Product Manager, IBM Cognitive Systems
    IBM Canada Ltd.
    905-413-5334


    Steve Bradshaw via IBM Community ---05/20/2021 10:25:41 AM---Hi Stephen There is a firmware patch to address this, even without this patch it's not too bad, fine






  • 17.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed May 19, 2021 07:59 AM
    Is it a British thing to change the order of the letters in the product?  Here we call it NVMe.  You often refer to it as NMVe.  I wonder if Google searching compensates for this?

    ------------------------------
    Robert Berendt
    ------------------------------



  • 18.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed May 19, 2021 09:55 AM
    Hi Robert,

    Yes we Brittish make some interesting changes to the English language, like giving you an extra "u" in colour or favour.

    This, however,  is not a British thing, "normal" British people only ever call them NVMe (None Volatile Memory express).

    Me personally, I am Dyslexic and that combined with Auto Correct on my phone, means that I often make such mistakes.

    Thanks for asking, I'm happy to be able to correct it.

    ------------------------------
    Steve Bradshaw Friendly Techie Bloke
    ------------------------------



  • 19.  RE: What's all the fuss about NMVe's?

    IBM Champion
    Posted Wed May 19, 2021 01:05 PM
    Thank you.

    ------------------------------
    Robert Berendt
    ------------------------------



  • 20.  RE: What's all the fuss about NMVe's?

    Posted Sun June 20, 2021 10:28 AM
    The NVMe U.2 deskside is definitely louder and I missed that memo about the fan noise.  I have applied the latest firmware VL950_075_045 MH01891 which has helped lower the noise.  I used the NVMe U.2 because I can get the plant to create the namespaces ahead of time and it was enterprise rated.  As I am writing this it appears that the firmware fix is working.  I can now hear myself think...lol

    ------------------------------
    Jim
    Wannabe Curmudgeon...
    ------------------------------



  • 21.  RE: What's all the fuss about NMVe's?

    Posted Mon June 21, 2021 09:52 AM
    Hello James. You had indicated that you had applied the firmware fix and it "helped" with the fan noise. Is the noise down to comparable level as it would be with HDD's or would you say that is it still somewhat louder? I have a client with an open office and the server is located in the corner of the space. You can still hear the server but it is faint. If the replacement server with the NVMe's U.2's is louder it could pose a problem.  If anyone has any experience with the noise level on the S914's with the fix applied compared to HDD's I would appreciate it.

    ------------------------------
    Stephen Cataldo
    President
    Software Concepts Inc
    North Billerica MA
    978-584-0400 x714
    ------------------------------



  • 22.  RE: What's all the fuss about NMVe's?

    Posted Mon June 21, 2021 10:15 AM
    Yes, I think it's comparable to a system with HDDs.   I don't think you'll have an issue with noise after that fix, but golly gee, you will have it before the fix.  Hopefully you aren't putting it in during working hours, though.  This would be only with the 41G models with NVMe U.2 in the front.  I've got a decibel meter on the phone but didn't get a before and after.

    ------------------------------
    Jim
    Wannabe Curmudgeon...
    ------------------------------



  • 23.  RE: What's all the fuss about NMVe's?

    Posted Mon June 21, 2021 11:18 AM
    Edited by James Sparkman Mon June 21, 2021 12:21 PM
    Replying to myself here.   Went out to econfig cloud and configured a deskside system with NVMe U.2.  I don't see any warning messages about fan noise with the U.2 NVMe feature.  Is it somewhere else not intuitively obvious to the un-initiated?  It should be in big and bold characters.

    I did find the noise warnings in the Sales Manual, however.

    ------------------------------
    Jim
    Wannabe Curmudgeon...
    Senior Systems Engineer
    CAS Severn, Inc
    Laurel, MD
    ------------------------------