IBM i Global

IBM i Global

Connect, learn, share, and engage with IBM Power.

 View Only
Expand all | Collapse all

Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

  • 1.  Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Tue November 12, 2024 07:04 AM
    Edited by Mohammad Safir Tue November 12, 2024 07:06 AM

    Hi , just wanted to understand how the Mapping looks like for each NVMe Drive from  (ESR0)-EXP24U.2 NVMe SAS Storage Enclosure to an IBM i Partition on E1080 which has dedicated (EJ24)-PCIe x16 to CXP Converter Cards.

    Can we Mirror NVMe Drives from first NED to an NVME Drive from Second NED?

    Can we share single NED with multiple IBM i Partition without VIOS?.

    How do we calculate Usable Capacity on NED?

    If there is an Webex session already there on NED mapping with IBM i  partitions, please share.



    ------------------------------
    Mohammad Safir
    ------------------------------



  • 2.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Tue November 12, 2024 07:58 PM
    Edited by Satid S Tue November 12, 2024 08:06 PM

    Dear Mohammad

    >>>> Can we Mirror NVMe Drives from first NED to an NVME Drive from Second NED? <<<<

    Yes, please look at the details in my post in this past thread: https://community.ibm.com/community/user/power/discussion/ibmi-mirror-nvm     

    By the way, since NVMe is much more physically stable than HDD of the past, doing mirror would waste too much usable disk space.   And if you decide to go for RAID-5, it would be good NOT to allocate all name spaces from the same NED to an LPAR.  Try to divide name spaces to both NEDs and allocate then to the same LPAR for best performance.

    >>>> Can we share single NED with multiple IBM i Partition without VIOS?.  <<<<

    Yes, but if you do mirroring, you should allocate mirrored-pair name spaces from the 2 NEDs to each LPAR.   You can do mirroring between different NVMe from the same NED but it is a less useful way to do it.

     

    >>>> How do we calculate Usable Capacity on NED? <<<<

    I see no way for knowing exactly.  After you format the NVMe, you can see what is available. You may be able to use almost all of it depending on the size of the name space you create.  DST/SST provides you with disk space info when you create name spaces.



    ------------------------------
    Satid S
    ------------------------------



  • 3.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Thu November 14, 2024 03:12 AM

    We have a Two CEC E1080 machine with three iOS partition without VIOS.. we need three different disk capacity requirement for each iOS partition, can we prepare capacity requirement for each partition to a single NED or we should provide three NEDs for each iOS partition? what would be best practice here.



    ------------------------------
    Mohammad Safir
    ------------------------------



  • 4.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Thu November 14, 2024 08:07 PM
      |   view attached

    Dear Mohammad

    >>>>  can we prepare capacity requirement for each partition to a single NED or we should provide three NEDs for each iOS partition <<<<

    You can do it any way as there is no mandatory requirement nor restriction.   How many NED do you have?      Since you indicate there are 2 CECs, I would wish you have 2 NEDs, each of which connects to each CEC/IO drawer.  Then, for the best HW availability and performance, I would mirror NVM in one NED against the other NED and distribute all mirrored name spaces to all LPARs according to each need.   I also would deploy distributed dual HW (LAN, Tape interface card. etc) in those 2 CECs/IO drawers as well.   This is just an old idea that existed for a long time now as you can see in the presentation I attach herewith.



    ------------------------------
    Satid S
    ------------------------------



  • 5.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Sun November 17, 2024 01:57 AM

    Thanks Satid S

    Your recommendation to use Two NEDS with two CECs remains the same even if we use IBM i in Native Mode or configuration based on VIOS?



    ------------------------------
    Mohammad Safir
    ------------------------------



  • 6.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Sun November 17, 2024 08:46 PM

    >>>> Your recommendation to use Two NEDS with two CECs remains the same even if we use IBM i in Native Mode or configuration based on VIOS? <<<<

    The idea applies to both cases because this is about HW resource which has nothing specific to do with OS. 



    ------------------------------
    Satid S
    ------------------------------



  • 7.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Wed November 20, 2024 09:30 AM

    Mohammad   Fortra provides an IBM Power Capacity Planning service which not only includes the model and core analysis but storage and memory as well.   This includes internal NVMe drive in the CEC or external.   If you are interested, please let me know and I can tell you how to engage.    randy.watson@fortra.com



    ------------------------------
    Randy Watson
    President
    MPG
    Boulder CO
    30039399785
    ------------------------------



  • 8.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Tue November 12, 2024 08:01 PM

    >>>> If there is an Webex session already there on NED mapping with IBM i  partitions, please share. <<<<

    If you go to Youtube and search with "ibm i nvme", you will see 2 videos on this subject. 



    ------------------------------
    Satid S
    ------------------------------



  • 9.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Sat November 30, 2024 05:59 AM

    My response was crafted with AI assistance, tailored to provide detailed and actionable guidance for your query.

    Understanding NVMe Drive Mapping in IBM i with EXP24U.2 on E1080

    Your inquiry spans a few critical areas of using NVMe storage with IBM i partitions on an E1080 system. Let's break this down and address each question in detail:


    1. Mapping NVMe Drives from EXP24U.2 to IBM i Partitions

    Key Concepts:

    • EXP24U.2 NVMe SAS Storage Enclosure (ESR0): These enclosures connect to E1080 systems via dedicated PCIe converter cards like EJ24.
    • Mapping Drives: The NVMe drives in EXP24U.2 are directly attached to IBM i partitions when configured without VIOS.
    • Dedicated Converter Cards: The (EJ24) PCIe x16 to CXP Converter Cards ensure each storage enclosure communicates with a specific partition.

    Mapping Considerations:

    • The mapping of drives is deterministic, as each EXP24U.2 is tied to a specific PCIe converter card and associated bus.
    • In the HMC (Hardware Management Console), you'll see each drive and its associated slot, allowing you to allocate drives to specific IBM i partitions.
    • Ensure that partitions requiring direct access have sufficient resources (dedicated I/O slots and memory).

    2. Mirroring NVMe Drives Across NEDs

    • NVMe Mirroring Across NEDs: Yes, you can mirror an NVMe drive from one enclosure to another, even if the enclosures are on different NEDs (Node Enclosure Drawers).
      • This is accomplished using IBM i native RAID configurations.
      • Ensure both NEDs are visible to the IBM i partition managing the mirroring.
      • Use the STRSST (Start System Service Tools) command and navigate to disk management to set up the mirroring.

    3. Sharing a Single NED Across Multiple IBM i Partitions (Without VIOS)

    • Direct Access Without VIOS:
      • NVMe drives in an EXP24U.2 can be shared across IBM i partitions if the drives are configured in independent ASPs (iASPs).
      • The drives need to be explicitly assigned to each partition through HMC or the system console.
      • Partition ownership must be non-overlapping for logical units (LUNs); no drive can be accessed by multiple partitions simultaneously unless using an iASP or another sharing mechanism.
    • Considerations:
      • Performance may be impacted when sharing storage, especially if workloads are I/O intensive.
      • You must carefully manage the assignment of drives in the enclosure to avoid conflicts.

    4. Calculating Usable Capacity on NED

    • Usable capacity depends on:
      1. Raw Drive Capacity: Sum of all NVMe drive capacities in the NED.
      2. RAID Configuration:
        • RAID-5: Subtract the capacity of one drive for parity.
        • RAID-6: Subtract the capacity of two drives for parity.
      3. Hot Spares: If configured, subtract the capacity of hot spare drives.
    • Example Calculation:
      • A NED with 24 x 1.92TB NVMe drives in RAID-5 would have:
        • Raw Capacity: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>24</mn><mo>×</mo><mn>1.92</mn><mo>=</mo><mn>46.08</mn><mtext> </mtext><mtext>TB</mtext></mrow><annotation encoding="application/x-tex">24 \times 1.92 = 46.08 \, \text{TB}</annotation></semantics></math>
        • Usable Capacity: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>46.08</mn><mo>−</mo><mn>1.92</mn><mo>=</mo><mn>44.16</mn><mtext> </mtext><mtext>TB</mtext></mrow><annotation encoding="application/x-tex">46.08 - 1.92 = 44.16 \, \text{TB}</annotation></semantics></math>
      • Adjust for multiple RAID arrays or different drive capacities as applicable.

    5. Webex Sessions or Documentation on NED Mapping

    IBM often provides technical briefings or training sessions for specific configurations like NED mapping. You can check:

    • IBM Knowledge Center: Look for EXP24U.2 documentation.
    • IBM Support and Partner Portal: Search for training materials specific to NVMe configuration on E1080.
    • Your IBM Contact or Reseller: They can provide direct links to upcoming Webex sessions or recorded ones if available.


    ------------------------------
    Saif Ali Sabri
    ------------------------------



  • 10.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Sat November 30, 2024 06:09 AM
    Thanks Saif for the input...in the last section on how to calculate Usable Capacity on NED24, you mentioned RAID-5 and RAID-6.
    Are theses supported by native IBM i OS without using RAID controller as they are not availble on these systems anymore.

    Regards





  • 11.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Mon December 02, 2024 12:53 AM

    Dear Mohammad

    RAID-5/6 disk protection is supported by microcode in SAS Adapter, not from IBM i. I check in E1080 Sales Manual and find that EJ24 adapter that you use does not support this.  You need EJ14 instead for RAID-5/6 support.



    ------------------------------
    Satid S
    ------------------------------



  • 12.  RE: Multiple EXP24U.2 NVMe SAS Storage Enclosures on E080

    Posted Mon December 02, 2024 02:38 AM

    Hello Satid, our thread is regarding the NED connect with IBM i on E1080. clearly EJ14 does not support connectivity with NED.



    ------------------------------
    Mohammad Safir
    ------------------------------