Hi,

We are building a test system for internal training purpose. Plan is to build 10 IBM i LPAR on S922 with 1TB disk size, no SAN, only vSCSI and dual VIOS. Any idea what should be the best option to size Power System. We have sized RAM/CPU, 2 10G NICs, no HBAs, as we donot need tape and backup.
I see two options.
1. Have four 3.2TB NVMEs installed in CEC, we get 12.8TB size. Assign those NVMEs to VIOS and create virtual disk from those NVME and assign to IBM i LPAR. Personally I have not done this method. I did option 2 below. Has anyone did that and can share more details ?
2. Have 4 SSD for VIOS in CEC and add disk drawer for 10TB size and attach via dual SAS adapter, one to VIOS1 and other to VIOS2. Create virtual disks and assign to IBM I LPAR.

Now a days standard is to use SAN via External Storage. We donot need SAN infrastructure.

Anyone can share best method for this.

Regards
Angela

------------------------------
Angela B
------------------------------

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom
------------------------------

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B
------------------------------

Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom
------------------------------

Dear Angela

From this blog post (look at the Best Practices section), I see you can use only one VIOS if you buy only 2 adapter for NVMe : NVMe device support in Virtual IO Server (VIOS)

Since you stick with 3TB NVMe unit, I guess you want to use NVMe from backplane only but I have no idea on using NVMe from backplane. I suggest you ask your question in PowerVM group for a better chance to get an answer.

As for sharing LAN card in S922, you can use SR-IOV rather than VIOS but you need to order LAN cards that support SR-IOV.  This helps reduce workload in VIOS which means less memory needed there.  I used to configure SR-IOV in HMC and it was quite simple. The downsize was it did not support link aggregation (this may change by now, need to check more) but I see this should not be a big deal for your training system. If you still want LAN availability in an LPAR, you can use the old IBM i manual config of virtual IP address. 



------------------------------
Satid Singkorapoom
------------------------------

Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B
------------------------------


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom
------------------------------

Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B
------------------------------


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom
------------------------------


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B
------------------------------

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B
------------------------------

Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom
------------------------------


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Original Message:
Sent: 5/9/2022 9:53:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom
------------------------------


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B

Thank You Brad.

One Question - Do NVMe support hardware level encryption ? I don't see latest information, but some blogs say, it does not support hardware encryption for IBMi.

Kind Regards
Angela B.

------------------------------
Angela B
------------------------------

Original Message:
Sent: Mon May 09, 2022 12:21 PM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Hi Angela

If you would like to know more about i hosting i, Larry and I recorded a webinar on IBM i hosting IBM i versus VIOS Hosting IBM i, we covered a lot of different topics, some technical some not.  I hope that this may be of use to you.  

https://youtu.be/2liJ5cDIn8M

Cheers Brad

Original Message:
Sent: 5/9/2022 9:53:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Original Message:
Sent: 5/10/2022 5:17:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Thank You Brad.

One Question - Do NVMe support hardware level encryption ? I don't see latest information, but some blogs say, it does not support hardware encryption for IBMi.

Kind Regards
Angela B.

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 12:21 PM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Hi Angela

If you would like to know more about i hosting i, Larry and I recorded a webinar on IBM i hosting IBM i versus VIOS Hosting IBM i, we covered a lot of different topics, some technical some not.  I hope that this may be of use to you.  

https://youtu.be/2liJ5cDIn8M

Cheers Brad

Original Message:
Sent: 5/9/2022 9:53:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B

Hi Brad,

Thank You.

Regards,
Angela B.

------------------------------
Angela B
------------------------------

Original Message:
Sent: Tue May 10, 2022 10:13 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Hi Angela

My understanding is that:

1)  HW encryption is not available on NVMe for Internal Storage with IBM i today but that an IBM Idea (Formerly known as an RFE) had been created so please feel free to lend your support to that.

2) You can use Software Encryption on Secondary ASPs that are supported on NVMe for Internal Storage with IBM i today

3) You can also use Hardware Encryption on NVMe on External storage (Flash System SAN)

Regards, Brad

Original Message:
Sent: 5/10/2022 5:17:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Thank You Brad.

One Question - Do NVMe support hardware level encryption ? I don't see latest information, but some blogs say, it does not support hardware encryption for IBMi.

Kind Regards
Angela B.

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 12:21 PM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Hi Angela

If you would like to know more about i hosting i, Larry and I recorded a webinar on IBM i hosting IBM i versus VIOS Hosting IBM i, we covered a lot of different topics, some technical some not.  I hope that this may be of use to you.  

https://youtu.be/2liJ5cDIn8M

Cheers Brad

Original Message:
Sent: 5/9/2022 9:53:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B


Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Dear Angela

>>>>  One Question - Do NVMe support hardware level encryption ? <<<<

As far as I remember, IBM i itself never has any HW-level internal disk encryption since its time of AS/400.  The current ones available are ASP and tape Encryption options which are SW-level and they have degraded disk and tape performance when in use.  The HW-level encryption I know of is in many tape devices and in DS8000 Enterprise SAN Disk, all of which work with IBM i with barely noticeable performance degradation.

------------------------------
Satid Singkorapoom
------------------------------

Original Message:
Sent: Tue May 10, 2022 05:16 AM
From: Angela B
Subject: Power System for IBM i using NVME

Thank You Brad.

One Question - Do NVMe support hardware level encryption ? I don't see latest information, but some blogs say, it does not support hardware encryption for IBMi.

Kind Regards
Angela B.

------------------------------
Angela B
------------------------------


Original Message:
Sent: Mon May 09, 2022 12:21 PM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Hi Angela

If you would like to know more about i hosting i, Larry and I recorded a webinar on IBM i hosting IBM i versus VIOS Hosting IBM i, we covered a lot of different topics, some technical some not.  I hope that this may be of use to you.  

https://youtu.be/2liJ5cDIn8M

Cheers Brad

Original Message:
Sent: 5/9/2022 9:53:00 AM
From: Angela B
Subject: RE: Power System for IBM i using NVME

Hi Steve,

Thank you for your inputs. I am checking i hosting on i solution.

Best Regards
Angela. B

------------------------------
Angela B


Original Message:
Sent: Mon May 09, 2022 07:46 AM
From: Steve Bradshaw
Subject: Power System for IBM i using NVME

Greetings one and all,

If you have not already ordered this system, it looks to me like you might benefit from using IBM i hosting IBM i rather than VIOS hosting IBM i. 

Either way when it comes to sizing your IBM i clients virtual disks, there are a number of factors to consider but the main one is to make sure you have a larger number of smaller virtual disks rather than a smaller number of larger disks.  I would suggest you start with the amount of storage you need for any one of these 10 LPARs then divide it through so that you have a minimum of 5 virtual disks, preferably more.  So, if for example one of your LPARs needed 500 GB of storage you could present 10 x 50GB virtual disks  but if they wanted 1TB of storage you might present 10 x 100GB disks.

Regards, Brad

Steve Bradshaw
Rowton IT Solutions Ltd

Unit 1, Amptronic Estate, Heath Mill Road, Wombourne, South Staffs, WV5 8AP.  
Web www.RowtonIT.com    Support Rowton.IT

Mobile +44 (0) 7970 978409

Office +44 (0) 1902 745409

Original Message:
Sent: 5/8/2022 11:37:00 PM
From: Satid Singkorapoom
Subject: RE: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Hi Satid,

Thanks for letting me know about single controller in back-plane for NVMe EJ1W. Your high level idea is what I thought of implementing.

With Regards
Angela.B

------------------------------
Angela B
------------------------------

Original Message:
Sent: Sun May 08, 2022 11:37 PM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

I did some more research and found that when using NVMe from S922 storage backplane, this does not support split backplane. My high level idea is that you mount all 4 NVMe units onto the storage backplane and allocate 2 units to VIOS1 and the remaining 2 units to VIOS2.  Then within each VIOS, you create vSCSI disk units and I suggest you create each vDisk units at around 100GB size to ensure sufficient number to be allocated to all IBM i LPARs.  Then allocate vDisk units equally from VIOS1 and VIOS2 to each IBM i LPAR. Then within each IBM i LPAR, you start disk mirroring making sure you mirror vDisk units from VIOS1 against those from VIOS2.   

You also need to use VIOS vSCSI LVM client only (and I have no idea what this means but it is specified here : https://www.ibm.com/support/pages/ibm-i-io-support-details  )

Please keep in mind that, since split-backplane is not supported for NVMe, all NVMe units are under a single controller, If the controller fails, all disk units are gone.  The mirroring in IBM i LPAR is therefore at disk-unit-level only.   From my past experience, the base controller rarely fails.   

Please also check this idea with someone who has experience with setting up VIOS to confirm its feasibility.


------------------------------
Satid Singkorapoom
------------------------------


Original Message:
Sent: Fri May 06, 2022 12:59 PM
From: Angela B
Subject: Power System for IBM i using NVME

Hi Satid,

Thanks for your reply. Training workloads are moderate on both counts. So if I do disk mirroring for 2 (3.2TB) and another 2 (3.2TB)disks, I will get 6.4TB after mirroring. Can I install VIOS1 on same two disk(6.4TB) and create virtual disks from same VIOS1 for IBM i LPARs. Same for VIOS2, assigning 6.4TB (both 3.2 TB disks), install VIOS2 and create virtual disks for IBM i LPARs? Thanks for namespace guidance and CPU uncapping information.

With Regards
Angela


------------------------------
Angela B


Original Message:
Sent: Fri May 06, 2022 08:25 AM
From: Satid Singkorapoom
Subject: Power System for IBM i using NVME

Dear Angela

Are the training workloads in those 10 IBM i LPARs CPU-bound or disk IO-bound or just moderate on both counts?   If CPU-bound or moderate, I would go for whichever config that costs less.  But if both configs cost in a comparable range of investment, going for all-NVMe is better as you can hope to enjoy decent disk IO performance. 

I notice you do not want to use disk mirroring for your first config.  Is this intentional?  If you actually want disk mirroring (RAID-5 not supported for NVMe), you need to buy 6.4TB units.   Please also configure NVMe namespace of 188GB rather than 393GB as IBM i likes more disk units (the configured namespace become disk unit size for IBM i).

BTW, Do the 10 LPARs carry workloads at the same time consistently? If not, please use "Uncapped Partitioning" feature so that LPARs that are carrying workloads can steal CPU power from neighboring LPARs that are idle or use little CPU power.  

Just my 2 cents.

------------------------------
Satid Singkorapoom

Hi Angela.

Have you considered having IBM i hosting the 10 IBM i LPAR's? This would allow the NVMe's to be owned and managed by the hosting IBM i partition - and no VIOS will be involved. And only IBM i knowledge is required to configure this.

Have a look at this YouTube video for more information about the different hosting options and why you may want to have IBM i hosting IBM i:
Selecting you Power System hosting environment

IBM i hosting IBM i works very well in my experience...

Best regards,
Christian

------------------------------
Christian Jorgensen
IT System Administrator
Network of Music Partners A/S
------------------------------

Dear Christian

In the past 3 years, I investigated 2 cases of IBM i hosting IBM i with a result of consistently non-optimal disk response time for all client LPARs.  But the disk response time of the host LPAR is consistently good at all time (it did not run any user workload).  Both cases were from S914 machine with all SSD units attached to the base dual-controller FC#EJ1D with 1.8GB write cache.

I did a Google search for any best practice for such an implementation but found none.  Do you, by any chance, have such best practice guideline for this? 

Here is PDI chart of SSD response time in the host LPAR (never exceed 0.6 msec  which is very good):

Here is PDI chart of vdisk response time in the client LPAR (vary between 2-26 msec  which is very disappointing):

All PDI wait charts showed small amount of any wait components. So, I see that the bad vdisk response time is the only issue here.

From my observation in trying to identify the cause of this issue, I noticed only that too few vdisk units were created for client LPARs (as IBM i like sufficient number if disk units). But I wonder if there is any other configuration guideline but could not find any yet.

------------------------------
Satid Singkorapoom
------------------------------

Original Message:
Sent: Mon May 09, 2022 03:04 AM
From: Christian Jorgensen
Subject: Power System for IBM i using NVME

Hi Angela.

Have you considered having IBM i hosting the 10 IBM i LPAR's? This would allow the NVMe's to be owned and managed by the hosting IBM i partition - and no VIOS will be involved. And only IBM i knowledge is required to configure this.

Have a look at this YouTube video for more information about the different hosting options and why you may want to have IBM i hosting IBM i:
Selecting you Power System hosting environment

IBM i hosting IBM i works very well in my experience...

Best regards,
Christian

------------------------------
Christian Jorgensen
IT System Administrator
Network of Music Partners A/S
------------------------------

Hi Christian,

Thank you. Sure I will look at this. This looks interesting.

Best Regards
Angela. B

------------------------------
Angela B
------------------------------

Original Message:
Sent: Mon May 09, 2022 03:04 AM
From: Christian Jorgensen
Subject: Power System for IBM i using NVME

Hi Angela.

Have you considered having IBM i hosting the 10 IBM i LPAR's? This would allow the NVMe's to be owned and managed by the hosting IBM i partition - and no VIOS will be involved. And only IBM i knowledge is required to configure this.

Have a look at this YouTube video for more information about the different hosting options and why you may want to have IBM i hosting IBM i:
Selecting you Power System hosting environment

IBM i hosting IBM i works very well in my experience...

Best regards,
Christian

------------------------------
Christian Jorgensen
IT System Administrator
Network of Music Partners A/S
------------------------------