Thank you Evelyn.
Original Message:
Sent: Wed October 23, 2024 06:57 AM
From: Evelyn Perez
Subject: FS 7300 maximum drive count / limit
Hi Thiago -
This is slightly complicated because it has changed and will slightly depend on the version of code being run. Here are the driving factors (in case you are curious)
- Large chains was initially used as a key feature in order to get capacity and performance was a slight concern a the end of the chain, but drive performance ended up being a larger factor in those cases
- As drives got bigger and faster the need for very long chains reduced and we saw this reflected in the number of expansion drawers being sold/installed across our field population
- Our NVMe capacity grew along with all the functional improvements that line up with the use of our computation storage in the control drawer
This lead to the FlashSystem 7300, 9500 and follow ons shipping with a reduced chain length for SAS expansion at GA in 8.5.0. This was the FlashSystem 7300 limit in 8.5.0:
Maximum Chains: 2
Maximum 5U92s per chain: 2
Maximum 2UXX per chain: 5
Max expansions per IO Group: 10 2U or 4 5U, no intermix.
Max drives per IO group: 24 + 4X92 = 392 drives
Unfortunately, some EU legislation then passed for Power efficiency which ignored the capacity/power usage of storage devices, which means that we had to meet a minimum drive count, which then raised the chain limits for the 7300 in later releases to the following:
Maximum Chains: 2
Maximum 5U92s per chain: 2
Maximum 2UXX per chain: 6
Max expansions per IO Group: 12 2U or 4 5U, or up to 4 5U + 1 2U intermix
Max Drives per IO Group: 24 (controller) + 4X92 + 24(larger 2U drive count) = 440 drives
What happens if you exceed this limit?
We will refuse to manage the additional enclosure and the system will log an error. Worst case, if you put too many on, this will cause the code to fail to discover the drives in the chain and it could time out or run out of login resources leading to errors/disruption.
I will speak to the Redbook team to get those corrected, however, this is described in the earlier limits pages when it changed: https://www.ibm.com/support/pages/v850x-configuration-limits-and-restrictions-ibm-flashsystem-7200-and-7300.
I hope this helps.
Evelyn
------------------------------
Evelyn Perez
IBM Senior Technical Staff Member
IBM Storage Virtualize Software Architect for SVC and FlashSystem
Original Message:
Sent: Sun October 20, 2024 08:09 PM
From: Thiago Lucas
Subject: FS 7300 maximum drive count / limit
Hello
I'm really confused with how many expansion enclosures can be connected to a single FS 7300 control enclosure SAS chain.
Since "ever", all the way down to Storwize V7000 Gen1, the maximum weight for a SAS chain has been "10". Some lower end Storwize/Flashsystem models had/have 1 SAS chain, and from mid models onwards, 2 SAS chains. And I'm not talking about FS 9500 here, that's a different beast. I confess that I rarely check this information for mid models, but all models all the way until the FS 7200 had the same SAS chain weight support, verified. And specifically for the 7300 model, I can find information as this still being true (SAS chain weight = 10):
FlashSystem and Storwize 7x00 - Planning for SAS cabling to expansion enclosures
Some will say it refers to more than one model, but look at the last updated date and Figure 1..."FlashSystem 7300".... and all the way down:
| |
|---|
| Maximum expansion enclosures in SAS chain 1 | 10 |
| Maximum expansion enclosures in SAS chain 2 | 10 |
Another example, Redbook "Performance and Best Practices Guide for IBM Storage FlashSystem and IBM SAN Volume Controller (for IBM Storage Virtualize 8.6)" from September 2024, PDF page 107 to 110:
107:
1.10.2 IBM FlashSystem 7000 Expansion Enclosure 4657 Models 12G, 24G, and 92G
....
108:
An IBM FlashSystem 7300 clustered system can contain up to four IBM FlashSystem 7300 systems and up to 3,040 drives.
110:
SAS chain limitations
When attaching expansion enclosures to the control enclosure, you are not limited by the type of the enclosure. The only limitation for each of the two SAS chain is its chain weight. Each type of enclosure has its own chain weight:
Enclosures 12G and 24G have a chain weight of 1
Enclosure 92G has a chain weight of 2.5
The maximum chain weight is 10.
But, looking around other documentations, Sales Manual, Limits and Restrictions from 8.6.0.x (the last release that had "restrictions" listed, from then on only "configurationi limits" is presented), I can find multiple occurrences of the "reduced" SAS chain weight to a maximum of 5 or 6, depending on firmware, and a maximum of 440 drives per I/O group for the FS 7300, so it seems official. It seems like a change since the FlashSystems begin using License Machine Code (LMC) model, with the FS 5045 / 5300 / 7300, In the same redbook mentioned above, PDF page 235 it shows the latest information:
A single IBM FlashSystem 7300 Model 924 control enclosure can support up to 10 IBM FlashSystem 7000 expansion enclosures with a maximum of 440 drives per system.
And from a lot of other documentations this is ratified, 440 and weight of 5 or 6 max for each SAS chain.
Firstly, kind of misleading the mentioned information found on both recent redbook and online documentation. Yes, there are several other sources of information like mentioned sales manual, Product Guide for the FS7300, a lot of occurrences with the "new" maximum. But one usually trusts the first occurrence when searching for something on the run, and in my case, first information found was in the Performance redbook that I mentioned... "SAS chain weight unchanged, good!". And information was forwarded... Wrong information as it seems. A simple google search with the "sas chain weight 7300" context most probably would lead to incorrect information.. Or.. Is it really incorrect?
So after all this, I just want to be sure and get trusted information from our gurus about two questions:
- Is it really "impossible" to reach to the weight of 10 on each of the SAS chain in the FS 7300 as of today ?
- Also I would like to understand what happens if some tries to loop lets say a 7th 4657-24G enclosure to a SAS chain that already has 6x 24G enclosures hooked (todays maximum weight of 6 would have been reached)... The new added enclosure just wont be discovered ? Or what then ?
Thanks.
------------------------------
Thiago Lucas
------------------------------