Primary Storage

Expand all | Collapse all

Realtime/Live Mirrored Volumes

  • 1.  Realtime/Live Mirrored Volumes

    Posted Mon October 03, 2022 09:16 AM

    Hello,

    we got 2 new 5035 FC Storages. We done the initial setup on the first one and add the second to the first. Now we have a two-node cluster.

    We create a Pool0 with all SSDs in System1 (Raid6) and a Pool1 with all SSDs in System2.

    We download and start the IP Quorum App successful (connect).

    Our goal is to create a realtime/live mirrored voulmes.

     

    I create a mirrored Volume. Copy0 to Pool0 and Copy2 to Pool1.

    In the details of this volume I see the volume is striped ?!? So if I power off enclosure-1 I got the waring "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 1 volume."

     

    May be I misunderstand mirrored Volumes ?!?

     

    Now we add the Remote Copy license and modify the system Topology to HyperSwap. Now we got 3 sites. 1=enclosure1, 2=enclosure2, 3=quorum

    I create a HyperSwap Volume in Pool0 and Pool1. In the details of the volume, I found again "Virtualization type: Striped".

    And same behavior, If I shutdown enclosure1 I got the message "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 1 volume."

    If I shutdown I enclosure2 I got the warning "Powering off the control enclosure will impact system performance." And the volume is online.

     

     

    But we want the volume to remain accessible on both enclosure.

    Please help !



    ------------------------------
    Carsten Klein
    ------------------------------

    #StorageAreaNetworks
    #PrimaryStorage
    #Storage


  • 2.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 04, 2022 03:10 AM

    Thomas,

    First: If you do have two FS5035 and you add them together by creating a cluster, you have a 4-node cluster (each FS5035 has two nodes).

    Maybe also the context of a "striped" volume is not clear.
    A striped volume contains a volume copy that has extents that are allocated from multiple
    MDisks from the storage pool. By default, extents are allocated from all MDisks in the
    storage pool by using a round-robin algorithm.
    So it is independent of whether the volume is mirrored or not.

    The above is also described in Redbook Implementing the IBM FlashSystem with IBM Spectrum Virtualize V8.4
    https://www.redbooks.ibm.com/redbooks/pdfs/sg248492.pdf Chapter 6.2.1

    This same redbook also has a good explanation about volume mirrors. Chapter 6.2.5.

    Hopefully this, and the rest of the redbook content, is a good start to get some more clarity

    ------------------------------
    T Masteen
    ------------------------------



  • 3.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 04, 2022 11:13 AM
    Hello,

    "Striped" is an attribute for the "volume copy".
    So if you have mirrored volume, each "volume copy" will be striped in its respective pool.
    But each volume copy relies on 1 pool.

    Important to know is that both volume copies probably will sit in a different IOGroup.
    So in that case you need to ensure that host definition and SAN zonings are OK so that host has access to both IOGroups.

     
     




    ------------------------------
    Hans Populaire
    ------------------------------



  • 4.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 04, 2022 12:11 PM

    Thank you ! Yes you are right a 4-node cluster or 2 enclosure cluster 😊

     
    And yes I think the context of a "striped" volume is/was not clear.

    For me is a striped volume on 2 or more disks. If I lose one disk the volume is lost:

    But now I understand it is a mirrored volume, each "volume copy" will be striped in its respective pool.


    By the way what is the difference between Hyperswap and Mirrored Volume ?

     

    .... 

    The confusion starts with the GUI message:

     

    After my new knowledge from the red book, I tested the system again and create VVOL hosts. 

    While shutdown one enclosure the GUI message is the same:
    "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 2 volumes." 

    But now the Host is able to access both volumes. Maybe the GUI Warning is wrong ?

     

     



    ------------------------------
    Thomas Jung
    ------------------------------



  • 5.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 04, 2022 12:30 PM
    In a nutshell, "hyperswap" will add a "Site" attribute to each pool and each host.
    The method of syncing both volume copies relies on metro/global mirror principles with replication relationships/sessions between both volume copies.
    Internally some "change volumes" are created for this also.
    Hyperswap is used if you have the 2 enclosures installed in 2 different datacenters.

    For normal mirror, this is not the case.

    In the end there is a different way how a host will do reads :

    With hyperswap, the "preferred" volume copy used for reads will be the one where Site for "pool" and "host" will match, regardless if 1 is in 1 iogroup and the other one is in another iogroup.
    This because if enclosures are in 2 different datacenters, you want to ensure "read" operations will read from local enclosure only.


    Without hyperswap, all volume copies are threated with same priority and reads will be distributed across both IOgroups.






    ------------------------------
    Hans Populaire
    ------------------------------



  • 6.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 05, 2022 04:31 AM

    Thank you. We have only 1 enclosure in 2 different rooms. Both connected with 16gbit FC. If I understand you correctly, I didn't need a hyperswap. I will change the system topology. 

    Does anybody know why the GUI shows the waring "…the host will not be able to access 2 volumes." 

    In my first test, the ESX Host lost the volume. After configuring a VVOL Host, the host can access the volume with ony one enclosure. I think this was a path reconnect issue.

    Sorry for my many newbies questions 😊

     

     



    ------------------------------
    Thomas Jung
    ------------------------------



  • 7.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 05, 2022 10:28 AM
    Thomas,

    According to the GUI you are going to shutdown a controller (5035) completely. All volumes defined under this controller (enclosure) are no longer accessible to the Hos (as stated in the messsage).
    This is different when only 1 node (out of 2) of the controller goes down (during an upgrade for example). Then the volumes can still be accessed via the other node in that controller.

    I hope this is a bit clear.



    ------------------------------
    T Masteen
    ------------------------------



  • 8.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 05, 2022 12:31 PM

    Hello Masteen,

    no we have 2 enclosures with each 2 nodes. Site1=enclosure1, Site2=enclosure2, Site3=quorum

    The volume should work on both enclosures (4 nodes). So if Site1 is down (enclosure 1) the host(s) should have access to this volume (enclosure 2).

    In my screenshot I shutdown enclosure1. The volumes are online in enclosure2, alias site2, alias Bunker:









     

     



    ------------------------------
    Thomas Jung
    ------------------------------



  • 9.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 05, 2022 02:08 PM

    Your screenshots do show Hyperswap volumes and and after shutting down enclosure 1 the volume on site 2 is still online.
    So if the host does have access to both IO-groups (enclosures), then the host can still access the volume through the other enclosure.

    On youtube there is a demo of Hyperswap. Although this is about an SVC, the idea remains the same.
    https://www.youtube.com/watch?v=jruYJloo5qI

    ------------------------------
    T Masteen
    ------------------------------



  • 10.  RE: Realtime/Live Mirrored Volumes

    Posted Thu October 06, 2022 03:23 AM

    Yes I wrote the same:

    While shutdown one enclosure the GUI Warning message is:
    "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 2 volumes." 

    But the Host is able to access both volumes ?!? Maybe the GUI Warning is wrong ?


    And my question is:

    Does anybody know why the GUI shows the warning "…the host will not be able to access 2 volumes." 


    In my opinion, the warning should not appear. (And with captcha code even more dramatic)

    Thank you.



    ------------------------------
    Thomas Jung
    ------------------------------



  • 11.  RE: Realtime/Live Mirrored Volumes

    Posted Thu October 13, 2022 12:41 PM

    Hello,

    we have new information, but the issue is not solved.

    As recommended, we disabled hyperswap for the (small) one site environment.

    If we now shutdown enclosure-1, the mirrored Volume goes offline. Although a pool in the second enclousure is online.


    Pool1 Online:

     

    Are the IO groups the issue ???

    When I create a volume, I put one copy in pool0 and a second copy in pool1:


    The I/O Group is set to "automatic":


    After volume creation I see in the properties for this volume:

    Caching: io_grp0, Accessible: io_grp0

    Since enclosure1 contains io_grp0, I assume the volume is offline. Logical ?

     
    Cross-checking:
    If I create a volume manualy on io_grp1 is works on enclosure-2:


    A high availability mirrored volume should worked on both io groups.



    Please help. Thank you !!



    ------------------------------
    Thomas Jung
    ------------------------------



  • 12.  RE: Realtime/Live Mirrored Volumes

    Posted Thu October 13, 2022 04:34 PM
    I think that is a generic message as the system is not aware the volumes are mirrored to a pool hosted on a separate system.

    ------------------------------
    IBM-SAN Admin
    ------------------------------



  • 13.  RE: Realtime/Live Mirrored Volumes

    Posted Fri October 14, 2022 12:18 AM

    "I think that is a generic message as the system is not aware the volumes are mirrored to a pool hosted on a separate system."

    It is a pity, Okay thank you!

     

    But tried and tested there is no access to a mirrored volume when the IO Enlosure is shutdown.

    So I think we have to use the hyperswap modus to create a high availability mirrored volume.

    A voume with automatically change the io group. Is this right ?



    ------------------------------
    Thomas Jung
    ------------------------------



  • 14.  RE: Realtime/Live Mirrored Volumes

    Posted Fri October 14, 2022 04:50 AM
    If host can access both IOgroups, and SAN zoning is also done for both IOgroups, you can try to create volume using "Custom" tab.
    There you can set "Accessible IOgroups" to "All" for a volume.

    I think then you should be OK, but what could be is that you will have no caching when the enclosure that corresponds with "caching IOgroup" is down.

    I also think that youc an  to use CLI to update  "accessible IOgroups" for existing volumes.


    ------------------------------
    Hans Populaire
    ------------------------------



  • 15.  RE: Realtime/Live Mirrored Volumes

    IBM Champion
    Posted Fri October 14, 2022 08:57 AM
    Hallo Thomas,

    Ja, der HyperSwap-Modus ist für hohe Verfügbarkeit notwendig. Andernfalls müssen Sie die Volumes klassisch spiegeln oder replizieren und dann im Falle eines Ausfalls manuell umschalten.

    Der Unterschied zwischen einem HyperSwap-Volume und einem gespiegelten Volume (vDisk Mirror) besteht darin, dass ein Hyperswap-Volume über zwei IO-Gruppen mit unterschiedlichen Site-IDs schreibt und mit geringer Latenz repliziert, während ein vDisk Mirror innerhalb einer IO-Gruppe in zwei verschiedene Pools oder Child-Pools schreibt.
    Für einen SAN-Volume-Controller könnte dies auch für einen HyperSwap-Cluster nur dann hochverfügbar sein, wenn er den Speicher selbst virtualisiert.

    D.h. HyperSwap aus ist gleichbedeutend mit nicht hochverfügbar.

    Wichtig sind auch diese Angaben hier: Configuration Limits and Restrictions for IBM FlashSystem 50x5.
    Diese gibt es für jedes Firmware Release und sollte unbedingt beachtet werden.
    Dort stehen auch einige wichtige Informationen wie z.B.: VVols nicht mit Hyperswap
    https://www.ibm.com/support/pages/node/6557236

    Grundsätzlich kann man sagen. das in den funktionalen Updates (CDM) der Hyperswap Cluster performanter läuft als im LongTerm Support-Release. Hier muss man für sich selber abwägen ob LTS notwendig ist.

    Falls hier weitere Fragen sind oder Beratungsbedarf notwendig ist, schreib mich kurz an.

    ------------------------------------------------------------------------------------------------------------
    Hello Thomas,

    yes, HyperSwap mode is necessary for high availability. Otherwise, you need to mirror or replicate the volumes classically and then switch manually in case of failure.

    The difference between a HyperSwap volume and a mirrored volume (vDisk Mirror) is that a Hyperswap volume writes across two IO groups with different site IDs and replicates with low latency, while a vDisk Mirror writes to two different pools or child pools within one IO group.
    For a SAN volume controller, this could also be highly available for a HyperSwap cluster only if it virtualizes the storage itself.

    I.e. HyperSwap off is equivalent to not highly available.

    VVols is not supported with Hyperswap, important is this information here: Configuration Limits and Restrictions for IBM FlashSystem 50x5.
    This is available for every firmware release and should be respected.
    https://www.ibm.com/support/pages/node/6557236

    Basically you can say that in the functional updates (CDM) the Hyperswap Cluster runs more performant than in the LongTerm Support-Release. Here one must weigh for itself whether LTS is necessary.

    If you have further questions or need advice, please contact me.

    ------------------------------
    Patrik Groß
    ------------------------------



  • 16.  RE: Realtime/Live Mirrored Volumes

    Posted Fri October 14, 2022 12:23 PM

    Hallo Patrik,

    vielen vielen Dank für die tolle Erklärung !!! 😊

    Dann ist es klar, HyperSwap muss für unseren HA-Betrieb genutzt werden. Den hier soll niemand manuell eingreifen.

    Ist der Performance Unterschied zwischen CDM 8.5.1.x signifikant zum LTS 8.5.0.5? Da so ein SAN-System in der Regel nicht so häufig gewartet wird (FW-Updates), denke ich LTS könnte das bessere für uns sein.

    Ich melde mich hier mit den Ergebnissen nach FW-Update und der HyperSwap Config.

     

    PS: Woher war meine Muttersprache ersichtlich? 😊



    ------------------------------
    Thomas Jung
    ------------------------------



  • 17.  RE: Realtime/Live Mirrored Volumes

    Posted Mon October 17, 2022 11:37 AM

    Finale Rückmeldung:

    Es funktioniert mit Hyperswap 😊

    Das Volume ist nun auf beiden IO Gruppen zugänglich:


    Das Enclosure1 (mit Node1+2) als auch das Enclosure2 (mit Node 3+4) kann problemlos heruntergefahren oder im laufenden Betrieb getrennt werden.

    Die aktiv arbeitende VM auf dem Volume bekommt davon nichts mit. Auch die Reconnects nach dem booten der Nodes funktioniert schnell und zuverlässig (Round Robin).

    Leider gibt es immer noch die blöde Meldung in der IBM GUI, dass das Volume ohne Enclosure-2 nicht mehr verfügbar sein soll. Das ist aber nachweislich falsch:


    Ich hoffe das hier IBM später man nachbessert.

    Mit je 6x 1,92TB SAS SSD (Raid6) erreichen die beiden 5035 übrigens über 60.000 IOPS:

    32 Outstanding I/Os; Default Access (100% Random, 67% Read, 2K), IBM LTS 8.5.0.5





    ------------------------------------------------------------------------------------------------------------------



    Final feedback:

    It works with Hyperswap 😊

    The volume is now accessible on both IO groups:



    Enclosure1 (with Node1+2) as well as Enclosure2 (with Node 3+4) can be powered down or hot disconnected. A working VM on the volume does not stop.

    Unfortunately there is still the stupid message in the IBM GUI:
    But this is wrong.


    I hope that IBM will correct this later.

    The system with 6x 1.92TB SAS SSD (Raid6) in each enclousre, reached over 60,000 IOPS:

    32 outstanding I/Os; Default Access (100% Random, 67% Read, 2K), IBM LTS 8.5.0.5



    ------------------------------
    Thomas Jung
    ------------------------------



  • 18.  RE: Realtime/Live Mirrored Volumes

    Posted Mon October 17, 2022 01:30 PM
    I think that setting access iogroup to "all" instead of the default (default value is "caching iogroup") should work without hyperswap (and you will not be hit by the limitations introduced by hyperswap)

    Sent from my iPhone





  • 19.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 18, 2022 07:49 AM

    In my test environment unfortunately no. I must set the copy-volume manual online.

    Of course, you can script it. But now VMWare show a new Datastore with a new ID.

    This datastore (volume) have the same content, but it is a new volume.

    Therefore, have to use hyperswap.



    ------------------------------
    Thomas Jung
    ------------------------------



  • 20.  RE: Realtime/Live Mirrored Volumes

    Posted Tue October 18, 2022 12:45 PM
    Are you sure about this ?
    Is a "standard" config you can create a volume using Custom Tab and select "All" for accessible IOgroups :


    If you then check the volume, you will see that it is "accessible" via all IOgroups meaning that host should be able to access is via any iogroup 
    (this is similar to what you see with hyperswap volume).



    With this I would expect that you can access volume all the time.





    ------------------------------
    Hans Populaire
    ------------------------------



  • 21.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 19, 2022 02:10 AM
    Yes I am sure. The system will be installed in the customer racks this weekend. I will send you screenshots of our mirror volume tests.

    ------------------------------
    Thomas Jung
    ------------------------------



  • 22.  RE: Realtime/Live Mirrored Volumes

    Posted Wed October 19, 2022 02:52 AM

    Never had to implement this, but if system behaves as you are seeing, then mirror copies in a standard cluster are not working as expected.

    Maybe you should report this to IBM.

     

    It is fine to implement hyperswap but then you should be aware of several limitations as mentioned earlier.

    One example is working with flashcopies (snapshots).

     

    Regards,

     

     

     

     

     






  • 23.  RE: Realtime/Live Mirrored Volumes

    Posted Mon October 24, 2022 06:05 AM

    Hello Hans,

    unfortunately the weekend was too short for further tests. The system is running fine in hyperswap mode 🙂

    The system passed all failure tests. Without any manual admin intervention.

    I'm sure we need hyperswap. I found this IBM slide, here you see the info "same LUN ID with hyperswap":

    https://docplayer.org/74947892-Vergleich-ibm-storwize-local-hyperswap-vs-ibm-svc-stretched-cluster.html

    Regrads,
    Thomas



    ------------------------------
    Thomas Jung
    ------------------------------



  • 24.  RE: Realtime/Live Mirrored Volumes

    Posted Mon October 24, 2022 06:17 AM
    The remark about "same LUNI ID" is linked to hyperswap.
    Briefly explained, "hyperswap" works with replication between 2 separate LUNS and hyperswap makes sure those 2 LUNS have same ID.

    But with mirror copies, you only have 1 LUN with mutiple copies.
    All copies have the same LUN ID.



    Hans

    ------------------------------
    Hans Populaire
    ------------------------------