The remark about "same LUNI ID" is linked to hyperswap.
Briefly explained, "hyperswap" works with replication between 2 separate LUNS and hyperswap makes sure those 2 LUNS have same ID.
But with mirror copies, you only have 1 LUN with mutiple copies.
All copies have the same LUN ID.
Original Message:
Sent: Mon October 24, 2022 06:04 AM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Hello Hans,
unfortunately the weekend was too short for further tests. The system is running fine in hyperswap mode 🙂
The system passed all failure tests. Without any manual admin intervention.
I'm sure we need hyperswap. I found this IBM slide, here you see the info "same LUN ID with hyperswap":
https://docplayer.org/74947892-Vergleich-ibm-storwize-local-hyperswap-vs-ibm-svc-stretched-cluster.html
Regrads,
Thomas
------------------------------
Thomas Jung
Original Message:
Sent: Wed October 19, 2022 02:51 AM
From: Hans Populaire
Subject: Realtime/Live Mirrored Volumes
Never had to implement this, but if system behaves as you are seeing, then mirror copies in a standard cluster are not working as expected.
Maybe you should report this to IBM.
It is fine to implement hyperswap but then you should be aware of several limitations as mentioned earlier.
One example is working with flashcopies (snapshots).
Regards,
Original Message:
Sent: 10/19/2022 2:10:00 AM
From: Thomas Jung
Subject: RE: Realtime/Live Mirrored Volumes
Yes I am sure. The system will be installed in the customer racks this weekend. I will send you screenshots of our mirror volume tests.
------------------------------
Thomas Jung
Original Message:
Sent: Tue October 18, 2022 12:44 PM
From: Hans Populaire
Subject: Realtime/Live Mirrored Volumes
Are you sure about this ?
Is a "standard" config you can create a volume using Custom Tab and select "All" for accessible IOgroups :
If you then check the volume, you will see that it is "accessible" via all IOgroups meaning that host should be able to access is via any iogroup
(this is similar to what you see with hyperswap volume).
With this I would expect that you can access volume all the time.
------------------------------
Hans Populaire
Original Message:
Sent: Tue October 18, 2022 07:49 AM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
In my test environment unfortunately no. I must set the copy-volume manual online.
Of course, you can script it. But now VMWare show a new Datastore with a new ID.
This datastore (volume) have the same content, but it is a new volume.
Therefore, have to use hyperswap.
------------------------------
Thomas Jung
Original Message:
Sent: Mon October 17, 2022 01:29 PM
From: Hans Populaire
Subject: Realtime/Live Mirrored Volumes
I think that setting access iogroup to "all" instead of the default (default value is "caching iogroup") should work without hyperswap (and you will not be hit by the limitations introduced by hyperswap)
Original Message:
Sent: 10/17/2022 11:37:00 AM
From: Thomas Jung
Subject: RE: Realtime/Live Mirrored Volumes
Finale Rückmeldung:
Es funktioniert mit Hyperswap 😊
Das Volume ist nun auf beiden IO Gruppen zugänglich:
Das Enclosure1 (mit Node1+2) als auch das Enclosure2 (mit Node 3+4) kann problemlos heruntergefahren oder im laufenden Betrieb getrennt werden.
Die aktiv arbeitende VM auf dem Volume bekommt davon nichts mit. Auch die Reconnects nach dem booten der Nodes funktioniert schnell und zuverlässig (Round Robin).
Leider gibt es immer noch die blöde Meldung in der IBM GUI, dass das Volume ohne Enclosure-2 nicht mehr verfügbar sein soll. Das ist aber nachweislich falsch:
Ich hoffe das hier IBM später man nachbessert.
Mit je 6x 1,92TB SAS SSD (Raid6) erreichen die beiden 5035 übrigens über 60.000 IOPS:
32 Outstanding I/Os; Default Access (100% Random, 67% Read, 2K), IBM LTS 8.5.0.5
------------------------------------------------------------------------------------------------------------------
Final feedback:
It works with Hyperswap 😊
The volume is now accessible on both IO groups:
Enclosure1 (with Node1+2) as well as Enclosure2 (with Node 3+4) can be powered down or hot disconnected. A working VM on the volume does not stop.
Unfortunately there is still the stupid message in the IBM GUI:
But this is wrong.
I hope that IBM will correct this later.
The system with 6x 1.92TB SAS SSD (Raid6) in each enclousre, reached over 60,000 IOPS:
32 outstanding I/Os; Default Access (100% Random, 67% Read, 2K), IBM LTS 8.5.0.5
------------------------------
Thomas Jung
Original Message:
Sent: Fri October 14, 2022 12:22 PM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Hallo Patrik,
vielen vielen Dank für die tolle Erklärung !!! 😊
Dann ist es klar, HyperSwap muss für unseren HA-Betrieb genutzt werden. Den hier soll niemand manuell eingreifen.
Ist der Performance Unterschied zwischen CDM 8.5.1.x signifikant zum LTS 8.5.0.5? Da so ein SAN-System in der Regel nicht so häufig gewartet wird (FW-Updates), denke ich LTS könnte das bessere für uns sein.
Ich melde mich hier mit den Ergebnissen nach FW-Update und der HyperSwap Config.
PS: Woher war meine Muttersprache ersichtlich? 😊
------------------------------
Thomas Jung
Original Message:
Sent: Fri October 14, 2022 03:03 AM
From: Patrik Groß
Subject: Realtime/Live Mirrored Volumes
Hallo Thomas,
Ja, der HyperSwap-Modus ist für hohe Verfügbarkeit notwendig. Andernfalls müssen Sie die Volumes klassisch spiegeln oder replizieren und dann im Falle eines Ausfalls manuell umschalten.
Der Unterschied zwischen einem HyperSwap-Volume und einem gespiegelten Volume (vDisk Mirror) besteht darin, dass ein Hyperswap-Volume über zwei IO-Gruppen mit unterschiedlichen Site-IDs schreibt und mit geringer Latenz repliziert, während ein vDisk Mirror innerhalb einer IO-Gruppe in zwei verschiedene Pools oder Child-Pools schreibt.
Für einen SAN-Volume-Controller könnte dies auch für einen HyperSwap-Cluster nur dann hochverfügbar sein, wenn er den Speicher selbst virtualisiert.
D.h. HyperSwap aus ist gleichbedeutend mit nicht hochverfügbar.
Wichtig sind auch diese Angaben hier: Configuration Limits and Restrictions for IBM FlashSystem 50x5.
Diese gibt es für jedes Firmware Release und sollte unbedingt beachtet werden.
Dort stehen auch einige wichtige Informationen wie z.B.: VVols nicht mit Hyperswap
https://www.ibm.com/support/pages/node/6557236
Grundsätzlich kann man sagen. das in den funktionalen Updates (CDM) der Hyperswap Cluster performanter läuft als im LongTerm Support-Release. Hier muss man für sich selber abwägen ob LTS notwendig ist.
Falls hier weitere Fragen sind oder Beratungsbedarf notwendig ist, schreib mich kurz an.
------------------------------------------------------------------------------------------------------------
Hello Thomas,
yes, HyperSwap mode is necessary for high availability. Otherwise, you need to mirror or replicate the volumes classically and then switch manually in case of failure.
The difference between a HyperSwap volume and a mirrored volume (vDisk Mirror) is that a Hyperswap volume writes across two IO groups with different site IDs and replicates with low latency, while a vDisk Mirror writes to two different pools or child pools within one IO group.
For a SAN volume controller, this could also be highly available for a HyperSwap cluster only if it virtualizes the storage itself.
I.e. HyperSwap off is equivalent to not highly available.
VVols is not supported with Hyperswap, important is this information here: Configuration Limits and Restrictions for IBM FlashSystem 50x5.
This is available for every firmware release and should be respected.
https://www.ibm.com/support/pages/node/6557236
Basically you can say that in the functional updates (CDM) the Hyperswap Cluster runs more performant than in the LongTerm Support-Release. Here one must weigh for itself whether LTS is necessary.
If you have further questions or need advice, please contact me.
------------------------------
Patrik Groß
Original Message:
Sent: Fri October 14, 2022 12:18 AM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
"I think that is a generic message as the system is not aware the volumes are mirrored to a pool hosted on a separate system."
It is a pity, Okay thank you!
But tried and tested there is no access to a mirrored volume when the IO Enlosure is shutdown.
So I think we have to use the hyperswap modus to create a high availability mirrored volume.
A voume with automatically change the io group. Is this right ?
------------------------------
Thomas Jung
Original Message:
Sent: Thu October 13, 2022 04:33 PM
From: IBM-SAN Admin
Subject: Realtime/Live Mirrored Volumes
I think that is a generic message as the system is not aware the volumes are mirrored to a pool hosted on a separate system.
------------------------------
IBM-SAN Admin
Original Message:
Sent: Thu October 06, 2022 03:23 AM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Yes I wrote the same:
While shutdown one enclosure the GUI Warning message is:
"Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 2 volumes."
But the Host is able to access both volumes ?!? Maybe the GUI Warning is wrong ?
And my question is:
Does anybody know why the GUI shows the warning "…the host will not be able to access 2 volumes."
In my opinion, the warning should not appear. (And with captcha code even more dramatic)
Thank you.
------------------------------
Thomas Jung
Original Message:
Sent: Wed October 05, 2022 02:08 PM
From: T Masteen
Subject: Realtime/Live Mirrored Volumes
Your screenshots do show Hyperswap volumes and and after shutting down enclosure 1 the volume on site 2 is still online.
So if the host does have access to both IO-groups (enclosures), then the host can still access the volume through the other enclosure.
On youtube there is a demo of Hyperswap. Although this is about an SVC, the idea remains the same.
https://www.youtube.com/watch?v=jruYJloo5qI
------------------------------
T Masteen
Original Message:
Sent: Wed October 05, 2022 12:30 PM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Hello Masteen,
no we have 2 enclosures with each 2 nodes. Site1=enclosure1, Site2=enclosure2, Site3=quorum
The volume should work on both enclosures (4 nodes). So if Site1 is down (enclosure 1) the host(s) should have access to this volume (enclosure 2).
In my screenshot I shutdown enclosure1. The volumes are online in enclosure2, alias site2, alias Bunker:
------------------------------
Thomas Jung
Original Message:
Sent: Wed October 05, 2022 10:28 AM
From: T Masteen
Subject: Realtime/Live Mirrored Volumes
Thomas,
According to the GUI you are going to shutdown a controller (5035) completely. All volumes defined under this controller (enclosure) are no longer accessible to the Hos (as stated in the messsage).
This is different when only 1 node (out of 2) of the controller goes down (during an upgrade for example). Then the volumes can still be accessed via the other node in that controller.
I hope this is a bit clear.
------------------------------
T Masteen
Original Message:
Sent: Wed October 05, 2022 04:31 AM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Thank you. We have only 1 enclosure in 2 different rooms. Both connected with 16gbit FC. If I understand you correctly, I didn't need a hyperswap. I will change the system topology.
Does anybody know why the GUI shows the waring "…the host will not be able to access 2 volumes."
In my first test, the ESX Host lost the volume. After configuring a VVOL Host, the host can access the volume with ony one enclosure. I think this was a path reconnect issue.
Sorry for my many newbies questions 😊
------------------------------
Thomas Jung
Original Message:
Sent: Tue October 04, 2022 12:30 PM
From: Hans Populaire
Subject: Realtime/Live Mirrored Volumes
In a nutshell, "hyperswap" will add a "Site" attribute to each pool and each host.
The method of syncing both volume copies relies on metro/global mirror principles with replication relationships/sessions between both volume copies.
Internally some "change volumes" are created for this also.
Hyperswap is used if you have the 2 enclosures installed in 2 different datacenters.
For normal mirror, this is not the case.
In the end there is a different way how a host will do reads :
With hyperswap, the "preferred" volume copy used for reads will be the one where Site for "pool" and "host" will match, regardless if 1 is in 1 iogroup and the other one is in another iogroup.
This because if enclosures are in 2 different datacenters, you want to ensure "read" operations will read from local enclosure only.
Without hyperswap, all volume copies are threated with same priority and reads will be distributed across both IOgroups.
------------------------------
Hans Populaire
Original Message:
Sent: Tue October 04, 2022 12:11 PM
From: Thomas Jung
Subject: Realtime/Live Mirrored Volumes
Thank you ! Yes you are right a 4-node cluster or 2 enclosure cluster 😊
And yes I think the context of a "striped" volume is/was not clear.
For me is a striped volume on 2 or more disks. If I lose one disk the volume is lost:
But now I understand it is a mirrored volume, each "volume copy" will be striped in its respective pool.
By the way what is the difference between Hyperswap and Mirrored Volume ?
....
The confusion starts with the GUI message:
After my new knowledge from the red book, I tested the system again and create VVOL hosts.
While shutdown one enclosure the GUI message is the same:
"Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 2 volumes."
But now the Host is able to access both volumes. Maybe the GUI Warning is wrong ?
------------------------------
Thomas Jung
Original Message:
Sent: Tue October 04, 2022 11:13 AM
From: Hans Populaire
Subject: Realtime/Live Mirrored Volumes
Hello,
"Striped" is an attribute for the "volume copy".
So if you have mirrored volume, each "volume copy" will be striped in its respective pool.
But each volume copy relies on 1 pool.
Important to know is that both volume copies probably will sit in a different IOGroup.
So in that case you need to ensure that host definition and SAN zonings are OK so that host has access to both IOGroups.
------------------------------
Hans Populaire
Original Message:
Sent: Mon October 03, 2022 05:52 AM
From: Carsten Klein
Subject: Realtime/Live Mirrored Volumes
Hello,
we got 2 new 5035 FC Storages. We done the initial setup on the first one and add the second to the first. Now we have a two-node cluster.
We create a Pool0 with all SSDs in System1 (Raid6) and a Pool1 with all SSDs in System2.
We download and start the IP Quorum App successful (connect).
Our goal is to create a realtime/live mirrored voulmes.
I create a mirrored Volume. Copy0 to Pool0 and Copy2 to Pool1.
In the details of this volume I see the volume is striped ?!? So if I power off enclosure-1 I got the waring "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 1 volume."
May be I misunderstand mirrored Volumes ?!?
Now we add the Remote Copy license and modify the system Topology to HyperSwap. Now we got 3 sites. 1=enclosure1, 2=enclosure2, 3=quorum
I create a HyperSwap Volume in Pool0 and Pool1. In the details of the volume, I found again "Virtualization type: Striped".
And same behavior, If I shutdown enclosure1 I got the message "Powering off the configuration node temporarily logs you out of the management GUI and the host will not be able to access 1 volume."
If I shutdown I enclosure2 I got the warning "Powering off the control enclosure will impact system performance." And the volume is online.
But we want the volume to remain accessible on both enclosure.
Please help !
------------------------------
Carsten Klein
------------------------------
#Storage
#PrimaryStorage
#StorageAreaNetworks