By Shrirang S Bhagwat and Abhishek Jaiswal
IBM India Software Development Labs
Overview
The storage industry is seeing a heavily inclined trend towards the Ethernet environment. Driving factors for such trend is technology advancements over the span of years, which enabled lower total cost of ownership (TCO) in the Ethernet environment. Also, cloud-based solutions require flexibility in environment for which Ethernet is a perfect candidate.
Ethernet data centers are the first choice for enterprise and cloud service providers. This is because Ethernet-only solutions can provide scalability and flexibility along with performance which is comparable with Fibre channel. At the same time, cost-effective TCO makes it more desirable. High-availability site-level protection is a must for enterprise-level businesses.
IBM HyperSwap is a dual-site active-active solution providing continuous availability of storage data in case of outages, be it planned or unplanned. The surviving site always provides access to data during outages.
Introduction
IBM® Spectrum Virtualize has introduced Ethernet-based IBM HyperSwap® in release 8.2.1, which is an active-active high availability solution. It supports short distance as well as long distance configurations where both sites can be interconnected over various layer 2 or layer 3 networks. Few examples are, Dense Wavelength Division Multiplexing (DWDM), Coarse Wavelength Division Multiplexing (CWDM), Virtual eXtensible Lan (VXLAN) and so on.
Ethernet-based IBM HyperSwap solution offers to work with long-distance converged inter-switch links (ISL) shared between different types of traffic.
Ethernet based IBM Hyperswap allows customer to operate at lower TCO because:
- Ethernet-based IBM HyperSwap can be achieved without any specialized equipment. For example, Fibre Channel over IP (FCIP) routers.
- It reduces the number of leased lines (ISLs) between two sites.
- It reduces the management hassle of separate ISLs for different traffic classes.
Abstract
A customer deployment case study is described where the customer had Fibre Channel (FC) based long distance HyperSwap solution in place with two 1 Gbps ISLs. Due to the anticipated increase in workload, customer originally wanted to upgrade their system with converged long distance two 10 Gbps ISLs. IBM pitched in with the Ethernet-based HyperSwap that uses two 10 Gbps converged ISLs to satisfy the requirement, that eliminated the need to upgrade existing 1Gbps FCIP equipment to handle higher bandwidths.
Figure 1: Existing FC-based HyperSwap setup
Figure 2: New Ethernet-based HyperSwap solution
Background
Referenced customer had long-distance dual-site IBM HyperSwap with two 1 Gbps ISL in the FC environment. The existing HyperSwap solution included FCIP routers supporting 1 Gbps link speed for long-distance ISLs.
Customer’s problem statement
- Upgrade to FC-based high bandwidth HyperSwap
Anticipating workload increase, customer wished to upgrade to storage controllers that supports better performance over high-speed networks, for example 10 Gbps, 16 Gbps, 25 Gbps, 32 Gbps, and so on. In order to cope with such performance, the existing two 1 Gbps ISL also needed to be upgraded with 10 Gbps capacity ISLs. Achieving 10 Gbps long distance ISL in a FC environment required the customer to upgrade to new FCIP routers, that support 10 Gbps speed, for each site which costs significantly higher and was a showstopper to performance upgrades.
- Replacing existing FCIP 1 Gbps long-distance ISLs with Ethernet 10 Gbps long distance ISLs
ISL deployment between data centres usually needs a convergence of different types of traffic over the ISL as it is easier to manage a smaller number of ISLs than separate ISLs for separate type of traffic between two data centres. Thus, customers needed us to support converged ISL where at least LAN traffic, host-attach traffic, and storage (node-to-node) traffic could flow on the same ISL without affecting HyperSwap functionality.
For the Ethernet-based HyperSwap function, it is of utmost importance that storage traffic is given an end-to-end guaranteed minimal bandwidth and ensure a lossless network under all circumstances over converged ISLs.
It is also cost effective to purchase lesser number of ISLs from network providers.
Solution
Deployment of long-distance Ethernet-based HyperSwap solution with iWARP technology over converged ISL
The solution included upgrading the existing IBM appliance for example, [IBM System Storage® SAN Volume Controller (SVC)] to higher bandwidth capable appliances (for example, IBM Storwize® V7000 Gen 3) and replacing FCIP-based two 1 Gbps ISL with Ethernet two 10 Gbps long-distance ISL. For a distance of approximately 10 km, customer has purchased two 10 Gbps Ethernet ISLs with round-trip time (RTT) less than 1 millisecond and 0% packet loss.
The solution uses IP quorums as a tie breaker during a split-brain situation in the HyperSwap cluster configuration. Being Ethernet-only data centres, IP Quorum fits well in the environment.
Figure 3: Customer’s production environment
Challenges
Customer environment did not have independent 10 Gbps ISLs for different classes of traffic. Rather they have two 10 Gbps ISLs where host traffic, storage traffic, and LAN (such as VMware VMotion and so on) traffic is converging. This increases the threat of network congestion on ISLs, because any kind of traffic running through ISLs might congest ISL switch ports and adversely affect all types of traffic flowing over the same link.
There are two separate issues here:
- There are different types of traffic converging into same ISL. Different traffic types would need different Quality of Service (QOS).
- It is possible for one class of traffic to congest the entire link and result in packet drops for all other traffic flowing over the ISL.
Approach and Solution
IBM Spectrum Virtualize allows storage administrator to configure Class of Service (COS) for three different classes of traffic, namely storage, host-attach, and backend controller. In this case, host-attach and storage traffic were converging over a single 10 Gbps ISL. Marking traffic with a COS value would allow network administrators to apply QoS for host-attach and storage traffic independently. It would also enable network administrator to provide a minimum guaranteed bandwidth to storage traffic over the ISL.
To handle the congestion scenario, priority flow control (PFC) is configured. Traffic causing congestion can be identified using QoS and can be paused by sending priority pause frames to targeted traffic originator. PFC also allows storage class as NODROP so that switches will do their best to avoid any drops in traffic marked with a NODROP class.
Recommendations
Ethernet-based HyperSwap is an IBM Solution for Compliance in a Regulated Environment (SCORE), which means that in order to deploy such solutions, you must contact IBM support and request for SCORE. Contact your local IBM representatives for the same. (See section: References)
There are couple of recommendations for a configuration where ISL is shared between various types of data traffic.
- DCBX enabled L3 capable Ethernet switches are recommended.
- Storage traffic must be tagged with a class of service (COS) value ranging from 1-7
- QoS settings must ensure that storage traffic is given a minimal guaranteed bandwidth and attributed to the NODROP Minimum guaranteed bandwidth must be derived from the peak host workload.
- Use of PFC would enable loss-less network and ensure that congestion in network traffic belonging to one class would not affect that of another class.
Conclusion
It is quite evident that Remote Direct Memory Access (RDMA)-based Ethernet-based HyperSwap is not only a cheaper and flexible solution, but it is also comparable with Fibre Channel in performance. Ethernet based IBM Hyperswap enables All-Ethernet data centres and long-distance high availability active-active solutions.
References
About authors
Shrirang Bhagwat is a development lead in the Spectrum Virtualize team at IBM Systems Labs, India. He specializes in block storage for Ethernet environment. He can be reached at: shbhagwa@in.ibm.com.
@Abhishek Jaiswal is a senior developer in the Spectrum Virtualize team at IBM Systems Labs, India. He has expertise in block storage and working on the Ethernet mission for the IBM Spectrum Virtualize product. He can be reached at: ajaiswa9@in.ibm.com.
#StorageAreaNetworks#Storage#PrimaryStorage