Primary Storage

You cannot change the speed of light and flash storage may notice it

By Archive User posted Tue May 21, 2019 05:20 PM


Originally posted by: smo_

You cannot change the speed of light and flash storage may notice it

especially when using data replication between data centers


With flash storage the distance between data center has more impact on the overall response time.



The speed of light in glass does not change whether you are using nearline, spinning disk or flash storage but the distance matter more when using flash storage



Let’s calculate the latency due to the distance

The speed of light is pretty much given: c0 = 300'000 km/s

The refraction index (n, for our fiber optic nfo) is defined as the speed of light in vacuum (c0) divided by its speed in a specific medium (cfo).

For fiber optics that's typically in a range of 1.4 to 1.6. So let's take nfo 1.5 as an example.
nfo = c0 / cfo

Rearranging that equation for the speed in fiber optics gives us:
cfo = c0 / nfo = 300'000 km/s / 1.5 = 200'000 km/s


The reciprocal tells us how long the light travels through a 1 km fiber cable:
tfo = 1 km / cfo = 1 km / 200'000 km/s = 5 * 106 s = 5 µs



As a rule of thumb, a link with a round-trip latency of 1 ms spans approximately 100 km.

or seen from another angle a link that spans 100 km has an approximate round-trip time of 1ms
(which adds to the latency in synchronous data replication across that distance).



   Now let’s see in % the influence of the distance to the overall response time



We can consider between 10% to 20 % as a noticeable impact, you will reach this for example starting 20 KM distance for a flash storage with 2ms response time


This previous table does not take into account the possible latency that can be added with compression, IPsec , FEC

Encryption/compression is in the range of 5.5 μsec per node; (BROCADE 6510)
Forward error correction (FEC) adds 400 ns between E_Ports (enabled by default).

Latency for locally switched ports is in the range of 700 ns ( Brocade300)

Brocade extension guide mention:  IPsec has been implemented in HW and operates at line rate, adding negligible propagation delay.


Whatever the synchronous technology you are using, you need to give the round-trip time to the frame


How to overcome this

There are solutions that can spoof the acknowledge.
For example WAN accelerator (that can be used with FCIP)  can locally acknowledge TCP segment to the local device.

The draw back is that any lost packet within a window requires that the entire window be re transmitted – unless your TCP/IP stack on the server uses selective acknowledgements”



Here is an interesting description of the different replication to use


but you still cannot beat the speed of light in glass, except with hollow-core photonic crystal fibres

* The speed of light in vacuum, commonly denoted c, is a universal physical constant important in many areas of physics. Its exact value is 299,792,458 metres per second


note: alternative title for this blog entry could have been "In the middle of difficulty lies opportunity" (Albert Einstein) 

tiny url




Thu September 17, 2020 01:54 PM

Good article...  It's definitely a problem that's gotten worse since my days of working with the IBM 2029 Fiber Saver and trying to explain to customers why the speed of light isn't necessarily fast enough for what they want.

Tue July 02, 2019 05:51 AM

Originally posted by: Jon_Herd

Brilliant article and very well explained to de-mistify the "rules" of fiber transmission and latency - nice one !