Hi, I'm instaled new FS5300 with code 9.1.0.1 on ISCSI. Customer get extremly low performance (latency up to 70ms) at low load. When load is high performances are OK (in micro sec). Problem is there is a lot of smal data files to R/W and this latensis are not OK. Loks like storage is sllping.

Thank for sugestions

BR, Peter

------------------------------
Peter ROBNIK
------------------------------

Hi Peter,

You can raise a ticket with IBM regarding this issue. They will want all the logs, configuration and host information for the find the right solution and/or answer for this issue. Without this information, it's not possible to answer your question.

------------------------------
Nezih Boyacioglu
------------------------------

Hello Peter, are you using a dedicated iSCSI network or are you using the same ethernet management port as the FS5300 storage? You can also review this document https://www.redbooks.ibm.com/redbooks/pdfs/sg248543.pdf in the section: SCSI performance issues

------------------------------
Jose Ferrada
Ingeniero de Proyectos
S&A Consultores Chile S. A.
Santiago
------------------------------

Has the network got Jumbo Frames turned on?

------------------------------
Mike Lasham
Solution Constultant
Spectrum Consulting
Auckland
0272112001
------------------------------

Hey Peter, 

I believe the key factor here is that you stated the performance is horrible (70ms for a FS5300) when low IO is being pushed to storage, however if bigger IO is being pushed, performance improves. 
I would be interested in knowing if problem is also worse when reading data while the IO load is low, than writing.

To me this sounds like very possible TCP Delayed Ack behavior, which is a TCP network protocol which essentially buffers acknowledgment (ACK) packets for received data to merge them with subsequent ACKs or outgoing data segments, thereby enhancing network and host efficiency by reducing the number of ACK transmissions. While advantageous in high-throughput environments, this mechanism can introduce additional latency and degrade performance in low I/O or congested network conditions, as the host must wait for an ACK before sending the next data segment. Consequently, disabling delayed ACK is often recommended in storage environments (particularly with iSCSI) to minimize latency and optimize performance. This parameter can typically be configured within the host operating system or the virtual machine's network adapter settings.

Have a look at this IBM doc for the recommended config for iSCSI hosts.
iSCSI performance analysis and tuning

Let me know if this helps you!

Kevin Torres
IBM Storage Virtualize L2/SME

------------------------------
Kevin Torres Ocon
------------------------------

Hello Peter,

This could be due to the standard behaviour of the San storage when under low load. This is explained here:

https://www.ibm.com/support/pages/why-low-io-rates-can-result-high-response-times-reads-and-writes

Regards

Natan Bonašin

IBM Storage Virtualize SME

CIC Zagreb

------------------------------
Natan Bonasin
------------------------------