Global Storage Forum

Global Storage Forum

Connect, collaborate, and stay informed with insights from across Storage

 View Only

Understanding VMware NMP Round Robin with IBM FlashSystem Through Simulation

By Nezih Boyacioglu posted 05/15/26 07:30 AM

  

One of the most common VMware storage tuning recommendations over the years has been changing the Round Robin IOPS limit from the default value of 1000 to 1. Many VMware administrators working with all-flash storage have encountered this suggestion, particularly in environments using enterprise arrays such as IBM FlashSystem. 

This recommendation has become so common that many people apply it as standard practice without ever understanding the changes it makes to the ESXi host. That was actually the reason I decided to build a small simulator around it.
 
You can try the simulator here: VMware Round Robin Simulator
The default screen begins with an 800 IOPS workload across four active paths. Then, you can switch between 1,500, 2,100, and 3,200 IOPS workloads to see how ESXi distributes traffic differently as the workload increases. The simulator also allows you to switch between 2, 4, and 8 path configurations to see how the number of paths changes Round Robin behavior.

The idea behind the simulator is simple. In traditional SCSI-based VMware environments, the ESXi Native Multipathing Plug-in (NMP) uses an IOPS counter to determine when to switch from one storage path to another. With the default configuration, ESXi sends traffic through the same path until 1,000 I/O operations are completed, then rotates to the next available path.

On paper, that sounds perfectly reasonable. But in practice, especially with modern flash arrays, this can result in uneven path utilization.

For example, consider a datastore connected through four active paths. If the workload stays below the threshold at which aggressive path rotation occurs, ESXi may heavily utilize a single path while the remaining paths stay relatively quiet. In older spinning-disk environments, this behavior was not always noticeable because storage latency was already relatively high. Modern flash systems changed that completely.

Low-latency arrays make frontend path behavior much easier to observe. Queue distribution, path utilization, and congestion patterns suddenly become much more important. This explains why the IOPS=1 recommendation became so popular in the all-flash era. Rather than waiting for 1,000 I/Os before switching paths, ESXi rotates paths almost continuously. This allows all available paths to participate in the workload much more evenly.

This concept is typically illustrated with static diagrams or configuration snippets, but the actual runtime behavior is highly dynamic. Observing the workload distribution move across the paths in real time makes the recommendation much easier to understand than simply reading "set IOPS=1" in a best practices document.

This is not to say that the default configuration is "wrong." VMware environments have different workload patterns, storage architectures, and operational requirements. However, understanding how the path selection logic behaves helps explain why so many storage vendors and architects have historically recommended more aggressive path rotation for high-performance storage environments.

Even today, as NVMe-oF adoption continues to grow, many enterprise VMware environments still run on traditional SCSI architectures. This makes NVMe behavior relevant not only for performance optimization but also for troubleshooting and general SAN visibility.

0 comments
26 views

Permalink