I am evaluating the feasibility and the performance of attaching a GPFS directory on a KVM which has already been connected to an Infiniband network via SR-IOV
1) I managed to mount the GPFS via the virtual IB, but it is very unstable and caused hiccups for the whole cluster, as the GPFS on the VM kept being kicked out by other nodes, so I decided not to go in this direction
2) I mount the GPFS directory via NFS with three different ways on the same VM, i.e.
a) NFS server (with IB-mounted GPFS) on a neighbour node via a 10GE network
b) NFS server on the same neighbour node via 100G IB (IPoIB)
c) NFS server on the VM host (which is also IB-mounted the GPFS) via IPoIB
and evaluate the write speeds on various mount points with dd:
For a single chunk of a 1GB and 2 GB file, the IB-mounted GPFS on the VM host is as expected far superior (1.4 - 2.9 GB/s) to the NFS via IPoIB on the VM (~500-560MB/s, regardless of where the NFS servers is hosted), and NFS via 10GE on the VM is the worst (~300MB/s)
Interestingly, when I turn to test with a large number of small files (1024bytes x 1048576), the NFS via 10GE on the VM is 3.6MB/s, the NFS via IPoIB on the VM is between 5.1-6.2MB/s, the IB-mounted GPFS on the VM host is only 1.4MB/s.
Can anyone tell if this is the expected result and why?
Thank you!
~ Dominic
------------------------------
Dominic Chien
------------------------------