The other day I was looking to get a baseline of the built-in ethernet adapter of my recently upgraded vSphere home lab running on the Intel NUC. I decided to use iPerf for my testing which is a commonly used command-line tool to help measure network performance. I also found a couple of articles from well known VMware community members: Erik Bussink and Raphael Schitz on this topic as well which were quite helpful. Erik's article here outlines how to run the iPerf Client/Server using a pair of Virtual Machines running on top of two ESXi hosts. Although the overhead of the VMs should be negligible, I was looking for a way to benchmark the ESXi hosts directly. Raphael's article here looked promising as he found a way to create a custom iPerf VIB which can run directly on ESXi.

I was about to download the custom VIB and I had just remembered that the VSAN Health Check plugin in the vSphere Web Client also provides some proactive network performance tests to be run on your environment. I was curious on what tool was being leveraged for this capability and in doing a quick search on the ESXi filesystem, I found that it was actually iPerf. The iPerf binary is located in /usr/lib/vmware/vsan/bin/iperf and looks to have been bundled as part of ESXi starting with the vSphere 6.0 release from what I can tell.

One interesting thing that I found when trying to run iPerf in "server" mode is that you would always get the following error:

bind failed: Operation not permitted

The only way I found to fix this issue was to basically copy the iPerf binary to another file like iperf.copy which it would then allow me to start iPerf in "server" mode. You can do so by running the following command on the ESXi Shell:

cp /usr/lib/vmware/vsan/bin/iperf /usr/lib/vmware/vsan/bin/iperf.copy

Running iPerf in "Client" mode works as expected and the copy is only needed when running in "server" mode. To perform the test, I used both my Apple Mac Mini and the Intel NUC which had ESXi running with no VMs.

I ran the iPerf "Server" on the Intel NUC by running the following command:

/usr/lib/vmware/vsan/bin/iperf.copy -s -B [IPERF-SERVER-IP]

Note: If you have multiple network interfaces, you can specify which interface to use with the -B option and passing the IP Address of that interface.

I ran the iPerf "Client" on the Mac Mini by running the following command and specifying the address of the iPerf "Server":

/usr/lib/vmware/vsan/bin/iperf -m -i t300 -c [IPERF-SERVER] -fm

I also disabled the ESXi firewall before running the test, which you can do by running the following command:

esxcli network firewall set --enabled false

Here is a screenshot of my iPerf test running between my Mac Mini and Intel NUC. Hopefully this will come in handy for anyone needing to run some basic network performance tests between two ESXi hosts without having to setup additional VMs.

esxi-iperf

5 thoughts on “Quick Tip - iPerf now available on ESXi

  1. Nice, that was useful. Helped me noticed I missed a MTU setting between my freenas iSCSI and my ESXi host.

  2. Wondering if you can SSH in twice and then run both the Server and the Client but bind them each to a different vmK NIC to do a simple burn in test and verify your NICs are good (Switch ports potentially too)

  3. Hi – I’m seeing that iperf runs with a TCP listen on 5001, but the firewall rule that I can enable for VSAN health opens 5001 UDP. I get a timeout from a client to the server. Is there any way to enable a new firewall rule for TCP 5001 allow via the vSphere UI, or is the only way to set that using editing of the service.xml file on the host?

    • You would need to edit the service.xml to also enable TCP and you can either manually tweak it via init script to ensure it persists or create a custom VIB which adds in the new firewall rule

  4. Hmm … so I went through that, rule shows up as SRC and DST for port 5001 on TCP, server runs (shows listening on 5001), client that connects to other iperf instances (Windows 10, tested against Avamar / Data Domain works fine) fails to connect. I can run a server on the Windows 10 system and connect from the ESX host, but going the other direction (or from another iperf equipped VM on the flat network) fails. Firewall rules look like:

    IPERF

    outbound
    tcp
    dst
    5001

    inbound
    tcp
    src
    5001

    Firewall refreshed, and when using the esxcli network ip connection list | grep -i listen command the server shows listening on 5001 (TCP):
    [root@gandalf:/usr/lib/vmware/vsan/bin] esxcli network ip connection list | grep -i listen
    tcp 0 0 0.0.0.0:5001 0.0.0.0:0 LISTEN 954798 newreno iperf.usable

    Thoughts?

    Regards,
    Levi

Thanks for the comment!