Nested esxi lab for nsx – network issues

Preparing a lab of nested esxi host for nsx can be a pain

Beside the issues with deploying the vcsa, ( browser compatability, ephemeral dvS port, create DNS first..) which are all things that will help in planning production deploys. There were a number of weird network issues that had me pulling my hair out.

Here are some tip and elaborations that might help


Install vmware tools for nested ESXi


Create an iso of the vib and upload it to a datastore attached to the physical host
Connect the iso to the virtual esxi’s cd drive

ssh to the virtual esxi and run the following

# vmkload_mod iso9660
# /sbin/vsish -e set /vmkModules/iso9660/mount mpx.vmhba32:C0:T0:L0
# ls -al /vmfs/volumes/mpx.vmhba32:C0:T0:L0
# cp  /vmfs/volumes/mpx.vmhba32:C0:T0:L0/esx-tools-for-esxi-9.7.2-0.0.5911061.i386.vib  /tmp
# /sbin/vsish -e set /vmkModules/iso9660/umount mpx.vmhba32:C0:T0:L0
# cd /tmp
# esxcli software vib install -v /tmp/esx-tools-for-esxi-9.7.2-0.0.5911061.i386.vib


Settings to avoid mac and uuid duplication (important on cloned esxi hosts)

This was a killer, and only showed when deploying nsx controllers, vCenter didn’t seem to mind the duplicate mac, but nsx controller deployments failed.

Run the following on the nested host
# esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Delete the line entry /system/uuid in /etc/vmware/esx.conf and run /sbin/ to save it persistently. This ensure a new system uuid gets generated at boot time


Packet loss dup! packets and failing controllers in nsx when using jumbo frames

NSX Controller deployed then failed, pings to and from the controllers show intermittent timeout and packets loss,

You should change the nic for the nested vms to e1000e, can’t remember where I discovered this…

Power off a the virtual machine

Log in as root to the terminal of the VMware ESX host.

Locate and open the virtual machine configuration file (.vmx)

For each network interface you wish to change to e1000e, modify the virtualDev line:
ethernetN.virtualDev = “e1000” to ethernetN.virtualDev = “e1000e”

# vim-cmd vmsvc/getallvms

In this example, the Vmid is 2848.
Reload the .vmx file using this command:

# vim-cmd vmsvc/reload Vmid
# vim-cmd vmsvc/reload 2848


This way you don’t have to reconfigure the network


See  Nested ESXi Lab for a greenfield nested build



Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.