Preparing a lab of nested esxi host for nsx can be a pain
Beside the issues with deploying the vcsa, ( browser compatability, ephemeral dvS port, create DNS first..) which are all things that will help in planning production deploys. There were a number of weird network issues that had me pulling my hair out.
Here are some tip and elaborations that might help
Install vmware tools for nested ESXi
https://labs.vmware.com/flings/vmware-tools-for-nested-esxi
Create an iso of the vib and upload it to a datastore attached to the physical host
Connect the iso to the virtual esxi’s cd drive
ssh to the virtual esxi and run the following
# vmkload_mod iso9660
# /sbin/vsish -e set /vmkModules/iso9660/mount mpx.vmhba32:C0:T0:L0
# ls -al /vmfs/volumes/mpx.vmhba32:C0:T0:L0
# cp /vmfs/volumes/mpx.vmhba32:C0:T0:L0/esx-tools-for-esxi-9.7.2-0.0.5911061.i386.vib /tmp
# /sbin/vsish -e set /vmkModules/iso9660/umount mpx.vmhba32:C0:T0:L0
# cd /tmp
# esxcli software vib install -v /tmp/esx-tools-for-esxi-9.7.2-0.0.5911061.i386.vib
Settings to avoid mac and uuid duplication (important on cloned esxi hosts)
This was a killer, and only showed when deploying nsx controllers, vCenter didn’t seem to mind the duplicate mac, but nsx controller deployments failed.
Run the following on the nested host
# esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
Delete the line entry /system/uuid in /etc/vmware/esx.conf and run /sbin/auto-backup.sh to save it persistently. This ensure a new system uuid gets generated at boot time
Packet loss dup! packets and failing controllers in nsx when using jumbo frames
NSX Controller deployed then failed, pings to and from the controllers show intermittent timeout and packets loss,
You should change the nic for the nested vms to e1000e, can’t remember where I discovered this…
Power off a the virtual machine
Log in as root to the terminal of the VMware ESX host.
Locate and open the virtual machine configuration file (.vmx)
For each network interface you wish to change to e1000e, modify the virtualDev line:
ethernetN.virtualDev = “e1000” to ethernetN.virtualDev = “e1000e”
# vim-cmd vmsvc/getallvms
In this example, the Vmid is 2848.
Reload the .vmx file using this command:
# vim-cmd vmsvc/reload Vmid
# vim-cmd vmsvc/reload 2848
This way you don’t have to reconfigure the network
See Nested ESXi Lab for a greenfield nested build
ref:
http://blog.vconsult.nl/2013/11/25/issue-with-jumbo-frames-after-upgrading-nested-esxi-servers-in-the-lab-to-5-5-and-fix/
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003020
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026043