Unable to deploy NSX controller

NSX controller deployment fails, Operation timed out in nested lab

 

From the C# client you see NSX_Controler_xxxXXXxxxX being created, it hangs a 0 progress for a while then gets deleted

These can be a pain to diagnose, this is what I did

 

1. Copy to clipboard the Task that gives the name of the failed Controller, paste it to notepad, you will need the name

Deploy OVF template
NSX_Controller_84d70f5e-7715-4d03-9831-e55dda239ae1
Operation
timed out.

 

2. Go the the NSX web and download the tech support log

3. Open up the log (best with Notepad++ or similar) and search/find the first entry for the controller

Device URL Import Key: /NSX_Controller_84d70f5e-7715-4d03-9831-e55dda239ae1/VirtualLsiLogicController0:0 … 2015-10-24 20:18:25.425 GMT  INFO taskScheduler-25 ResourcePoolVcOperationsImpl:124 – HTTP method: POST 2015-10-24 20:18:25.423 GMT  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 – Extending HttpNfcLease.. 2015-10-24 20:18:25.435 GMT  WARN VirtualMachineDvfilterMonitor-1 VirtualMachineWorkQueue$WorkQueue:255 – Host not found for Vm vm-142, bypassing. 2015-10-24 20:18:25.436 GMT  WARN VirtualMachineDvfilterMonitor-1 VirtualMachineWorkQueue$WorkQueue:279 – no host found for vm-142, removing.

So in my case the host was not found, I’ve seen a couple of people report issues with space, dns… etc

See this post on controller commands to diagnose those issues

KB 2091375 point to outdated arp entries in the physical switch..

As the log indicated a connectivity issue between nsx manager and hosts I ssh to nsx and saw this

 

controller error 1

 

 

 

 

Now that’s really freaky, same mac for .54 and .55

On windows machine I even got this…

controller error 2

 

 

 

 

 

Wow… look at that mac duplication… nasty but cool, never seen that before, but at least I’ve found the problem

Ok, so this is a lab environment, the hosts are nested and clones, their vmnics are different but all the vmk0 nics are duplicates of the first virtual machine’s vmnic, (template from which the other vms were created)

 

KB: 1031111 discusses mac duplication and offers as a workaround

esxcfg-advcfg -s 1 /Net/FollowHardwareMac

Which worked a treat

 

The lesson: be careful in nested environments when cloning nested hosts

And read NSX for Newbies – Part 2: Home lab by Giuliano Bertello

 

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.