In the previous post I wrote about virtual machine cpu limits and finished off by mentioning that limits are not exposed to the guest OS.
That cpu limits can cause additional overhead, that your vm is hyperventilating because it cannot get enough cpu per cycle is a worry, but when it happens how do you detect it?
Here is an example, PerfTest-1 is a single vCPU vm with no cpu limit running on a 1,864 Mhz physical physical cpu
(I am using load storm to simulate application threads)
The OS is calculating cpu usage at just over 50%
vCenter is showing slightly higher values, but similar – (note: all vCenter graphs are realtime )
From this we see that usage in MHz is just over 1000 MHz
So lets cap the cpu to 750Mhz
From the OS the cpu has increased slightly, but the vm is certainly nowhere near critical usage
From the vCenter the percentage is looking ok, in fact according to vCenter it has even reduced to 40%
But look at the MHz value, if the limit is 750, then this vm is actually maxed out, though neither the OS or vCenter are triggering alarms.
Although it looks pretty good a first glance, operating system cpu percentage values have become obsolete due to the limit – 40% is actually 40% of the 1.864 GHz physical cpu on the host.
If you want to look at this from the OS, check this post – measure windows cpu when limits are applied
From the vCenter vm Performance tab we should look at CPU Ready values and CPU MHz Usage to get a clear picture.
This vm is using all it’s MHz
And a horrible 4470 ms ready peak, we can convert CPU Ready Summation values in Milliseconds to % here and we see that 4000 ms is 20% ready…
(yeap, thats the time you spend waiting in line at the grocery store)
Another useful counter is Demand, here we see the vm wants 1109 (avg.) cpu but it is limited to 750 MHz
To sum up the performance indicators we need to look at are CPU Ready, Usage in MHz, and Demand, when Demand is greater than Limit, ready times will increase.
From vCOPS we get a warning on the health badge, and on closer inspection we see the same Demand, and Limit values.
However nothing is shouting out HEY, READY VALUES ARE 20% ! or CPU Demand is much greater than your limit !
Please note Entitlement is a composite value based on total cluster capacity, vm shares, limits and reservations , and although here it is the same as Limit, VMware state its is not useful for performance monitoring…
I ran an additional test and set the limit to 500 MHz.
The ready value shot up to 8000 ms (40%) and not sniff of an alarm
If demand is above the limit, your ready values will skyrocket and neither vCenter or the OS will flag a cpu usage warning
Incidentally VCOPS has a link that leads you to a section on limits..
The Virtual Machine Has a CPU Limit Set / A limit is set on the CPU resources of the virtual machine.
Normal Demand value is higher than or equal to Limit, and the Demand value is higher than or equal to Limit.
If the Normal Demand line reaches the limit and the current demand is near or above the limit, an overly restrictive limit might be set for the virtual machine.
Consider increasing the limit for the virtual machine.
So that’s the impact, if demand is more than limit your vms may be running through cycles like hyperventilating maniacs, oh and zero alarms…!
In these follow up posts, we look at how to detect overly restrictive limits, setting a custom vcenter alarm to detect high ready issues, and understanding application cpu needs through vSphere performance console.