virtual machine cpu limit considerations

Several years ago some of VMware’s most respected technical experts made recommendations against using virtual machine cpu limits.
yellow-bricks limiting-your-vcpu

More recently VMware stated “Resource limits on a virtual machine… form an artificial boundary… you need to take the possible impact into consideration before applying limits to either memory or CPU.
Impact of virtual machine CPU resource limits (1033115)

So why does this feature exist, is it a viable method to apply costing controls in the same way telephone companies put MB limits on internet usage, and what exactly are the possible impacts!

To appreciate the effects of virtual machine cpu limits we need to understand that vSphere dynamically determines entitlement to resources.

vSphere groups or cluster resources into pools, and then uses shares when deciding how resources in the pool are divided.
Shares are not fixed because the available resources may vary. For example add more physical cores and the pool of pcpu resource will increase.

memory values - shares variable

Priority is determined for each virtual machine by calculating priority values. vSphere’s cpu scheduler uses a queuing system; one for priority traffic and another for everybody else.

Those with priority have the opportunity to go first, but if they do not need cpu, the next vm with priority gets to go.
It’s a bit like the priority boarding queue at an airport, the queue is shorter so waiting time is reduced. However vms are very polite and let those in a rush go ahead

memory values - simple vcpu queue

Priority is calculated by  MHzUsed / Shares = MHzPerShare,  the lowest value has the higher priority
Notice, that vm-3 has 2000 shares, double the other vms, so it jumps the queue and goes ahead of vm2

memory values - vcpu queues (1)


From this we see that cpu usage and shares are the values taken into account when scheduling physical cpu cycles to virtual machines, so where do virtual machine cpu limits come into play.**

According to these VMware Experts Limits were originally used as a troubleshooting tool by developers to induce a state of overcommitment without having to overcommit the entire esx host.

Artificially capping resource allocation for a virtual machines additionally requires extra calculations from the local host scheduler. If a vcpu exceeds its cpu time, it is descheduled until it is allowed to run, while being descheduled cpu cycles are wasted.***

So a vm is descheduled and has to go back for more cpu.

Think in terms of the waiting line at the grocery store, if we start to say the maximum you can spend is $10 or $20 then some will need to go through the line two, three or more times to get all the weekly shopping done. Everybody has to work harder.

cpu limits - limit repass


In the above example vm3 needs 2 cycles to satisfy a demand above the limit


So limits are usually not good, but another point of concern is that limits are not exposed to the guest OS!
Now start to think in terms of the  impact of monitoring cpu usage from the OS or vCenter…
Could this mean that an organization’s cpu usage monitoring is basically worthless if cpu limits have been applied?

As a follow up check out the post  virtual cpu limits and monitoring


**There is a more technical description of how shares work at reservations-and-cpu-scheduling/
***vSphere 5.1 clustering deep dive, D. Epping and F. Denneman



In these follow up posts, we look at how to detect overly restrictive limits, setting a custom vcenter alarm to detect high ready issues, and understanding application cpu needs through vSphere performance console.

Detecting high vm cpu usage

Configure high cpu ready alarm

performance snorkel dives home

Join the Conversation


Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.