This week we have been presented with an interesting scenario to solve. A medium sized VMware View deployment across HP Blades (BL460) and C7000 enclosures was presenting dissimilar performance on virtual desktops with exact same workload. After a little analysis we found out that some hosts were performing below the desired performance numbers. Measuring the environment was possible to identify that CPU, Memory and disk IO were not an issue.

The symptoms:

• Virtual machine application performance on HP hardware is poor when compared to another host.
• Applications run slower when they are located on an HP host. When they are moved to another host, performance improves.
• CPU READY shows a high value.

After a little while one of my colleagues onsite (Thanks Simon) found out that the HP blades had a different Power Regulator setting. After changing the setting all performance issues were gone. Interesting enough this is a Default setting on HP blades.

Let’s see what Power Regulator is responsible for:

In Dynamic Power Savings mode, Power Regulator determines the amount of time each processor in the system is spending in the operating system’s idle loop. When the ratio of time spent in the idle loop to the time performing useful work is high, the algorithm instructs the processor to set its power state to Pmin (the lowest power and performance mode of the processor).

When in HP Dynamic Power Savings mode, the Power Regulator algorithm continuously monitors application and processor loading up to eight times a second and adjusts the P-state accordingly. This continuous monitoring results in optimized P-state transitions.

ESX only sees CPU cycles & MHZ presented to the hypervisor. If the power regulator decides to change the power consumption, therefore changing CPU core frequency, the hypervisor will not be aware of such changes. The Power Regulator algorithm should match processor power consumption at any point in time, but for a unknown reason it does not. (yes, firmware was up-to-date)

If the BL460 were fully loaded and CPU utilisation was high this problem would have never been noticed because the Power Regulator algorithm would never kick in to reduce the CPU core frequency. However, because the environment was under-utilised the Power Regulator was kicking in. Despite HP in their document “Power Regulator for ProLiant servers” mention that dynamic power algorithm executes on a per-core basis I can assume that the process is averaging CPU utilisation or something similar.

Luckily the guys at VMware had already published a KB1018206 for the issue.

There is an alternative option to disabling the feature. Change it to Custom and set it to OS Control Mode to have it managed by ESX 4.0 and later.

Just thought would be nice to share since this is a Default setting and many HP setups could be performing a whole lot better. Double check your settings.