This post brought to you by VMwarewolf

All about Time in VMware

Multiple symptoms, one solution>>

Symptoms:
1. Unable to ssh into your ESX host
2. Unable to login to the ESX console
3. ESX shows not responding in VirtualCenter
4. HA error (but esx host responding)
5. Unable to VMotion (but esx host responding)

Check ps output and if you see

# ps -ef
root 6232 0.0 0.0 0 0 ? Z Sep24 0:00 [cimservera <defunct>]
root 6377 0.0 0.0 0 0 ? Z Sep24 0:00 [cimservera <defunct>]
root 6496 0.0 0.0 0 0 ? Z Sep24 0:00 [cimservera <defunct>]

Solution:
http://kb.vmware.com/kb/1007887

This post brought to you by VMwarewolf

Defunct cimservera processes causing havoc

This is a new behavior introduced in Update 2.

Virtual Machine Migrations Are Not Recommended When the ESX Server Host Is Entering the Maintenance or Standby Mode

No virtual machine migrations will be recommended (or performed, in fully automated mode) off of a host entering maintenance or standby mode, if the VMware HA failover level would be violated after the host enters the requested mode. This restriction applies whether strict HA admission control is enabled or not.

The solution is to either disable HA on the cluster, or migrate the machines manually. Once all machines are off the host, it will go into maintenance mode.

The release notes mention this here: http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html#knownhaissues

Think you have enough HA capacity that this is not your problem?  Try un-configuring HA on your cluster and test. If it goes into Maintenance Mode after this, then this is indeed it.

My understanding is that this will be corrected in a future release but don’t hold me to it.

This post brought to you by VMwarewolf

Host won’t go into Maintenance Mode

Some customer are reporting the message:

HA agent on <esxhostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Network:
Consider using the Advanced Cluster Settings das.allowNetwork to control network usage.

VMware has just published two Knowledgebase articles for this issue.  KB 1006606 covers the network compliance check, which will also link to KB 1006541 covering how to address the issue.  

I am also told VMware is working to have this included in the Update 2’s release notes.

This post brought to you by VMwarewolf

Incompatible HA Network

Common reports include:

“insufficient resources to satisfy HA failover level on cluster” and “unable to contact a primary HA agent in cluster”

We think we know what is causing this, so I’m sharing this here in hopes of saving you a call.

The key is that all of your hostnames are in lower case only.

So, in Virtualcenter check-
-each host in the cluster, go to the configuration tab, DNS and routing, hostname
-everything needs to be lowercase
On each one of your ESX hosts-

ssh into each host and check-
-lower case names only in /etc/hosts
lowercase names only in /etc/sysconfig/network
cat /proc/sys/kernel/hostname

if not correct :

hostname <FQDN>

after this you can check the settings are correct using-

uname -n
hostname -a
hostname -s
hostname -d
hostname -f
hostname

in addition to this also make sure hostname in /etc/vmware/esx.conf is lowercase-

/adv/Misc/HostName = <FQDN in lowercase>
when done enable HA on cluster again.

Check this link if you are seeing the HA incompatible network message.

This post brought to you by VMwarewolf

HA problem with Virtualcenter 2.5 Update 2

A question was raised about the calculation used in admission control. I was previously under the impression that Configured Memory (in a VM) was the number used in this consideration. Some further investigation has revealed this is incorrect. It is the Reserved Memory, plus overhead, that is used in this calculation. VMs with no reservation still consume the overhead.

Here is vmkernel overhead-

Once you have these correct numbers you can continue the calculations in my previous post.

Remember, this pertains to ESX 3.0.x and not ESX 3.5 or 3i.

These numbers have changed in VC 2.5 and ESX 3.5. It is explained in detail on page 136 – 137 (Resource Management Guide).

This post brought to you by VMwarewolf

VMware HA Admission Control

HA Failover Capacity

Failover Capacity is determined using a slot size value that is calculated on the cluster. Slots are calculated by a combination of the total CPU and Memory that are in the physical hosts. The calculation for failover capacity works as follows:

Let’s say you have 4 ESX servers in your VMware HA cluster and Configured Failover capacity on the cluster is set to 1.

Physical memory in the hosts is as follows:

ESX1 = 16 GB
ESX2 = 24 GB
ESX3 = 32 GB
ESX4 = 32 GB

In the cluster you have 24 VM’s each configured and running. Of the 24 VM’s running, determine the VM which has the highest “configured memory”. For this example let’s say this is 2GB. All other VMs are configured with less or equal to 2GB.

With this information we can now do the calculation:
1. Pick the ESX host which has the least amount of RAM. In this case it is ESX1 and the minimum amount of RAM is = 16 GB

2. Divide the value found in step 1 with value for the maximum RAM in a VM. In my example this gives us 8 (16 divided by 2). This means we have 8 slots available per ESX host in the cluster.

3. Since we have 4 hosts and the configured failover capacity for the cluster is 1, we are left with 3 hosts in a failure situation. Hence the total number of VMs that can be powered on these 3 servers is 24 VMs. (i.e. 8 multiplied by 3 = 24)

4. If the total number of VMs in the cluster exceeds 24 then it will give us “Insufficient resources to satisfy HA failover” and the “current failover capacity will be shown as 0″. If the number is less than 24, we should not get this message.

Note: If you are still seeing the message and you have less VM’s running than in the calculation allows for, check both the CPU and Memory reservations on both VM’s and resource pools, as this can skew the calculation. You should avoid unnecessary memory or cpu reservations on VM’s as this can cause these types of errors to occur, because we have to ensure that the resource is available.

There are multiple ways to fix, or get around this calculation. The most common are as follows:
• Set the “Allow Virtual Machines to be powered on even if they violate availability constraints” in the configuration of the cluster. In this case it ignores the above calculation and will try to power on as many VM’s as possible in case of HA failover. If this is the option chosen you can also set restart priority in the ‘Virtual Machine Options’ section of the cluster configuration. This way any high priority VM’s are powered on first, and then the lower priority up to the point where we cannot power any further VM’s on
• If you have one VM which is configured with a very high amount of memory, you can either lower its configured memory, or take it out of the cluster and run it on any other standalone ESX host. This will increase the number of slots available with the current hardware
• Increase the amount of RAM on servers so that there are more slots available with the current RAM reservations.
• Remove any CPU reservations on any VM(s) that are greater than the max speed of the processors in the hosts. For example if the CPU Usage on the summary tab of your ESX Server shows as follows:

Then you will see the error message popup if you have a CPU reservation greater than 2793MHz on a VM.
Note: The above calculation method is very limited and is going to be revised in future releases of VirtualCenter to improve calculations for HA failover.

This post brought to you by VMwarewolf

HA Failover Capacity

A High Availability setup is only as good as its weakest link. In this session, learn more details about the inner workings of VMware HA and best practices for setting up and managing a robust production environment. We will review common pitfalls related to DNS, network and SAN setup, isolation response, and capacity planning, and will explain advanced configuration options available for customization. We will also share troubleshooting tips, and discuss some future directions.

View Session

This post brought to you by VMwarewolf

VMware HA Guidelines and Best Practices

VMware HA continuously monitors all ESX Server hosts in a cluster and detects failures. An agent placed on each host maintains a “heartbeat” with the other hosts in the cluster and loss of a heartbeat initiates the process of restarting all affected virtual machines on other hosts. You create and manage clusters using VirtualCenter. The VirtualCenter Management Server places an agent on each host in the cluster so each host can communicate with other hosts to maintain state information and know what to do in case of another host’s failure. (The VirtualCenter Management Server does not provide a single point of failure.) If the VirtualCenter Management Server host goes down, HA functionality changes as follows. HA clusters can still restart virtual machines on other hosts in case of failure; however, the information about what extra resources are available will be based on the state of the cluster before the VirtualCenter Management Server went down. HA monitors whether sufficient resources are available in the cluster at all times in order to be able to restart virtual machines on different physical host machines in the event of host failure. Safe restart of virtual machines is made possible by the locking technology in the ESX Server storage stack, which allows multiple ESX Servers to have access to the same virtual machines file simultaneously.

Host failure detection occurs 15 seconds after the HA service on a host has stopped sending heartbeats to the other hosts in the cluster. A host stops sending heartbeats if it is isolated from the network. At that time, other hosts in the cluster treat this host as failed, while this host declares itself as isolated from the network. By default, the isolated host powers off its virtual machines. These virtual machines can then successfully fail over to other hosts in the cluster. If the isolated host has SAN access, it retains the disk lock on the virtual machine files, and attempts to fail over the virtual machine to another host fails. The virtual machine continues to run on the isolated host. VMFS disk locking prevents simultaneous write operations to the virtual machine disk files and potential corruption.

A lot of folks forget that the default behavior for VMs in a host which becomes isolated is to power down.  We get a lot of calls asking “Why did my VMs power down unexpectedly?”

If the network connection is restored before 12 seconds have elapsed, other hosts in the cluster will not treat this as a host failure. In addition, the host with the transient network connection problem does not declare itself isolated from the network and continues running. In the window between 12 and 14 seconds, the clustering service on the isolated host declares itself as isolated and starts powering off virtual machines with default isolation response settings. If the network connection is restored during that time, the virtual machine that had been powered off is not restarted on other hosts because the HA services on the other hosts do not consider this host as failed yet. As a result, if the network connection is restored in this window between 12 and 14 seconds after the host has lost connectivity, the virtual machines are powered off but not failed over.

For more information on HA see-

• http://download3.vmware.com/vmworld/2006/tac9413.pdf
• http://kb.vmware.com/KanisaPlatform/Publishing/894/2956923_f.SAL_Public.html
• http://www.vmware.com/pdf/vmware_ha_wp.pdf

This post brought to you by VMwarewolf

How does the VMware HA work

This is particularly useful to those using VMotion and/or VMware HA (high availability).

Login to your ESX hosts with your favorite secure shell program and look at your /etc/hosts file.

The file should look something like this:

# Do not remove the following line, or various
# programs that require network functionality
# will fail.
127.0.0.1 localhost.localdomain localhost
192.168.14.2 myesxserver.foo.org

See the last line with the fully qualified domain name (FQDN) of the ESX server beside the IP address of that server? What you want to do is append to that line the shortname of the host as well. What you end up with looks like this:

# Do not remove the following line, or various
# programs that require network functionality
# will fail.
127.0.0.1 localhost.localdomain localhost
192.168.14.2 myesxserver.foo.org myesxserver

The typical way to do this is to insert a tab, then the name you chose for your server, up to (but not including) the first dot. You want to add every ESX host machine that is in your cluster to each other’s hosts file. Not only does this make HA much more robust, it makes DNS lookups redundant, and that’s a good thing. Ask yourself, if my DNS has an outage for just 12 seconds, do I really want all of my HA nodes going into isolation mode?

That’s it! Save your changes and exit.

Why do we need to do this? I’m not sure why it helps with VMotion, but HA needs it. HA you see was not written by the same developers as ESX. HA was developed by Legato, which is owned by EMC, as is VMware. It’s a marriage made in heaven, but the devil’s in the details!

***Here’s a Bonus Tip!

You need to also do this:

http://kb.vmware.com/kb/1339

Make this simple change to your ESX hosts now and save yourself a call into support.

This post brought to you by VMwarewolf

My Number 1 Tip for ESX