The Knowledge base team has been ultra-fast in responding to these, and two new KBs have appeared:

VMotion stops working after upgrading to vSphere 4.0 (1011294)

and

VMotion stops working after upgrading to vSphere 4 with a Licensing server is not configured error (1011296)

If you notice any issues with VMotion with vSphere, be sure to check those two articles out.

This post brought to you by VMwarewolf

VMotion stops working in vSphere

Unable to Power On virtual machine with “A General System error occurred: Internal error”

A virtual machine fails to power on, fails to leave suspend mode, or fails to migrate with VMotion and the following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
————–
Module License Power on failed.

Currently running virtual machines will continue to run but will not be able to perform migration with VMotion, suspend or power on virtual machines.

An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13.  These will be available from the page: http://www.vmware.com/download/vi.  Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2.

The Update patch bundles will be released separately later in the week.

This KB article will be updated as soon as more information is available, check back frequently for updates and additions.

A workaround to this ESX bug is to set the date back a day on your ESX hosts and you will be able to power on the VM’s again. If you use ntp, disable that for the time being and manually set your hosts back a day.  Be careful, if you have your VMware tools configured to sync VM time to the host time then this will alter your VM times.

***UPDATE***

Here is the latest public news from VMware on this>>

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully.  This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2.  The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:
-    VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2 (pre-Update 2 releases are not impacted by this problem).
-    Reports of problems with ESX 3.5 U1 with the following patches applied.
1.    ESX350-200806201-UG
2.    ESX350-200806202-UG
3.    ESX350-200806217-UG

-    No other VMware products are affected.

What has been done?:

-    Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
-    VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
-    A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs.  A new static page has been  published at http://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.
-    The phone system has been updated to advise customers of the problem
-    Vmware partners have been notified of the issue.

Workarounds:
1)    Do not install ESX 3.5 U2 if it has been downloaded from VMware’s website or elsewhere prior to August 12, 2008.
2)    Set the host time to a date prior to August 12, 2008.  This workaround has a number of very serious side affects that could impact product environments.  Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken.  These include, but are not limited to database servers, mail servers, & domain administration systems.

Next Steps:
VMware to send an email to all customers who have downloaded this version. This effort is underway and should happen before 11 am today.

Resolution:

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today.  The target timeframe is 6pm, August 12, 2008 PST.

FAQ:

1. What would this express patch do?

More information will be provided in subsequent communication updates.

2.   Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?

Yes.  We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week.  We will provide an ETA for the update patch bundles subsequently.  NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA.  They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.

3.   Why does VMware plan to reissue the upgrade media before the patch bundles?  That is a wrong priority call!

This is not a matter of priority.  Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.

4.   Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

5.   Does this issue affect VC 2.5 Update 2?

No.

6.   What is VMware doing to make sure that the problem won’t happen again?

We are making improvements on all fronts.  The product team had endeavored to deliver a release with support customers deem important.  But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused.  We have identified where the holes are and they will be addressed to restore customers’ confidence.

This post brought to you by VMwarewolf

ESX 3.5 Update 2 bug

This post brought to you by VMwarewolf

VMotion Error Messages

Operation timed out
Tasks: A general system error occurred:
Failed waiting for data. Error 16. Invalid argument

The VMware Knowledgebase writers have assembled a great list of possible reasons for this problem and I keep a printed out copy on my cubicle wall for this (ie: I didn’t write this)

Each step provides instructions or a link to a document with further details on how to eliminate possible causes and take corrective action to resolve the timeout. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution.

1. Verify that restarting the VMware Management Agents do not resolve the issue. For more information, see Restarting the Management agents on an ESX Server (1003490).
2. Verify that VMkernel networking configuration is valid. For more information, see Unable to set VMkernel gateway as there are no VMkernel interfaces on the same network (1002662).
3. Verify that VMkernel network connectivity exists using vmkping. For more information, see Testing vmkernel network connectivity with the vmkping command (1003728).
4. Verify that Console OS network connectivity exists. For more information, see Testing network connectivity with the Ping command (1003486).
5. Verify that Name Resolution is valid on ESX. For more information, see Identifying issues with and setting up name resolution on ESX Server (1003735).
6. Verify that time is synchronized across environment. For more information, see Verifying time synchronization across environment (1003736).
7. Verify that valid limits are set for the VM being VMmotioned. For more information, see VMware VMotion fails if target host does not meet reservation requirements (1003791).
8. Verify that hostd is not spiking the console. For more information, see Checking for resource starvation of the ESX Server service console (1003496).
9. Verify that VM is not configured to use a device that is not valid on target host. For more information, see Troubleshooting migration compatibility error: Device is a connected device with a remote backing (1003780).

***UPDATE (2/12/2010) There is now a great video on YouTube on how to fix this problem here: http://www.youtube.com/watch?v=42n-osx64ek

Note: If VMotion continues to fail at 10 percent after trying the steps in this article, open a case with tech support

This post brought to you by VMwarewolf

VMotion fails at 10 percent

“Starting with ESX Server 3.5 and VirtualCenter 2.5, DRS applies a cap to the memory overhead of virtual machines to control the growth rate of this memory. This cap is reset to a virtual machine-specific computed value after VMotion migrates the virtual machine. Afterwards, if the virtual machine monitor indicates that the virtual machine requires more overhead memory, DRS raises this cap at a controlled rate (1MB per minute, by default) to grant the required memory until the virtual machine overhead memory reaches a steady-state and as long as there are sufficient resources.

For VirtualCenter 2.5, this cap is not increased to satisfy the virtual machine’s steady-state demand as expected. Thus, the virtual machine operates with an overhead memory that is less than its desired size, which in turn may lead to higher observed virtual machine CPU usage and lower virtual machine performance in a DRS-enabled cluster.”

If you feel you are running into this issue read http://kb.vmware.com/kb/1003638

I read over this knowledge base article and tried these settings myself.  Interestingly, every time I set the Mem.VMOverheadGrowthLimit parameter to 5 and closed the window, the setting is getting set back to 0 on me.  I even see it getting set back in the vmkernel logs.

I can only assume virtualcenter is doing this. I think a much better approach is the second method given in the knowledgebase:

To fix multiple ESX Server hosts

If this parameter needs to be changed on several hosts (or if the workaround fails for the individual host) use the following procedure to implement the workaround instead of changing every server individually:

   1. Log on to the VirtualCenter Server Console as an administrator.

   2. Make a backup copy of the vpxd.cfg file (typically it is located in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg).

   3. In the vpxd.cfg file, add the following configuration after the <vpxd> tag:

      <cluster>

              <VMOverheadGrowthLimit>5</VMOverheadGrowthLimit>

      </cluster>

      This configuration provides an initial growth margin in MB-to-virtual machine overhead memory. You can increase this amount to larger values if doing so further improves virtual machine performance.

   4. Restart the VMware VirtualCenter Server Service.

      Note: You will need to restart the VMware VirtualCenter Server Service, after which the new value for the overhead limit should be pushed down to all the clusters in VirtualCenter.

This post brought to you by VMwarewolf

VMotion in 3.5 DRS enabled Cluster causes Guest CPU to rise Dramatically

In some cases after this upgrade is done VMotion is disabled and warm migrations are not possible. The solution is to re-enable VMotion on the host.

Re-enable VMotion on the host:

1. Select the host in VI Client and navigate to the Networking section of its Configuration tab.
2. Click the Properties link for the switch associated with the VMkernel port.
3. In the switch Properties dialog box, select the appropriate network label and click Edit.
4. Select the VMotion Enabled checkbox and click OK.

This post brought to you by VMwarewolf

VMotion Is Disabled After ESX Server Upgrade

Here is how you can ensure selected virtual machines will never run on the same host-

1. Right-click on your cluster in VI client, select Edit Settings…
2. In the wizard, on the left select Rules
3. On the right, click Add…
4. Provide a name (like www-separate)
5. For Type select Separate Virtual Machines
6. Click Add…
7. Check off the VMs you want to stay separated.

Ta-da!

This post brought to you by VMwarewolf

How-to Keep Virtual Machines Mutually Exclusive

This post brought to you by VMwarewolf

Network copy failed for file nvram

Download the VMotion Info tool here…

Need more information on VMotion compatibility?

• Vmotion CPU Compatibility for Intel processors – http://kb.vmware.com/kb/1991
• Vmotion CPU Compatibility for AMD processors – http://kb.vmware.com/kb/1992
• Vmotion – How to override CPU masks due to cpu mismatch – http://kb.vmware.com/kb/1993
• Vmotion requirements – http://www.vmware.com/pdf/vi3_admin_guide.pdf (Chapter 14)
• Vmotion between Apples and Oranges: Understanding CPU Compatibility Restraints for Vmotion – http://download3.vmware.com/vmworld/2006/tac1356.pdf
• Vmotion and CPU Compatibility – http://www.vmware.com/community/thread.jspa?messageID=452231
• Vmotion and VM Compatibility Matrix for Dell PowerEdge Servers – http://www.dell.com/downloads/global/solutions/vmotion_compatiblity_matix.pdf
• Vmotion and VM Compatibility Matrix for HP Proliant Servers – ftp://ftp.compaq.com/pub/products/servers/vmware/vmmotion-compatibility-matrix.pdf
• HP Vmotion Compatibility Matrix – http://h18004.www1.hp.com/products/servers/software/vmware/hpvmotion-compatibility-matrix.html

This post brought to you by VMwarewolf

Are your servers VMotion Capable?

This is particularly useful to those using VMotion and/or VMware HA (high availability).

Login to your ESX hosts with your favorite secure shell program and look at your /etc/hosts file.

The file should look something like this:

# Do not remove the following line, or various
# programs that require network functionality
# will fail.
127.0.0.1 localhost.localdomain localhost
192.168.14.2 myesxserver.foo.org

See the last line with the fully qualified domain name (FQDN) of the ESX server beside the IP address of that server? What you want to do is append to that line the shortname of the host as well. What you end up with looks like this:

# Do not remove the following line, or various
# programs that require network functionality
# will fail.
127.0.0.1 localhost.localdomain localhost
192.168.14.2 myesxserver.foo.org myesxserver

The typical way to do this is to insert a tab, then the name you chose for your server, up to (but not including) the first dot. You want to add every ESX host machine that is in your cluster to each other’s hosts file. Not only does this make HA much more robust, it makes DNS lookups redundant, and that’s a good thing. Ask yourself, if my DNS has an outage for just 12 seconds, do I really want all of my HA nodes going into isolation mode?

That’s it! Save your changes and exit.

Why do we need to do this? I’m not sure why it helps with VMotion, but HA needs it. HA you see was not written by the same developers as ESX. HA was developed by Legato, which is owned by EMC, as is VMware. It’s a marriage made in heaven, but the devil’s in the details!

***Here’s a Bonus Tip!

You need to also do this:

http://kb.vmware.com/kb/1339

Make this simple change to your ESX hosts now and save yourself a call into support.

This post brought to you by VMwarewolf

My Number 1 Tip for ESX