The Knowledge base team has been ultra-fast in responding to these, and two new KBs have appeared:

VMotion stops working after upgrading to vSphere 4.0 (1011294)

and

VMotion stops working after upgrading to vSphere 4 with a Licensing server is not configured error (1011296)

If you notice any issues with VMotion with vSphere, be sure to check those two articles out.

This post brought to you by VMwarewolf

VMotion stops working in vSphere

This post brought to you by VMwarewolf

All about Time in VMware

• ESX disconnects randomly from VirtualCenter
• ESX disconnects when performing VI Client tasks from VirtualCenter.
• Tasks randomly timeout after a long idle time
• “An error occurred communicating to the remote host” pops up.

This only refers to Update 3. This issue seem to be affecting customers with firewalls using state-ful inspection, such as a Juniper ISG 2000, Software Revision 6.1.0R3.

The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX.

A knowledgebase article is in the works, but Engineering is still actively investigating this problem. One work-around that I have seen mentioned (not verified to work!) is as follows:

• Create a dummy VM on each host (e.g. 16 MB RAM, no disk, no network).
• Set CPU affinity to the last core to prevent VMotion.
• Create a new Scheduled Task that performs:
  • Change power state: Power-on dummy VM. Every hour, on the hour.
• Create another Scheduled Task that performs:
  • Change power state: Power-off dummy VM. Every hour, 30 minutes after the hour.

Like I said, Engineering is aware of this issue and actively investigating it.

This post brought to you by VMwarewolf

ESX disconnects randomly or when doing VI client tasks from VC, task randomly timeout after a long idle time

A new article I saw on VIOPS today that helps one understand some of the basic premises of permissions and roles. Sometimes a good overview like this one, written from thirty-thousand feet, helps one solidify VMware’s security model in one’s mind. The article is broken down into major sections:

1. Concepts
2. Definitions
3. Datastores and Networks have no direct privileges
4. Definitions
5. VMs inherit privileges from two sources
6. Clusters and Hosts implicitly are resource pool
7. Privileges Needed to Create a Virtual Machine
8. Privileges Needed for various Inventory Manipulations

Anything to help understand VMware’s roles and permissions is a good thing. Read VI3 Roles and Permissions

Concepts behind this are explained in the paper Management VirtualCenter Roles and Permissions.

This post brought to you by VMwarewolf

Understanding VMware Roles and Permissions

C:\Program Files\VMware\Infrastructure\VirtualCenter Server\Logs

Look for this message:

[2008-01-17 01:05:43.917 'App' 304 error] Unable to get exclusive access to VC repository.

This means there are some connections to the virtualcenter database (repository) that need to be freed before virtualcenter can use it. It will not start until it can get this exclusive access.

You can either restart the entire database server or locate any connections to the database and terminate them.  Then restart the virtualcenter service.

This post brought to you by VMwarewolf

Unable to get exclusive access to VC repository

Failed to initialize: not well-formed (invalid token)

The problem seems to happen if one or more ESX hosts being managed have Update 2 on them, but Virtualcenter itself is running Update 1 or earlier.

The problem doesn’t show right away.  Not until Virtualcenter server is actually restarted, and it won’t start.

The resolution is to upgrade Virtualcenter to Update 2. Apparently some schema changes are applied to the SQL database that resolve this. After this is done the service should start ok.

This post brought to you by VMwarewolf

Failed to initialize not well-formed invalid token

Unable to Power On virtual machine with “A General System error occurred: Internal error”

A virtual machine fails to power on, fails to leave suspend mode, or fails to migrate with VMotion and the following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
————–
Module License Power on failed.

Currently running virtual machines will continue to run but will not be able to perform migration with VMotion, suspend or power on virtual machines.

An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13.  These will be available from the page: http://www.vmware.com/download/vi.  Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2.

The Update patch bundles will be released separately later in the week.

This KB article will be updated as soon as more information is available, check back frequently for updates and additions.

A workaround to this ESX bug is to set the date back a day on your ESX hosts and you will be able to power on the VM’s again. If you use ntp, disable that for the time being and manually set your hosts back a day.  Be careful, if you have your VMware tools configured to sync VM time to the host time then this will alter your VM times.

***UPDATE***

Here is the latest public news from VMware on this>>

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully.  This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2.  The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:
-    VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2 (pre-Update 2 releases are not impacted by this problem).
-    Reports of problems with ESX 3.5 U1 with the following patches applied.
1.    ESX350-200806201-UG
2.    ESX350-200806202-UG
3.    ESX350-200806217-UG

-    No other VMware products are affected.

What has been done?:

-    Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
-    VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
-    A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs.  A new static page has been  published at http://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.
-    The phone system has been updated to advise customers of the problem
-    Vmware partners have been notified of the issue.

Workarounds:
1)    Do not install ESX 3.5 U2 if it has been downloaded from VMware’s website or elsewhere prior to August 12, 2008.
2)    Set the host time to a date prior to August 12, 2008.  This workaround has a number of very serious side affects that could impact product environments.  Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken.  These include, but are not limited to database servers, mail servers, & domain administration systems.

Next Steps:
VMware to send an email to all customers who have downloaded this version. This effort is underway and should happen before 11 am today.

Resolution:

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today.  The target timeframe is 6pm, August 12, 2008 PST.

FAQ:

1. What would this express patch do?

More information will be provided in subsequent communication updates.

2.   Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?

Yes.  We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week.  We will provide an ETA for the update patch bundles subsequently.  NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA.  They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.

3.   Why does VMware plan to reissue the upgrade media before the patch bundles?  That is a wrong priority call!

This is not a matter of priority.  Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.

4.   Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

5.   Does this issue affect VC 2.5 Update 2?

No.

6.   What is VMware doing to make sure that the problem won’t happen again?

We are making improvements on all fronts.  The product team had endeavored to deliver a release with support customers deem important.  But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused.  We have identified where the holes are and they will be addressed to restore customers’ confidence.

This post brought to you by VMwarewolf

ESX 3.5 Update 2 bug

Some customer are reporting the message:

HA agent on <esxhostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Network:
Consider using the Advanced Cluster Settings das.allowNetwork to control network usage.

VMware has just published two Knowledgebase articles for this issue.  KB 1006606 covers the network compliance check, which will also link to KB 1006541 covering how to address the issue.  

I am also told VMware is working to have this included in the Update 2’s release notes.

This post brought to you by VMwarewolf

Incompatible HA Network

Common reports include:

“insufficient resources to satisfy HA failover level on cluster” and “unable to contact a primary HA agent in cluster”

We think we know what is causing this, so I’m sharing this here in hopes of saving you a call.

The key is that all of your hostnames are in lower case only.

So, in Virtualcenter check-
-each host in the cluster, go to the configuration tab, DNS and routing, hostname
-everything needs to be lowercase
On each one of your ESX hosts-

ssh into each host and check-
-lower case names only in /etc/hosts
lowercase names only in /etc/sysconfig/network
cat /proc/sys/kernel/hostname

if not correct :

hostname <FQDN>

after this you can check the settings are correct using-

uname -n
hostname -a
hostname -s
hostname -d
hostname -f
hostname

in addition to this also make sure hostname in /etc/vmware/esx.conf is lowercase-

/adv/Misc/HostName = <FQDN in lowercase>
when done enable HA on cluster again.

Check this link if you are seeing the HA incompatible network message.

This post brought to you by VMwarewolf

HA problem with Virtualcenter 2.5 Update 2

Release Notes links:

VMware ESX/ESXi Server 35 Update 2 http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html

ESX Server 3i version 3.5 Update 2 Installable http://www.vmware.com/support/vi3/doc/vi3_esx3i_i_35u2_vc25u2_rel_notes.html

ESX Server 3i version 3.5 Update 2 Installable Embedded http://www.vmware.com/support/vi3/doc/vi3_esx3i_e_35u2_vc25u2_rel_notes.html

VMware Virtual Center 25 Update 2 http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html

Update Manager Update 2 http://www.vmware.com/support/vi3/doc/vi3_vum_10u2_rel_notes.html

Converter Enterprise Update 2 http://www.vmware.com/support/vi3/doc/vi3_vec_10u2_rel_notes.html

Build Numbers

• VMware ESX/ESXi Server 35 Update 2 Build 103908
• ESX Server 3i version 3.5 Update 2 Installable Embedded Build 103909
• ESX Server 3i version 3.5 Update 2 Installable Build 103909 VMware Infrastructure Management Installer Build 104263 (EN)
• VirtualCenter Server 2.5 Update 2 Build 104215 (EN)
• VMware Infrastructure Management Installer Build 104265 (G11n)
• VirtualCenter Server 2.5 Update 2 Build 104217 (G11n)
• VMware Infrastructure Client Build 104249 (G11n)
• VMware Consolidated Backup 1.5 Build 102898
• Update Manager Update 2 Build 103284
• Converter Enterprise Update 2 Build 102422 (EN)
• Converter Enterprise Update 2 Build 102445 (G11n)

This post brought to you by VMwarewolf

VI3.5 Update 2 Release Notification