Down the left side in yellow are common problems we get calls on. To the right of each problem are the steps you would take to resolve the problem. I have included hyperlinks to help get you to the answers fast.

Feel free to share with all!

This post brought to you by VMwarewolf

Common Licensing issues in VMware Infrastructure

Related posts:

1. Common Network issues in VMware Infrastructure
2. Common system management issues in VMware Infrastructure
3. Security Architecture Design and Hardening in VMware Infrastructure 3
4. Best Practices for Performance - VMware Infrastructure 3
5. VCP Equals a lot of CASH

The main problems are shown in yellow.  The steps to resolve these issues are listed on the same line, in order of operation.  Each line represents a different problem and set steps to resolve it.

Feel free to share with all!

This post brought to you by VMwarewolf

Common Network issues in VMware Infrastructure

Related posts:

1. Common system management issues in VMware Infrastructure
2. Common Licensing issues in VMware Infrastructure
3. Best Practices for Performance - VMware Infrastructure 3
4. Incompatible HA Network
5. Security Architecture Design and Hardening in VMware Infrastructure 3

One of the things I am currently in charge of in my area of expertise are what we call Resolution Paths. Resolution Paths are collections of modular steps that can be used to solved tech support issues.  Being modular, they can be re-used in other paths. A good example is using the ping command to test network connectivity.  This step is used in all kinds of troubleshooting procedures. Put a number of these steps together, and you have a method.

What I have for you today is an index of sorts, of the top issues we see in the System Management team, along with all of the steps used in resolving these issues.  You might want to download this file and keep it handy somewhere.  All of the titles are linked to the public Knowledgebase articles for quick access.

Download this file, unzip it, and open in a browser.  That’s it!!

The main problems are shown in yellow.  The steps to resolve these issues are listed on the same line, in order of operation.  Each line represents a different problem and set steps to resolve it.  Make sense?

Feel free to share with all!

This post brought to you by VMwarewolf

Common system management issues in VMware Infrastructure

Related posts:

1. Common Network issues in VMware Infrastructure
2. Common Licensing issues in VMware Infrastructure
3. Security Architecture Design and Hardening in VMware Infrastructure 3
4. VCP Equals a lot of CASH
5. ESX Patch Management

Administrators and developers can use VIMA to run scripts and agents to manage ESX and ESXi systems. VIMA includes the VI Perl Toolkit and the VMware Infrastructure Remote CLI. VIMA also includes the vi-fastpass authentication component and the vi-logger logging component. vi-fastpass allows direct connection to established target servers. vi-logger allows you to collect logging information from ESX and ESXi hosts and store the information on VIMA for analysis.  

It is available as a free download!

Key VIMA Features:

• Manage ESX hosts
• App authentication
  (fast pass, credential store)
• 64-bit Linux OS
• VMware Tools
• VI Perl Toolkit
• RCLI
• Basic SNMP
• Centralized ESX Server Logging
• Updatable 
• Sample Code

Support Page: http://www.vmware.com/support/developer/vima/index.html

This post brought to you by VMwarewolf

Download VIMA today!

Related posts:

1. VMware Workstation 6 Download
2. How does the VMware HA work
3. There’s Help for Windows Vista
4. ESX Patch Management

A large part of the job is scrubbing rough notes that support engineers have jotted down in the haste of solving a customers crisis, and turning these diamonds in the rough consumable and re-usable.  I think I have something more to offer as well.  Running this blog and several others has taught me a few things about web sites and search.  For instance do a Google search for the phrase “vmotion fails“.  How is it that the number one result of this search is this blog, while VMware’s own Knowledgebase article Diagnosing VMware VMotion failure at 10% (1003734) is nowhere to be found? Another example is a search for “ESX logs“.

I know why.

No SEO (Search Engine Optimization).

I won’t go into the details of why here but the is that no one has made these important web pages search engine friendly. Microsoft has no trouble making their knowledgebase search able. Well, I’m really hoping I’m given a chance to help with that. Readers such as yourself will be able to find resolutions to your problems more easily and support staff will enjoy fewer repeat calls for the same problem over and over.  It’s win-win.

If you start seeing VMware knowledgebase articles in your search results over the next few months remember the VMwarwolf sent ya.

This post brought to you by VMwarewolf

VMware Knowledge needs to be made Findable

Related posts:

1. Which ESX log file
2. Which Virtual Center Log File
3. VMotion in 3.5 DRS enabled Cluster causes Guest CPU to rise Dramatically
4. Common system management issues in VMware Infrastructure
5. Unable to get exclusive access to VC repository

Apparently guest customization fails wherein the virtual machine boots (and every time it reboots), the customization screen flashes briefly and the Windows login screen appears.

VMware has been quick (today) to publish a knowledgebase article on it here >> Sysprep no longer runs on template after upgrading to VirtualCenter 2.5 Update 2

This is a known issue with VirtualCenter 2.5 Update 2 and the KB article suggests it will be addressed in VirtualCenter 2.5 Update 3. A workaround is also provided.

This post brought to you by VMwarewolf

Guest customization fails on VirtualCenter 2.5 Update 2

Related posts:

1. Customization fails after VC 2.5 Upgrade
2. VMotion in 3.5 DRS enabled Cluster causes Guest CPU to rise Dramatically
3. Failed to install the VirtualCenter Agent Service
4. VirtualCenter Best Practices
5. ESX Disconnected after Upgrading to VirtualCenter 2.0.1

The mechanism for Live Cloning goes something like this>>

• Whenever user selects “Live clone…”, create a snapshot (using VIM). Increment the snapshot counter for this.
• If the user does a clone of this VM, create a delta disk off the snapshot delta disk.
• The snapshot is protected so the user does not inadvertently destroy the snapshot chain.
• If consolidate is called on a chain with snapshots generated by live clone, then the snapshots will be collapsed. Nothing special here, since the delta disks already have this relationship set up, so the consolidate should just work.

The snapshot taken is a quiesced snapshot and first attempts to use the sync driver to check the I/O state. However, with transactional applications, such as MS SQL Server, the I/O may not stop/pause long enough for the sync driver to work properly. (This is similar to potential issues encountered with VCB.)

Because this is the method used to implement Live Cloning it is best to not use Live Cloning on VMs with transactional applications.

This post brought to you by VMwarewolf

Live Cloning

Related posts:

1. Forgotten Snapshot Locator
2. VMware Fusion Released
3. How does the VMware HA work
4. Convert Physical Machines to Virtual Machines – Free!

This is a new behavior introduced in Update 2.

Virtual Machine Migrations Are Not Recommended When the ESX Server Host Is Entering the Maintenance or Standby Mode

No virtual machine migrations will be recommended (or performed, in fully automated mode) off of a host entering maintenance or standby mode, if the VMware HA failover level would be violated after the host enters the requested mode. This restriction applies whether strict HA admission control is enabled or not.

The solution is to either disable HA on the cluster, or migrate the machines manually. Once all machines are off the host, it will go into maintenance mode.

The release notes mention this here: http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html#knownhaissues

Think you have enough HA capacity that this is not your problem?  Try un-configuring HA on your cluster and test. If it goes into Maintenance Mode after this, then this is indeed it.

My understanding is that this will be corrected in a future release but don’t hold me to it.

This post brought to you by VMwarewolf

Host won’t go into Maintenance Mode

Related posts:

1. VI3.5 Update 2 Release Notification
2. How does the VMware HA work
3. How-to Keep Virtual Machines Mutually Exclusive
4. VMware VirtualCenter 2.0.2 Released
5. VMware Infrastructure 3 Release Notes part deux

C:\Program Files\VMware\Infrastructure\VirtualCenter Server\Logs

Look for this message:

[2008-01-17 01:05:43.917 'App' 304 error] Unable to get exclusive access to VC repository.

This means there are some connections to the virtualcenter database (repository) that need to be freed before virtualcenter can use it. It will not start until it can get this exclusive access.

You can either restart the entire database server or locate any connections to the database and terminate them.  Then restart the virtualcenter service.

This post brought to you by VMwarewolf

Unable to get exclusive access to VC repository

Related posts:

1. Which Virtual Center Log File
2. Failed to install the VirtualCenter Agent Service
3. Which ESX log file
4. Failed to initialize not well-formed invalid token
5. Web Access Broken if Custom Port Specified

Failed to initialize: not well-formed (invalid token)

The problem seems to happen if one or more ESX hosts being managed have Update 2 on them, but Virtualcenter itself is running Update 1 or earlier.

The problem doesn’t show right away.  Not until Virtualcenter server is actually restarted, and it won’t start.

The resolution is to upgrade Virtualcenter to Update 2. Apparently some schema changes are applied to the SQL database that resolve this. After this is done the service should start ok.

This post brought to you by VMwarewolf

Failed to initialize not well-formed invalid token

Related posts:

1. Unable to get exclusive access to VC repository
2. Failed to install the VirtualCenter Agent Service
3. Network copy failed for file nvram
4. VirtualCenter 2.5 Upgrade Database Issues
5. Clustering Virtual Center for Availability

Average VCP salaries for job postings nationwide are 261% higher than average A+ salaries for job postings nationwide.

This post brought to you by VMwarewolf

VCP Equals a lot of CASH

Related posts:

1. Common Licensing issues in VMware Infrastructure
2. Common Network issues in VMware Infrastructure
3. Common system management issues in VMware Infrastructure
4. VMware HA Admission Control
5. VMotion in 3.5 DRS enabled Cluster causes Guest CPU to rise Dramatically

Dear VMware Customers,

We have released the express patches for the product expiration issue.  Please go to http://www.vmware.com/go/esxexpresspatches for download and KB articles.  Since our last update we have completed our verification tests that the express patches we’ve released are fully compatible with the VMware Update Manager.  Please see the KB articles for deployment information regarding Update Manager.

The KB articles are kept up-to-date.  Please refer to the KB articles for information and updates.

Thank you,

The VMware ESX Product Team

This post brought to you by VMwarewolf

Problem with ESX 3.5 Update 2 and ESXi Update 2

Related posts:

1. ESX 3.5 Update 2 bug
2. ESX Patch Management
3. VI3.5 Update 2 Release Notification
4. Host won’t go into Maintenance Mode
5. Network copy failed for file nvram

Unable to Power On virtual machine with “A General System error occurred: Internal error”

A virtual machine fails to power on, fails to leave suspend mode, or fails to migrate with VMotion and the following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
————–
Module License Power on failed.

Currently running virtual machines will continue to run but will not be able to perform migration with VMotion, suspend or power on virtual machines.

An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13.  These will be available from the page: http://www.vmware.com/download/vi.  Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2.

The Update patch bundles will be released separately later in the week.

This KB article will be updated as soon as more information is available, check back frequently for updates and additions.

A workaround to this ESX bug is to set the date back a day on your ESX hosts and you will be able to power on the VM’s again. If you use ntp, disable that for the time being and manually set your hosts back a day.  Be careful, if you have your VMware tools configured to sync VM time to the host time then this will alter your VM times.

***UPDATE***

Here is the latest public news from VMware on this>>

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully.  This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2.  The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:
-    VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2 (pre-Update 2 releases are not impacted by this problem).
-    Reports of problems with ESX 3.5 U1 with the following patches applied.
1.    ESX350-200806201-UG
2.    ESX350-200806202-UG
3.    ESX350-200806217-UG

-    No other VMware products are affected.

What has been done?:

-    Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
-    VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
-    A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs.  A new static page has been  published at http://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.
-    The phone system has been updated to advise customers of the problem
-    Vmware partners have been notified of the issue.

Workarounds:
1)    Do not install ESX 3.5 U2 if it has been downloaded from VMware’s website or elsewhere prior to August 12, 2008.
2)    Set the host time to a date prior to August 12, 2008.  This workaround has a number of very serious side affects that could impact product environments.  Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken.  These include, but are not limited to database servers, mail servers, & domain administration systems.

Next Steps:
VMware to send an email to all customers who have downloaded this version. This effort is underway and should happen before 11 am today.

Resolution:

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today.  The target timeframe is 6pm, August 12, 2008 PST.

FAQ:

1. What would this express patch do?

More information will be provided in subsequent communication updates.

2.   Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?

Yes.  We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week.  We will provide an ETA for the update patch bundles subsequently.  NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA.  They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.

3.   Why does VMware plan to reissue the upgrade media before the patch bundles?  That is a wrong priority call!

This is not a matter of priority.  Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.

4.   Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

5.   Does this issue affect VC 2.5 Update 2?

No.

6.   What is VMware doing to make sure that the problem won’t happen again?

We are making improvements on all fronts.  The product team had endeavored to deliver a release with support customers deem important.  But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused.  We have identified where the holes are and they will be addressed to restore customers’ confidence.

This post brought to you by VMwarewolf

ESX 3.5 Update 2 bug

Related posts:

1. VMotion in 3.5 DRS enabled Cluster causes Guest CPU to rise Dramatically
2. Failed to install the VirtualCenter Agent Service
3. Forgotten Snapshot Locator
4. Daylight-Saving Time Extension
5. VMware support for SAP

Some customer are reporting the message:

HA agent on <esxhostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Network:
Consider using the Advanced Cluster Settings das.allowNetwork to control network usage.

VMware has just published two Knowledgebase articles for this issue.  KB 1006606 covers the network compliance check, which will also link to KB 1006541 covering how to address the issue.  

I am also told VMware is working to have this included in the Update 2’s release notes.

This post brought to you by VMwarewolf

Incompatible HA Network

Related posts:

1. HA problem with Virtualcenter 2.5 Update 2
2. Network copy failed for file nvram
3. Common Network issues in VMware Infrastructure
4. Failed to initialize not well-formed invalid token
5. How does the VMware HA work

Common reports include:

“insufficient resources to satisfy HA failover level on cluster” and “unable to contact a primary HA agent in cluster”

We think we know what is causing this, so I’m sharing this here in hopes of saving you a call.

The key is that all of your hostnames are in lower case only.

So, in Virtualcenter check-
-each host in the cluster, go to the configuration tab, DNS and routing, hostname
-everything needs to be lowercase
On each one of your ESX hosts-

ssh into each host and check-
-lower case names only in /etc/hosts
lowercase names only in /etc/sysconfig/network
cat /proc/sys/kernel/hostname

if not correct :

hostname <FQDN>

after this you can check the settings are correct using-

uname -n
hostname -a
hostname -s
hostname -d
hostname -f
hostname

in addition to this also make sure hostname in /etc/vmware/esx.conf is lowercase-

/adv/Misc/HostName = <FQDN in lowercase>
when done enable HA on cluster again.

Check this link if you are seeing the HA incompatible network message.

This post brought to you by VMwarewolf

HA problem with Virtualcenter 2.5 Update 2

Related posts:

1. Incompatible HA Network
2. Network copy failed for file nvram
3. HA Failover Capacity
4. How does the VMware HA work
5. How-to Keep Virtual Machines Mutually Exclusive

Release Notes links:

VMware ESX/ESXi Server 35 Update 2 http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html

ESX Server 3i version 3.5 Update 2 Installable http://www.vmware.com/support/vi3/doc/vi3_esx3i_i_35u2_vc25u2_rel_notes.html

ESX Server 3i version 3.5 Update 2 Installable Embedded http://www.vmware.com/support/vi3/doc/vi3_esx3i_e_35u2_vc25u2_rel_notes.html

VMware Virtual Center 25 Update 2 http://www.vmware.com/support/vi3/doc/vi3_esx35u2_vc25u2_rel_notes.html

Update Manager Update 2 http://www.vmware.com/support/vi3/doc/vi3_vum_10u2_rel_notes.html

Converter Enterprise Update 2 http://www.vmware.com/support/vi3/doc/vi3_vec_10u2_rel_notes.html

Build Numbers

• VMware ESX/ESXi Server 35 Update 2 Build 103908
• ESX Server 3i version 3.5 Update 2 Installable Embedded Build 103909
• ESX Server 3i version 3.5 Update 2 Installable Build 103909 VMware Infrastructure Management Installer Build 104263 (EN)
• VirtualCenter Server 2.5 Update 2 Build 104215 (EN)
• VMware Infrastructure Management Installer Build 104265 (G11n)
• VirtualCenter Server 2.5 Update 2 Build 104217 (G11n)
• VMware Infrastructure Client Build 104249 (G11n)
• VMware Consolidated Backup 1.5 Build 102898
• Update Manager Update 2 Build 103284
• Converter Enterprise Update 2 Build 102422 (EN)
• Converter Enterprise Update 2 Build 102445 (G11n)

This post brought to you by VMwarewolf

VI3.5 Update 2 Release Notification

Related posts:

1. VMware Infrastructure 3 Release Notes part deux
2. Host won’t go into Maintenance Mode
3. ESX 3.5 Update 2 bug
4. VMware VirtualCenter 2.0.2 Released
5. Problem with ESX 3.5 Update 2 and ESXi Update 2

The fix for this is simple.  You need to grab the new deployment tools from Microsoft’s site here also referred to as the sysprep files. You will be downloading WindowsServer2003-KB892778-SP1-DeployTools-x86-ENU.cab

This file needs to be extracted on the Virtualcenter server and placed in the following folder-

C:\Documents and Settings\All Users\Application Data\Vmware\VirtualCenter\sysprep\svr2003

Once these files exist in the correct folder, you will be able to deploy your W2K3 template.

For more details read the Virtual Center 2 Template Usages and Best Practices http://www.vmware.com/pdf/vc_2_templates_usage_best_practices_wp.pdf

This post brought to you by VMwarewolf

Customization fails after VC 2.5 Upgrade

Related posts:

1. Guest customization fails on VirtualCenter 2.5 Update 2
2. VirtualCenter 2.5 Upgrade Database Issues
3. Upgrade to ESX 3.5 and VirtualCenter 2.5
4. VMotion fails at 10 percent
5. VirtualCenter Best Practices

Upgrading to ESX 3.5 and VirtualCenter 2.5 Best Practices

This document provides valuable steps which may be useful when upgrading to ESX 3.5 and VirtualCenter 2.5. You also want to read the Upgrade Guide. Do not skip.

This post brought to you by VMwarewolf

Upgrade to ESX 3.5 and VirtualCenter 2.5

Related posts:

1. VirtualCenter Best Practices
2. VirtualCenter 2.5 Upgrade Database Issues
3. Customization fails after VC 2.5 Upgrade
4. VMware VirtualCenter 2.0.2 Released
5. ESX Disconnected after Upgrading to VirtualCenter 2.0.1

VMware Video Vault

VMware Fusion 2 Retail (for running Windows on a Mac) US $33.00 (7 Bids) Auction Ends: Saturday Nov-22-2008 10:24:27 PST Bid on this Item   | Watch this Item

VMWARE FUSION VIRTUALIZATION SOFTWARE WINDOWS ON MAC US $1.00 (1 Bid) Auction Ends: Saturday Nov-22-2008 10:38:13 PST Bid on this Item   | Watch this Item

NEW HP VMware ESX Server Starter - License 430339-B21 US $1,530.99 Auction Ends: Saturday Nov-22-2008 12:17:33 PST Bid on this Item   | Buy this Item   | Watch this Item

NEW EMC VMware Workstation v.6.0 WS6-ENG-L-CP US $195.99 Auction Ends: Saturday Nov-22-2008 20:08:12 PST Bid on this Item   | Buy this Item   | Watch this Item

NEW EMC VMware Workstation v.6.0 WS6-ENG-W-CP US $194.98 Auction Ends: Saturday Nov-22-2008 20:08:12 PST Bid on this Item   | Buy this Item   | Watch this Item

This post brought to you by VMwarewolf

VMware Videos

Related posts:

1. New VMware Videos
2. VMware Knowledge needs to be made Findable
3. VMware Workstation 6 Download
4. Which ESX log file
5. Common Licensing issues in VMware Infrastructure