Thoughts on vCenter Server Remediation for Heartbleed bug – vCenter Server with Heartbeat

KB 2076692  documents the vCenter Server upgrade and remediation to address the Heartbleed vulnerability. VMware released 2 updates in vCenter 5.5 which are 5.5.0c (for those in 5.5 GA or pre U1) and 5.5 U1a (for those in 5.5U1). If you read the KB article carefully, updating to those versions is just the first step, you still need to re-issue the certificate for the VMware Directory Service, and change the password for the Administrator@vsphere.local account. Quoting the KB:

“After the vCenter Server environment is upgraded, the Single Sign-On component requires the SSL certificate for the VMware Directory Service to be re-issued and the administrator@vsphere.local password to be changed. Any other vsphere.local users that have been defined will also require their passwords to be changed.

Failure to carry out these actions continues to expose the system to compromise from the OpenSSL Heartbleed vulnerability.”

The steps for re-issuing the certificate is also documented in KB 2076692

So how do we perform this if vCenter Heartbeat is protecting your vCenter Server? In my past articles regarding vCenter Server upgrade, I documented the steps from KB 207181, the steps which i called a re-clone method. I believe that using this method is the easiest way to upgrade your vCenter Server and remediate for the heartbleed bug.

You just have to add the remediation steps for Heartbleed (re-issue the certificate and update the administrator@vsphere.local password) after Step 6 under Upgrading the Primary Node of KB 207181. Just make sure you are using the vCenter Server’s public FQDN and IP address when generating a new certificate.

Advertisements

Upgrading vCenter 5.5 to 5.5 U1 with vCenter Heartbeat installed [Part Three]

This is the last part of this 3 part series. In Parts One and Two, I discussed about this new “re-cloning” method for upgrading vCenter Server with heartbeat and started configuring the Primary vCenter server. In this part we will continue the configuration focusing on the Secondary vCenter Server.

1. Start the newly cloned vCenter Server. Make sure that both Principal and Channel NIC’s are disconnected before powering ON the VM. Also make sure that the vCenter Server Heartbeat service is stopped and set to Manual.

2. Reconfigure the IP address of the  Channel Nic from 1.1.2.1 to 1.1.2.2

3. Launch the vCenter Server Heartbeat Configure Server wizard and click the Machine tab. Change the physical server identity to Secondary.
vc9

4. Click the Public tab, configure the correct IP addresses, and click Finish. In my lab, I am changing the Management IP back to 192.168.11.108vc8

5. Change the server name to the Secondary node’s previous Management name and add it to a Workgroup. Reboot when requested. In the screenshot below, I am changing the hostname back to vc02.vc10

6. Review the Principal NIC’s IP Address before connecting the Principal NIC. This is to make sure we won’t have an IP conflict. The Channel NIC can also be connected.

vc11

7. Using the Service Control Manager, configure the VMware vCenter Server Heartbeat service Startup Type to Automatic. DO NOT Start the service just yet!

8. Join the server to the domain and reboot when requested. So now I am joining my vc02 server from WORKGROUP back to the lab.local domain.

vc12

9. When the secondary comes back up, we have basically brought up the 2 vCenter Nodes back to its original states but now upgraded to Update 1. Initiate Full replication and synchronization from Primary to Secondary

vc13           vc14

10. Uninstall and re-install the vCenter plugin.Navigate to the Applications: Plug-ins page, select the VirtualCenterNFPlugin and click Uninstall.

11. After the plug-in has been removed from the plug-ins list, click Install. Navigate to the location of the vCenter Server plug-in (the default location is C:\Program Files\VMware\VMware vCenter Server Heartbeat\R2\plugins\VMwareVirtualCenter\201.5.4.13406) and select the VirtualCenterNFPlugin.dll file.

vc15vc16

12. After the plug-in has been added, select it, click Edit, then enter the credentials used to connect to vCenter Server.

vc17

That’s it folks!! To cover all grounds, do a failover test to check that everything works just fine.

Upgrading vCenter 5.5 to 5.5 U1 with vCenter Heartbeat installed [Part Two]

In Part One , I mentioned about KB2070181 that demonstrates a different method of upgrading vCenter Server when Heartbeat is installed. In this article, I will show the steps I did to perform this task on the Primary vCenter Server. Details of my environment are as follows:

Primary vCenter Server:
Hostname: vc01.lab.local
Management IP address: 192.168.11.107/24
Channel IP Address: 1.1.2.2/24

Secondary vCenter Server:
Hostname: vc02.lab.local
Management IP address: 192.168.11.108/24
Channel IP Address: 1.1.2.2/24

vCenter Server Public Name: vc.lab.local
vCenter Server Public IP Address: 192.168.11.106

 

Both vCenter Servers are VM’s in a LAN deployment. The SQL Server database in my environment is in a different VM and is not protected by Heartbeat. The version of Heartbeat is 6.6 which is a supported version from  the KB. In a production environment, always make sure you have a backup.  Here are the steps:

1. Make the Primary Node active and verify.

vc1

2. To verify, double click the Configure Server icon on the desktop and go to machine tab

vc2

3.  Shutdown the secondary node. The KB article mentioned to delete the secondary vCenter Server but I won’t do that just yet. I did a snapshot of my Primary VM and in case something goes wrong I can just revert to the snapshot and still use both the primay and secondary vCenters.

4. Rename the Primary node from the Management Name to the Public Service Name and reboot when requested.  In below screenshot, I renamed my primary vCenter from vc01 to vc.

vc4

5. Shut down vCenter Server Heartbeat, leaving protected applications running.

6. Using the Service Control Manager, configure the VMware vCenter Server Heartbeat service Startup Type to Manual.

vc5

7. Upgrade all protected applications. In my environment, I upgraded the following components to 5.5 U1 in the following order: SSO, Web Client, Inventory Service and vCenter Server. If you have Update Manager, upgrade it as well. The upgrade went smoothly, the installer detected an existing installation and prompted that they will be upgraded to 5.5 U1 version.

8. Start vCenter Server Heartbeat and run protected service discovery by navigating to the vCenter Server Heartbeat Console Application tasks page.

vc6

8. Clone the server (do not start the newly cloned node yet)

9. Using the Service Control Manager, configure the VMware vCenter Server Heartbeat service Startup Type to Automatic.

10. Rename the server back to the Primary node’s previous Management Name, and reboot when requested.

vc7

That’s it for the Primary vCenter Server. In part 3 of this series, I will continue on with configuring the Secondary node which actually is just the clone of the newly upgraded primary vCenter Server.

Upgrading vCenter 5.5 to 5.5 U1 with vCenter Heartbeat installed [Part One]

vCenter Server heartbeat is a great product in environment’s that needs a highly available vCenter Server wherein HA is not enough as a means of protection. However, with the added resiliency of Heartbeat, it also introduces complexity in managing vCenter and mostly, upgrading your vCenter Server. There are a couple of articles and walkthroughs published by VMware to assist in this task. Some of the articles are as follows:

http://vmwarewalkthroughs.com

https://blogs.vmware.com/vsphere/2013/11/upgrading-vcenter-when-using-vcenter-heartbeat-part-2.html

The procedures for both articles above entails upgrading both the Primary and Secondary nodes one after the other which is a bit time consuming.

While  browsing our intenal Socialcast, I came to find out about another method of upgrading vCenter Server with Heartbeat which is published in KB 2070181. The article intrigued me as it does not require you to upgrade the vCenter components twice, but will rely on the following high level steps:

1. Rename the Primary vCenter back to the Public Name
2. Shutdown and delete the Secondary vCenter Server
3. Upgrade the vCenter Server components on the Primary Server
4. Clone the Primary Server to be used as a new Secondary Server
5. Reconfigure the Secondary Server’s hostname and IP Address for Heartbeat

I managed to test it out in my lab and found it to be a very good alternative method.  I will detail the steps I did in upgrading my 5.5 GA vCenter Server with Heartbeat to the latest 5.5 U1 using this KB article in my next blog post. I will also highlight some caveats when using this method.

 

 

 

 

Manually Changing the Packet Filter State in vCenter Heartbeat

In a working state of vCenter Heartbeat, the packet filter is the mechanism that exposes the public IP address on the Active vCenter Server and hides it on the Passive vCenter server. Using the nfpktfltr.exe getstate command, you can tell the state of the packet filter:

Active Server:

active

Passive Server:

passive

I encountered a scenario wherein a split-brain was encountered in the environment. In a split-brain situation, both the primary and secondary vCenters think they are the “Active” server.  vCenter Heartbeat was able to detect split-brain and it effectively shutdown the Heartbeat service on the secondary node.

I performed the same command to check the packet filter status before bringing up the secondary vCenter’s heartbeat services but saw the state on the Secondary as “Passthru”, meaning it will be exposed in the network (analogous to being Active).

Digging around the nfpktfltr.exe command switches, there is an option to set the packet filter state. The command is nfpktfltr.exe setstate.

switches

For the Secondary server I used nfpktfltr.exe setstate filter command to manually change the state and that did the trick.

Shutting Down vCenter Server with vCenter Heartbeat does not trigger a Failover!

I got this question from a customer regarding vCenter Server behavior with vCenter Heartbeat installed. Will an automatic failover be triggered if you shutdown the Active vCenter Server? My initial answer is Yes it should but the customer told me it did not failover when they shutdown their active vCenter server. I quickly checked the documentations and the closest I found is this:

Shutting Down Windows

Always stop vCenter Server Heartbeat before attempting to shut down Microsoft Windows. If an attempt is made to shut down Windows without stopping vCenter Server Heartbeat, a confirmation message is displayed. When the Windows Shutdown confirmation message is displayed, click Cancel and stop vCenter Server Heartbeat before attempting Windows shut down again. 

There was no mention of a failover. Out of curiousity, I tested it in my lab, and boy the customer was right! A failover will not be triggered. The documentations said a confirmation message will be displayed but I did not see it, the server just shuts down normally and no failover took place.

So make sure to do a manual failover before you shutdown the active vCenter Server or else you will be on for an extended downtime.

So when will an automatic failover be triggered?

1. System crash or hardware failure – i simulated it by doing a hard poweroff of the Primary vCenter server and it worked flawlessly

2. Network Failure – simulated this by disconnecting both the Management and Channel Nics.

3. Protected service failure – If the vCenter and other protected services stopped, an attempt will be performed to restart them in the same node, but if it fails again then a failover will take place.

vCenter Heartbeat saga continued

It’s almost christmas and yet I am still in a tough fight getting vCenter Heartbeat running on my current project. The latest hurdle is with the installation itself. For some strange reason, the install just hangs and will not proceed to completion. No errors are being thrown. It’s just stucked in this state.

Screen_Shot_2013-12-13_at_11.00.19_AM

Observing the system, it looks like vCenter Heartbeat installed fine. It’s just the Packet Filter that is missing. In vCenter Heartbeat, a packet filter driver is installed which controls who owns the heartbeat public IP address. It basically blocks the passive node effectively hiding it from the network through the configured Public IP.

Listing down a few of the troubleshooting steps to attempt to resolve this issue:

– Disabled UAC
– Ensure to use Run As Administrator when doing the install
– Tried both domain admin and local admin account
– Reinstalled a few times using different sets of installer

I finally decided to ring our good old reliable GSS team as I don’t want this to affect my project timeline. After an hour and a half of peeking through log files and server settings, we came to a conclusion that there is no issue with Heartbeat and its either Windows or another software installed on the server.  A few isolation methods and we found the culprit. Somehow the Antivirus software (Mcafee) is preventing successful installation. Disabling the Mcafee services during installation did the trick.

And so goes my Heartbeat adventure. I was able to complete both primary and secondary node and have successfully initiated failover and failback tests. Happy customer means a happy consultant 🙂