Friday, September 4, 2009

SQL Failure After Upgrade from VMWare 3.5 to VSphere 4 Fix


I ran into a problem with our Microsoft ISA Server after moving to VSPhere 4 today. After searching the 'net for a solution I found a lot of painful solutions, most of them had no effect or made matters worse.

I have finally tracked the issue down and corrected it myself. It's a 5 min. fix. If you want to skip all my jabbering and just get to the solution, check the section marked Solution below. Fair warning that the Problem and History section contains information you need to know to confirm that this is indeed the correct solution for you.

In this case the victim was an ISA server, but any SQL server or VM client that uses Msvcr71.dll could be affected.

Problem and History

After upgrading my ESXi server from version 3.5 to 4.X to allow for management via VSphere 4 I noticed external users were unable to connect to my reverse proxy server.

I logged into the ISA server via the VSphere manager as I was unable to connect via RDP (If you are running a SQL only server you will not have this problem). When I opened the ISA manager (Start-->All Programs --> Microsoft ISA Server --> ISA Server Management) I noticed that Monitoring showed both the Firewall and MSDE services were down (The Monitor can be accessed by Clicking Monitor on the left hand menu, and selecting the Dashboard tab). I then attempted to restart the services by clicking the Services tab, right-clicking on the Microsoft Data Engine and selecting Start. I repeated this for the Microsoft Firewall. Both times ISA reported that it was unable to start the service.

The Database instance for the firewall (not to be mistaken for the Microsoft Firewall Service) is MSSQL$MSFW. It can be manually started in the Services Administrative Tool (Start -->All Programs -->Administrative Tools --> Services) by right clicking on the MSSQL$MSFW service and selecting Start. In this case I got error events 7009 (Timeout waiting for the MSSQL$MSFW service to connect) and 7000 (The service did not respond to the start or control request in a timely fashion). OK, great.

I needed to know If SQL was actually still viable, so I ran sqlservr.exe (C:\Program Files\Microsoft SQL Server\MSSQL$MSFW\Binn\sqlserver.exe)* and braced for the excitement. I finally got something useful: The message "This application has failed to start because MSVCP71.dll was not found. Re-installing the application may fix this problem". I did not really feel much like taking half a day to reinstall and configure so I searched the server for msvcp71.dll. There were a couple of them with the added extension .F26FFD4A_Yadayada which probably would have worked if I copied them to %windir%\system32 and renamed them, but nothing named msvcp71.dll. I powered up the old ESXi 3.5 VM (I ALWAYS backup before an upgrade, and you should too) and sure enough msvcp71.dll existed in both the %windir%\system32 and more tellingly C:\Program Files\VMWare\VMWare Tools. Apparently the 3.5 and older tools use msvcp71.dll, and 4.0 does not. When I upgraded the VMWare tools the 3.5 tools are uninstalled along with msvcp71.dll from both the VMWare Tools directory and System32. VMWare tools 4.0 does not require msvcp71.dll, so the .dll is never returned.. Very very naughty.

Solution

Easy peasy mac and cheesy: Download msvcp71.dll from dll files at http://www.dll-files.com/dllindex/dll-files.shtml?msvcp71 or copy it from your (better have) backed up VM. Copy msvcp71.dll from wherever you unzipped the file to %windir%\system32. Reboot the server. Prayer may help here. Confirm the correct operation of SQL. Go get a beer.

*Yes bin is actually "binn" in the path

23 comments:

Ben said...

had the unfortunate situation of having the exact same issue. Did the vSphere upgrade this weekend, suddenly ISA 2006 isn't work, narrowed it down to sql not starting, did the manual start found the same dll was not there and a quick google led me to your page. Fix worked great, thanks!

Ben said...

meant to add that some of our VMs running symantec antivirus 10.0 had a similar message about msvcp71.dll missing after the tools update (in our case we were upgrading to the newer ver 11 but thought it worth mentioning) for the googlers.

Retr0Rob said...

Glad it helped!
It's really kind of strange that VMWare was so careless about this.

Paolo said...

Thank you very much ... same issue for me. Your solution work perfectly!

Anonymous said...

Thank you very much ... we have the same problem. Your solution work perfectly!

Retr0Rob said...

Im so happy that these things don't only happen to me!

Anonymous said...

Thank you very much. I called VMWare support and they did not know of this so they referred me back to call Microsoft. Your post help me.

Unknown said...

I host Exchange for a lot of users. This post just saved me a very long night. I got lucky and found it within 5 minutes of my ISA heart attack.

Anonymous said...

You Rock! Easy Fix.

Pli said...

Just upgraded vmware tools for our ISA in front of our Exchange environment for 1500 Users.

I was in agony for 10 minutes until I found your post!

Thanks!

Joe said...

Your solution clearly documents and solves the issue we had starting a SQL server in a vSphere VM. This was a critical system of ours. Thank you !!

habibalby said...

Thanx for the great blog, it's really helped me solve a nightmare :)

Thanks once again.

Habibalby

Anonymous said...

People like you make this world a better place. Midnight on a Friday doing the vSphere upgrade on my ISA VM and it died... google found in and 5-minutes later I'm up and running. Seriously, I'll shake your hand in heaven some day. :)

Cossie said...

Thank you for this post, it saved my evening :-)

For the googlers i would add that our HelpDesk software Track-IT crashes as well.

Again thank you for the post.

Retr0Rob said...

LOL, I never could keep Trackit going anyway! I sure am glad the post helped!

Unknown said...

You're cool, Rob. Thanks.

Anonymous said...

Thank you!! Been searching the whole day why the SQL servers didn´t start.. Earlier I updated the servers @ MS update and thought it was that update who broke the servers.. Now i know better. Weve just upgraded to Vsphere from 3.5.. Why is it always something like this hehe.
Thanks again! Im a happy man now :)

Matt Lavery said...

Great post, had this issue just today and so I had the developers screaming down my neck to get it resolved. Your blog saved the day

Anonymous said...

You save my night!. thank you

Jose Angel Rivera said...

Thanks man!!! You saved me too. This time the problem arised because of a sudden blackout and all the servers were turned off the hard way. Copied the .dll that you found and now we are ok!

Thanks...there is a beer for you!

Unknown said...

Cool, thanks. It's still a valid fix after all these years--after upgrading VMware Tools (vSphere 5). I found a copy under a symantec liveupdate folder and threw it into system32 and voila.

Anonymous said...

Dude, I worship you. Grab a six pack! You saved me from a long long night!

RyanAK said...

Thank you, Retrorob!!! I saw the error and somehow assumed this was a bigger deal than it was and reverted to the pre-vmware tools snapshot. You're right. Easy Peezy. also thank you for the dll-files.com linkage. what a handy site! I'm going to go have a beer now :)
*PS pls don't ask why im upgrading away from vmware tools 3.5 just now :P