So this is my first “official” post. How exciting!!!
Anyways, I wanted to write about an issue I recently encountered. At work, we recently deployed VMware ESX 3.5 with VirtualCenter 2.5. We hired a consulting company to deploy this for us, but after 2 whole days, it still was not working properly. We were specifically having issues with the virtual networking piece of the deployment. We wanted to deploy the virtual networking piece while leveraging VLAN tagging, but this was not working since we were seeing over 50% packet loss.
One would think that the consultant would have worked with us to fix this since we were paying big money for him to be onsite and deploy this, but no. All he could do was stare into the monitor and say “I don’t know what is wrong.” After me just getting completely frustrated at the fact that he was not helping at all, we just told him to leave and that I would research the issue.
I began be first completely redeploying the entire environment since I no longer trusted this consultants work. First I reinstalled VMware ESX 3.5 on all 3 of our servers. Once that was complete, I reinstalled VMware VirtualCenter with a fresh new database. I made the necessary vSwitch (virtual switch) changes, but was still having greater than 50% packet loss. After searching the VMware Knowledge Base and VMware Communities and not finding much, I decided to call up VMware support.
After being on the phone all of 1 minute and explaining our issue, I was pointed to KB #1003660 “Virtual switch tagging does not function when using Intel VT Quad Port Adapter“. So I proceeded to installing patch ESX350-200802412-BG using Update Manager. After installing and rebooting the servers, everything was working perfectly! No packet loss, and VMotion was working as advertised. Now we were ready to start deploying VM’s…except that once we deployed our first VM, we got VMwares PSOD (Pink Screen of Death).
After more research, I found a posting in the VMware Communities with the same PSOD message. Basically he was having the same issue, but on a SuperMicro server. We were using Dell PowerEdge 2950’s, but both of these servers had one thing in common, the Intel Xeon 5420 processor. The user who opened the thread was directed to VMware KB #1003733 “BIOS update may be required for Xeon 5400 and Xeon 5200“. So I went to Dell’s support site and downloaded the latest BIOS update for the PowerEdge 2950 servers and updated the BIOS on all 3 ESX server using a boot disk.
I proceeded with caution by only deploying 1 VM and monitoring for a few hours, since without the BIOS update the server would crash within 2-4 hours. After more than 6 hours and no crashes, I proceeded to deploy VM’s on all 3 servers and decided to monitor them over the weekend. When I came back Monday morning, I was glad that no crashes had occurred and had finally completed the deployment of our VMware Infrastructure.
The moral of this posting: Never think that just because you are paying a consultant a lot of money, that they are any good. The first thing that the consultant should have done was at a very minimum install all critical patches. Sheesh!
