Tuesday, March 17, 2009

Fracking KVM

I started 3 ESX v3.5 ESX servers and followed two of them through the boot boot sequence via a KVM.

They both hung at
starting console mouse services...
The Virtual Center server showed them as uncontactable.

Aaarrgghh!!

Luckily that is after the ssh service is started so I was able to ssh in and
service gpm stop
cd /etc/rc3.d
and for each service after S85gpm I started the service by hand. At which point they were visible in Virtual center Server. Hurrah!

Post Mortem:
  • Why only two of the three?
  • Was it the version of ESX?
  • What was the commonality?

Well, it wasn't the version of ESX. The two ESX servers which hung were running different builds of v3.5!

The common feature was that I had monitored the boot of the servers via the KVM. The server that hadn't been monitored had booted without issue. So I ensured the KVM was connected to a Windows system and from Virtual Center Server, I issued reboots to the two ESX servers which had hung and went outside for ten minutes to bite my fingernails. I needed a clean reboot anyway.

When I came back, the servers had rebooted successfully.

So the problem had been the KVM system we have.

Perhaps I should not have been surprised. Lets just say it is rumoured at work that when Noah installed a KVM on the Ark, he dismissed the system we've got as too antiquated!

So it goes!

No comments: