Showing posts with label ESX. Show all posts
Showing posts with label ESX. Show all posts

Sunday, March 29, 2009

VMware & Changes on the PCI bus

All the ESX servers at work are IBMs. In fact most all the x86 servers in the datacentre are IBMs. Heck, even quite a lot of our training is sourced through IBM.

I thought I had blogged about vmnics changing ID/name earlier on in the same manner as on this blog from SysAdminHell, but must have been mistaken. One of my first (bad) experiences with ESX was having to go in and undertake the same procedures as in that blog.

That blog entry only suggests that moving the physical NIC card causes the problem. My experience is that any change on the PCI bus can cause the vmnics to be renamed.

Perhaps that is only a problem with IBMs. Or perhaps not!

Tuesday, March 17, 2009

Fracking KVM

I started 3 ESX v3.5 ESX servers and followed two of them through the boot boot sequence via a KVM.

They both hung at
starting console mouse services...
The Virtual Center server showed them as uncontactable.

Aaarrgghh!!

Luckily that is after the ssh service is started so I was able to ssh in and
service gpm stop
cd /etc/rc3.d
and for each service after S85gpm I started the service by hand. At which point they were visible in Virtual center Server. Hurrah!

Post Mortem:
  • Why only two of the three?
  • Was it the version of ESX?
  • What was the commonality?

Well, it wasn't the version of ESX. The two ESX servers which hung were running different builds of v3.5!

The common feature was that I had monitored the boot of the servers via the KVM. The server that hadn't been monitored had booted without issue. So I ensured the KVM was connected to a Windows system and from Virtual Center Server, I issued reboots to the two ESX servers which had hung and went outside for ten minutes to bite my fingernails. I needed a clean reboot anyway.

When I came back, the servers had rebooted successfully.

So the problem had been the KVM system we have.

Perhaps I should not have been surprised. Lets just say it is rumoured at work that when Noah installed a KVM on the Ark, he dismissed the system we've got as too antiquated!

So it goes!

Enable ssh on ESXi

I was about to write a quick blog entry on how to enable ssh access to an ESXi server, but I did a quick google around and there are already a number of blog entries out there, including on:
and many others. It is such a frequently asked question that VMware should enable ssh by default.

What is interesting, to me anyway, is that this search brought two new (to me) vm help sites. Although at work, we always use hw on the HCL I realise some people do not have that luxury.

And that's that!

Monday, February 16, 2009

Useful VMware KB articles

Researching the background to a problem we've just suffered for the second time in three weeks on one of our ESX servers I came across a couple of extremely useful general articles in the VMware KnowledgeBase.

The first was called Investigating hosted virtual machine resources and the second was called Verifying the health of an operating system

A lot of common sense, but how often has it been said, that common sense is in extremely short supply!

RVTools #4 - version 2.3.1

I have downloaded the latest version of RVTools - v2.3.1. Its just a minor bugfix version.

I have only two minor criticisms.

The requirement to uninstall the previous version before installing the new version is annoying, but at least there is no requirement to reboot. Uninstall uses up a couple of minutes. It would be so nice if a graceful upgrade could be managed.

We have quite a few VMs. If you have scrolled to the bottom of the vInfo Tab to view the information for a VM and then want to check the information for that VM displayed on the vCPU Tab. Well, on selection of the vCPU Tab, the list automatically reverts to the top of the list.

I can think of instances when that is the behaviour you would like. I just suspect that more usually you will want to keep your relative position.

Of course, there are some Tabs when it is not possible or meaningful to maintain a relative position. If one of those were selected, then it is reasonable that the top of the list is selected.

Despite my quibbles, RVTools is still an excellent tool.

And that's that!

Tuesday, February 10, 2009

RVTools #3 - version 2.3

I spotted version 2.3 of the excellent RVTools had been released and was available for download.

As a major fanboy, obviously I immediately downloaded it.

There is an extra tab, vHost. I immediately went there and checked out the information displayed there. It was interesting and a bit scary how many different versions of ESX we're using. Especially as we were supposed to be harmonized on a single version. Well! Two versions: full blown ESX and ESXi.
Anyhow.

I checked out the vDatastore Tab.

And it crashed.


Oh!

Oh, pooh!

Now how do I let Rob know?

Monday, January 12, 2009

Modifying the Keyboard on Debian Linux

So I've downloaded the SyslogAppliance v0.0.4 to act as a Syslog Server in the company's DMZ. I ran the appliance initially in VM Player and then used the Enterprise Converter to import the appliance into a Virtual Machine on our VMwareESX server.

By default, the appliance is set up for a US audience with US keyboards. That's not a problem. After 20+ years in IT, you expect that and I know my way around a US keyboard, especially as I work for an American concern. Anyway, the ReadMe.txt file that's packaged with the appliance provides you with information on how to configure a UK keyboard.
...
VMWare Settings
- 256 MB memory (you should increase this for large log volumes)
- 64GB virtual disk
- listening on port 514/UDP and port 514/tcp
- US keyboard, to install a different local, do
$ apt-get install console-setup
$ dpkg-reconfigure console-setup
...

Obviously, despite both being Linux, it uses a different mechanism from that used by rPath in an earlier post on this blog. But two command lines isn't too painful.

appliance:/etc# apt-get install console-setup
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package console-setup
appliance:/etc#

Oh. That's unfortunate! Maybe this won't be so easy.

15 minutes of googling around apt-get and reading the more obvious looking results.

appliance:/etc# apt-get update
Get:1 http://security.debian.org lenny/updates Release.gpg [189B]
Ign http://security.debian.org lenny/updates/main Translation-en_US
Get:2 http://ftp.us.debian.org lenny Release.gpg [189B]
Get:3 http://security.debian.org lenny/updates Release [40.8kB]
Ign http://ftp.us.debian.org lenny/main Translation-en_US
Get:4 http://ftp.us.debian.org lenny Release [74.5kB]
Ign http://security.debian.org lenny/updates/main Packages/DiffIndex
Ign http://security.debian.org lenny/updates/main Sources/DiffIndex
Get:5 http://security.debian.org lenny/updates/main Packages [18.1kB]
Get:6 http://security.debian.org lenny/updates/main Source [5175B]
Get:7 http://ftp.us.debian.org lenny/main Packages [6942kB]
Get:8 http://ftp.us.debian.org lenny/main Source [2982kB]
Fetched 10.1MB in 23s (423kB/s)
Reading package lists... Done
appliance:/etc#
appliance:/etc# apt-get install console-setup
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
console-terminus xkb-data
The following NEW packages will be installed:
console-setup console-terminus xkb-data
0 upgraded, 3newly installed, 0 to remove and 70 not upgraded.
Need to get 1539kB of archives.
After this operation, 5968kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://ftp.us.debian.org lenny/main console-terminus 4.26-2.1 [420kB]
Get:2 http://ftp.us.debian.org lenny/main xkb-data 1.3-2 [645kB]
Get:3 http://ftp.us.debian.org lenny/main console-setup 1.28 [474kB]
Fetched 1539kB in 5s (264kB/s)
Preconfiguring packages ...
Selecting previously deselected package console-terminus.
(Reading database ... 18500 files and directories currently installed.)
Unpacking console-terminus (from .../console-terminus_4.26-2.1_all.deb) ...
Selecting previously deselected package xkb-data.
Unpacking xkb-data (from .../xkb-data_1.3-2_all.deb) ...
Selecting previously deselected package console_setup.
Unpacking console-setup (from .../console-setup_1.28_all.deb) ...
Setting up console-terminus (4.26-2.1) ...
Setting up xkb-data (1.3-2) ...
Setting up console-setup (1.28) ...
appliance:/etc#

Despite the quantity of output, it didn't take more than 10 to 20 seconds.

Now to configure the software.

appliance:/etc# dpkg-reconfigure console-setup
























And that's that!

Thursday, August 14, 2008

Error adding an ESX host to Virtual Center Server

A colleague set up a new ESX host we are going to host in our DMZ. He claimed to have followed the Work Instruction I created.

This was our first VMware Server in the DMZ. I was expecting some problems because of the requirement to punch some holes in the FireWall. The Server Configuration Guide is an excellent source of information for the details on how to manage an ESX server through a FireWall. It details the ports that need to be opened; the protocols those ports will be using; and the reasons why they need to be opened.

In the VI client attached to the VirtualCenter Server, I selected the DataCenter and selected Add Host from the menu. After resolving some FireWall problems, of which more in a later blog, I entered the server name, admin id and password, checked the returned information and clicked through the next three pages.

Only to receive the pop-up error message "Failed to install the VirtualCenter Agent Service"!

I googled online for other people's experiences. Most people encounter this sort of problem after an upgrade. In those instances, the problems occur when the VirtualCenter agent service on the ESX host hasn't been upgraded for some reason.

There were some suggestions that this could occur if /tmp/vmware-root doesn't exist. It did on my server.

Others suggest that a simple restart of the mgmt-vmware service would resolve the problem. It didn't on my server.

Others again suggested restarting the vmware-vpxa service. My server had no such service. Aha!
rpm -qa | grep -i vpxa
returned nothing.

A chap called Rene has a blog where he describes how to perform a manual upgrade process of the vpxa-vmware agent. Unfortunately, it is for an earlier version of VMware. The path referenced is slightly different on my VirtualCenter server. The version number of the file is distinctly different. The real scoop on manually upgrading/installing the vpxa-vmware service can be found in the this thread from the VMware Communities website.

N.B. The correct path on my VirtualCenter server is C:\program files\vmware\infrastructure\virtualcenter server\upgrade. Check the bundleversion.xml file for the correct file to copy across to server. For my VirtualCenter server v2.5 and ESX server v3.5 was vpx-upgrade-esx-7-linux-64192.

So I sftp-ed the file to the server. I ran the shell script. The service still wasn't installed! The rpm was still not installed! So I tried to install the rpm from the command line with rpm -ivh . It failed again, but at least provided me with a useful error.

The /opt partition did not have sufficient space left! Which was odd as it should be a 1Gb in size. I checked. It was 24 Mb. And already half used! Oh dear!

I created a /optn directory - there was plenty of space left on the / partition; tar-ed the contents of /opt to /optn; umount-ed /opt; renamed /optn to /opt; and commented out the /opt entry from the /etc/fstab file. I ran the install script again. Success! Huzzah!

Well! That's that!

Tuesday, August 5, 2008

Strange ESX Console Behaviour

I usually use a web interface to our KVM system. It can be a little bit clunky with ESX Server, especially since v3.5, but generally does the job.

A colleague built a new ESX server out in our DMZ and we were having some problems accessing it. The fireWall was being just a tad too restrictive. Strike that. It was being completely restrictive.

Anyhow, I was using the Web interface, but the ESX login seemed to be frozen. Alt-F1 & Alt-F11 worked, but nothing else.

I walked into the server room and used the console directly attached to the KVM system in there. Same problem!

I attached a keyboard and monitor directly to the server!! A complete PITA. Same Problem!
My colleague was being to get worried he'd built the server incorrectly.

I added an USB Keyboard to the system. Same Problem!

Whilst pondering what to do I randomly turned the Scroll, Num and Function Lock keys off and hit enter. Eureka!! It worked!!

After some experimenting, it turns out that the Scroll Lock was the problem. Which is a slight issue when double--click is the action that brings up the KVM server list!

Back at my desk I googled for ESX and Scroll Lock and there were two interesting posts. They both came from VMware's own community pages, with the more interesting being spot on! Although the other perhaps provides some explanation?!

Well! That's that!

Monday, July 14, 2008

Restore Vizioncore vRanger backups via the OS

The Development VMware ESX server at my company is an IBM 366 with a SCSI attached EXP400 external disk pack. This system arrived in the UK from a company site on the west coast of the US via a stop at the company HQ on the east coast of the US. To say that the hardware had been shunted from pillar to post would be a minor understatement.

We are using Vizioncore vRanger to back up the VMs on both this Development box and the Production VMware ESX server.

Just recently, the RAID 5 array on the EXP400 dropped 6 disks(out of 9)! Why it did this is a different story. Here I'll recount how we recovered from this.

There were a couple of VMs we had to get back online quickly. No problem, we had vRanger backing them up.

Ah! Now, well there was a problem. The backups were very successful, but for some reason we're still investigating, vRanger refused point blank to recover to the Production system.

However, we had access to the backup via the Windows OS. A quick google discovered this thread on the Vizioncore support site.

Restoring a VMWare machine from the '.tvzc' files of vRanger :
1.) Download FileZipper : http://www.vizioncore.com/Downloads/ProductSupport/vcbrestore.zip
2.) Download BSDTar: http://www.vizioncore.com/Downloads/ProductSupport/bsdtar.zip
3.) Install BSDTar (the zipfile contains an installable .exe)
4.) Extract the desired files :
FileZipper.exe -D -I "filename.tvzc" -O - | "c:\Program Files\GnuWin32\bin\bsdtar.exe" xzvf -
5.) Ensure file permissions are correct - I have cygwin installed on my PC, which can be invaluable!
6.) remove the .vzsnp extension from the end of files.
7) Within Virtual Center Server:
  • use Browse datastore on the Production Server's storage
  • Upload the restored files to the Production ESX server
  • select the .vmf file and Add to Inventory
8.) Start up the VM.

Bob's your parental sibling, of the usually male variety.


Thursday, July 10, 2008

phpBB installation

This is yet another case where my mileage hasn't actually varied. But I had to write about the phpBB installation as it is just so damn slick.

It must be roughly 6 or7 years ago when I first installed and configured a phpBB site on a Solaris 8 server with MySQL v3.23.42, Apache v1.3.26 and php v4.0.6. Even then the install was pretty good, although it left enough techie stuff to be done that you felt you'd undertaken a "real man's job"! Afterall, I had to compile the Apache, MySQL and PHP distributions.

In this case, I created:
  • a new VM on my development ESX server,
  • loaded up CentOS v5.2 configured as a Web & MySQL server
  • loaded up some additional php libraries
  • started the httpd & mysqld services
  • download and installed the latest phpMyAdmin
  • created a DB
  • created a DB User with appropriate priviledges
  • started and finished the phpBB configuration very quickly
The phpBB configuration is performed via a web page. It recognises the current state of the installation and just steps you through it. When it has acquired all the relevant configuration details, it creates the tables in the database and sets up the initial admin account. And then you are in a position to start using the system.

Bob's your parental sibling, usually of the male variety!


This does slightly simplify the process, but only in terms of creating/deploying a new machine. I had to add a new server into the company QIP (now called VitalQIP) system and push that out to DNS.

After deploying a server, there's also planning that has to be undertaken for administration, usage policies, backup and restoration for the system.

Tuesday, July 8, 2008

VMware Training

After 18 months of using ESX starting with v2.5.4 and upgrading through v3.0.2 to v3.5, where the only training I'd had was to watch the DVD training "Virtualize it With: VMware ESX Server 3.0" from the elias khnaser company, I've finally taken the VMware training course "VMware Infrastructure 3: Install and Configure".

After such a long time, was it worth it?

Absolutely!

Didn't I know most of it?

Yes, perhaps 85%. I must admit to gaining a certain satisfaction at realizing how much I already knew.

But then again its hard to really know DRS and HA from the manuals and training DVD when your environment is two sites each with a development (sort of - some of those VMs seem to be in production!) server and a single production server.So 4 ESX servers in all! I have more servers coming, so the timing made sense. I'll soon be running my production servers in a DRS cluster on each site. I'll be considering making them HA clusters too, although that isn't so clear cut.

The ability to discuss issues and ask questions in a class-room environment, where the answer can be "I do not know, but lets just try that..." and there is no fall out in terms of production system downtime, can be really useful.

Also you pick up the odd bit of wisdom such as that soak testing your memory beyond that performed by the bios startup is well worth the effort. Unless you are under-utilizing your server, ESX will exercise all your RAM in a way that most other OS simply will not. So a memory fault that might never have been discovered by another OS can be exposed in very short order. Much better to perform a thorough soak test before deployment, than have your VMs do it.


I suppose the other reason for undertaking the training is that it is a requirement for the VCP exam.

Ticksy, that!

Although in VMware's position I would have done the same.

I have taken the VCP mock exam of 20 questions and passed with 80%. I'm still really annoyed by the 4 questions that I answered incorrectly. In my defence two of the questions weren't particularly practical, but in the actual exam that won't cut it.

Monday, July 7, 2008

Vizioncore vRanger Configuration Take 2

You should be thinking about backup even before you start creating Virtual Machines. This is perhaps obvious. Although, it is still possible to defer that decision by using traditional "in VM" solutions.

One of the features of ESX is that you can have a display name for a VM in the GUI which bares no relation to the names of the files. Now the name for the files of the VMs are taken at the time of creation from the XXX form. By default this is also the display name used within the GUI. The display name can be changed later. To change the filenames used, requires that the VM is stopped and all the files and the directory used are modified. And modified correctly!

Now it is possible to use a wide range of characters in the name of a VM, e.g. this is legal:


Legal, but not sensible.

Whilst ESX has no problem with filenames with non-alphanumeric characters, both vRanger and VCB do. They will both fail to back up a VM with the name indicated. That may suggest something about how both utilities are architected, or perhaps the APIs they are utilizing. It doesn't matter. You have to deal with it.

When you create a VM, give it a sensible, simple but meaningful alphanumeric name. Afterwards you can choose rename from the right mouse button menu and change the display name to include whitespace and other characters.


Ensure that you have modified the System Resource Reservation parameters, which reserves resources for the backup process to utilize.


Foreach ESX Server:
On the Configuration-> Ststem Resource Reservation->Simple Tab, set
CPU : 1500Mhz
Memory : 800MB


The Simple setting equates to the host->system setting under the Advanced tab!

After changing these settings, it is necessary to reboot the ESX server, before they take effect. Consequently, if you can it is sensible to set this all up before you start serving Virtual Machines.

Then
  1. Verify that the ssh client service has been enabled on the ESX hosts to be backed up.
  2. Enter all ESX hosts into Ranger by IP or FQDN.
  3. Create a backup user on the ESX hosts. N.B. root ssh access is required for vmfs --> vmfs backups & restores.
  4. To verify correct configuration, it is recommended that initial attempts should be undertaken using Ranger's legacy mode.

Tuesday, July 1, 2008

CD-ROMs in VMware

In my experience, after you have added enough NICs to a VMware ESX machine to be useful, i.e. at least 6 to 8, you start to get into a situation whereby ESX is unable to identify a physical CD-ROM that might be attached.

At that point whenever you try to start a Virtual Machine which is configured to try to attach to the physical CD-ROM, it will take an inordinate time to boot. Essentially hanging on the way up. and even after the machine has fully booted. It seems as though the VM is only getting 5 to 10 seconds of CPU every 2 minutes or so.

This can be immensely frustrating.

I know. Before I worked out what was happening, I became quite impatient.

The fix is quite straitforward. Simply change the CD-ROM over to a client device. For often than not, that is the most useful setting. It is still straitforward to map an iso to the drive as well, should you need that.

Wednesday, June 18, 2008

Vizioncore vRanger configuration

Ensure that you have modified the System Resource Reservation parameters, which reserves resources for the backup process to utilize.


Foreach ESX Server:
On the Configuration-> Ststem Resource Reservation->Simple Tab, set
CPU : 1500Mhz
Memory : 800MB


The Simple setting equates to the host->system setting under the Advanced tab!

After changing these settings, it is necessary to reboot the ESX server, before they take effect. Consequently, it is a good idea to set this all up before you start serving Virtual Machines.

Then
  1. Verify that the ssh client service has been enabled on the ESX hosts to be backed up.
  2. Enter all ESX hosts into Ranger by IP or FQDN.
  3. Create a backup user on the ESX hosts. N.B. root ssh access is required for vmfs --> vmfs backups & restores.
  4. To verify correct configuration, it is recommended that initial attempts should be undertaken using Ranger's legacy mode.

Thursday, May 29, 2008

VMware ESX re-signaturing of the SAN Config

After a slightly strange power outage in the server room at work - the UPS stayed up everything else in the server room went down!! - I came across the situation that an ESX server had lost the primary connection to its SAN through the multipath fibre channel switch fabric.

Cue: extreme nervousness. To put it mildly.

There were a number of messages on the ESX Server console of the form:
cpu2:1034)LVM: ProbeDeviceInt:4903: vmhab1:1:0:1 may be snapshot: disabling access. See resignaturing section in SAN config guide

Actually, the last part of the error message is very good advice. A good read of the SAN configuration guide is well worth the time and effort.

Somewhere along the line the ESX host has lost this VMFS3 volume and picked it up on a different path, vmhba1:1:0:1. When the host came back up, it picked up the VMFS3 on the different path, but importantly kept information about this partition at it's previous path. This is why it's decided it's looking at a Snapshot, and responded in this manner.

So go into the console, click on the Configuration tab and select "Advanced Settings"

Expand the LVM section and set LVM.EnableResignature to 1
Then click OK to apply settings.

Select the "storage adapters" link under the configuration tab and click the "rescan" button (upper right).
Right click the vmhba (under the controller adapter for your machine) and click "rescan".

Then when you go to the summary tab, you right click and slect "refresh" and you should see your storage volume.

At least, that is what the manual would have you believe. My experience was rather different.

ESX was perfectly happy to see the volume on its new path as a new volume. Consequently, I had to remove all my inaccessible VMs and re-register them from the "new" volume. I may have had other options. This seemed to be the quickest at the time.

After all that, all the VMs started up without error, and other than a delay restarting the VMs, the users were unaware of the problem.