Restore Vizioncore vRanger backups via the OS

The Development VMware ESX server at my company is an IBM 366 with a SCSI attached EXP400 external disk pack. This system arrived in the UK from a company site on the west coast of the US via a stop at the company HQ on the east coast of the US. To say that the hardware had been shunted from pillar to post would be a minor understatement.

We are using Vizioncore vRanger to back up the VMs on both this Development box and the Production VMware ESX server.

Just recently, the RAID 5 array on the EXP400 dropped 6 disks(out of 9)! Why it did this is a different story. Here I'll recount how we recovered from this.

There were a couple of VMs we had to get back online quickly. No problem, we had vRanger backing them up.

Ah! Now, well there was a problem. The backups were very successful, but for some reason we're still investigating, vRanger refused point blank to recover to the Production system.

However, we had access to the backup via the Windows OS. A quick google discovered this thread on the Vizioncore support site.

Restoring a VMWare machine from the '.tvzc' files of vRanger :
1.) Download FileZipper : http://www.vizioncore.com/Downloads/ProductSupport/vcbrestore.zip
2.) Download BSDTar: http://www.vizioncore.com/Downloads/ProductSupport/bsdtar.zip
3.) Install BSDTar (the zipfile contains an installable .exe)
4.) Extract the desired files :
FileZipper.exe -D -I "filename.tvzc" -O - | "c:\Program Files\GnuWin32\bin\bsdtar.exe" xzvf -
5.) Ensure file permissions are correct - I have cygwin installed on my PC, which can be invaluable!
6.) remove the .vzsnp extension from the end of files.
7) Within Virtual Center Server:

• use Browse datastore on the Production Server's storage
• Upload the restored files to the Production ESX server
• select the .vmf file and Add to Inventory

8.) Start up the VM.

Bob's your parental sibling, of the usually male variety.

Blogging needs Wordpress not Wiki kludges

So our COO decided that we were going to provide a blogging solution in addition to a wiki. The "My blog" add-on to Mediawiki doesn't really cut it, although as a quick and dirty workaround it has a place.

I downloaded WordPress as a Jumpbox appliance. Quick, easy, restrictive. For a small company, it would be really good solution. For a larger company with an infrastructure to tie into, it is lacking. However, I'm really only talking about the free download version. I briefly considered registering the appliance, but didn't want any delay. So perhaps I am being slightly unfair. But hey its my blog!

I downloaded v2.5.1 of the WordPress application, created a CentOS v5.2 Linux VM configured as a Web & MySQL server and rolled my own! As a standalone application that you can install plugins into, its pretty straightforward and looks pretty good too.

I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually a bit tricky to set up. I remember it taking a number of hours to accomplish. Events since have wiped out quite a bit of my recollection of the event. It was quite cool after I had configured everything properly, though.

At this point, I realized that what the COO really wanted wasn't a single blog, but the ability for many VPs to have a blog.

Back to the drawing board?

Not completely. At this point, I downloaded v1.5.1 of the WordPress MultiUser software. A default installation is just as simple as the single user version of the application.

Again I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually very tricky to set up.

If you follow the above links to the Ldap plugins you'll discover that they are completely different. The wpmu-ldap plugin is different from the WordPress ldap plugin, written by different people.

The writer of the ldap plugin for WordPress MU has a blog here where he announced the release of the latest version. The maillists Aaron refers to at the bottom of his blog are an invaluable source of information, because to say the documentation is sparse is like saying that I'm an overweight bearded slaphead, i.e. a completely accurate and unbiased statement of fact.


Things I discovered whilst deploying WordPress MU and the ldap plugin are:
All the ldap files have to be owned by the httpd/apache/web server process owner. Otherwise the plugin isn't even seen. This is a file permissions problem, so not serious, but it can take an embarrassingly long time to track down. Or at least it did for me.

If after the WPMU ldap plug-in is enabled one of the files is edited by the root user and becomes owned by root, then the result is the infamous "White Screen of Death". Again, not something I immediately recognized. It took an embarrassingly long seeming hour to work it out!

The most obvious difference is that the single user wordpress plugin lets you specify the attribute to filter against, whereas the multiuser plugin lets you choose between linux LDAP and windows LDAP. Now the wpmu-ldap plugin maps linux to uid and windows to sAMAccountName. I was authenticating against Lotus Notes and needed cn! My only immediate option was to hack the source code.

Now its working its quite cool, but I did pick up some scars and a few more white hairs.

phpBB installation

This is yet another case where my mileage hasn't actually varied. But I had to write about the phpBB installation as it is just so damn slick.

It must be roughly 6 or7 years ago when I first installed and configured a phpBB site on a Solaris 8 server with MySQL v3.23.42, Apache v1.3.26 and php v4.0.6. Even then the install was pretty good, although it left enough techie stuff to be done that you felt you'd undertaken a "real man's job"! Afterall, I had to compile the Apache, MySQL and PHP distributions.

In this case, I created:

• a new VM on my development ESX server,
• loaded up CentOS v5.2 configured as a Web & MySQL server
• loaded up some additional php libraries
• started the httpd & mysqld services
• download and installed the latest phpMyAdmin
• created a DB
• created a DB User with appropriate priviledges
• started and finished the phpBB configuration very quickly

The phpBB configuration is performed via a web page. It recognises the current state of the installation and just steps you through it. When it has acquired all the relevant configuration details, it creates the tables in the database and sets up the initial admin account. And then you are in a position to start using the system.

Bob's your parental sibling, usually of the male variety!


This does slightly simplify the process, but only in terms of creating/deploying a new machine. I had to add a new server into the company QIP (now called VitalQIP) system and push that out to DNS.

After deploying a server, there's also planning that has to be undertaken for administration, usage policies, backup and restoration for the system.

VMware Training

After 18 months of using ESX starting with v2.5.4 and upgrading through v3.0.2 to v3.5, where the only training I'd had was to watch the DVD training "Virtualize it With: VMware ESX Server 3.0" from the elias khnaser company, I've finally taken the VMware training course "VMware Infrastructure 3: Install and Configure".

After such a long time, was it worth it?

Absolutely!

Didn't I know most of it?

Yes, perhaps 85%. I must admit to gaining a certain satisfaction at realizing how much I already knew.

But then again its hard to really know DRS and HA from the manuals and training DVD when your environment is two sites each with a development (sort of - some of those VMs seem to be in production!) server and a single production server.So 4 ESX servers in all! I have more servers coming, so the timing made sense. I'll soon be running my production servers in a DRS cluster on each site. I'll be considering making them HA clusters too, although that isn't so clear cut.

The ability to discuss issues and ask questions in a class-room environment, where the answer can be "I do not know, but lets just try that..." and there is no fall out in terms of production system downtime, can be really useful.

Also you pick up the odd bit of wisdom such as that soak testing your memory beyond that performed by the bios startup is well worth the effort. Unless you are under-utilizing your server, ESX will exercise all your RAM in a way that most other OS simply will not. So a memory fault that might never have been discovered by another OS can be exposed in very short order. Much better to perform a thorough soak test before deployment, than have your VMs do it.


I suppose the other reason for undertaking the training is that it is a requirement for the VCP exam.

Ticksy, that!

Although in VMware's position I would have done the same.

I have taken the VCP mock exam of 20 questions and passed with 80%. I'm still really annoyed by the 4 questions that I answered incorrectly. In my defence two of the questions weren't particularly practical, but in the actual exam that won't cut it.

Vizioncore vRanger Configuration Take 2

You should be thinking about backup even before you start creating Virtual Machines. This is perhaps obvious. Although, it is still possible to defer that decision by using traditional "in VM" solutions.

One of the features of ESX is that you can have a display name for a VM in the GUI which bares no relation to the names of the files. Now the name for the files of the VMs are taken at the time of creation from the XXX form. By default this is also the display name used within the GUI. The display name can be changed later. To change the filenames used, requires that the VM is stopped and all the files and the directory used are modified. And modified correctly!

Now it is possible to use a wide range of characters in the name of a VM, e.g. this is legal:


Legal, but not sensible.

Whilst ESX has no problem with filenames with non-alphanumeric characters, both vRanger and VCB do. They will both fail to back up a VM with the name indicated. That may suggest something about how both utilities are architected, or perhaps the APIs they are utilizing. It doesn't matter. You have to deal with it.

When you create a VM, give it a sensible, simple but meaningful alphanumeric name. Afterwards you can choose rename from the right mouse button menu and change the display name to include whitespace and other characters.


Ensure that you have modified the System Resource Reservation parameters, which reserves resources for the backup process to utilize.


Foreach ESX Server:
On the Configuration-> Ststem Resource Reservation->Simple Tab, set
CPU : 1500Mhz
Memory : 800MB


The Simple setting equates to the host->system setting under the Advanced tab!

After changing these settings, it is necessary to reboot the ESX server, before they take effect. Consequently, if you can it is sensible to set this all up before you start serving Virtual Machines.

Then

1. Verify that the ssh client service has been enabled on the ESX hosts to be backed up.
2. Enter all ESX hosts into Ranger by IP or FQDN.
3. Create a backup user on the ESX hosts. N.B. root ssh access is required for vmfs --> vmfs backups & restores.
4. To verify correct configuration, it is recommended that initial attempts should be undertaken using Ranger's legacy mode.

CD-ROMs in VMware

In my experience, after you have added enough NICs to a VMware ESX machine to be useful, i.e. at least 6 to 8, you start to get into a situation whereby ESX is unable to identify a physical CD-ROM that might be attached.

At that point whenever you try to start a Virtual Machine which is configured to try to attach to the physical CD-ROM, it will take an inordinate time to boot. Essentially hanging on the way up. and even after the machine has fully booted. It seems as though the VM is only getting 5 to 10 seconds of CPU every 2 minutes or so.

This can be immensely frustrating.

I know. Before I worked out what was happening, I became quite impatient.

The fix is quite straitforward. Simply change the CD-ROM over to a client device. For often than not, that is the most useful setting. It is still straitforward to map an iso to the drive as well, should you need that.

Mediawiki extensions

As I have written elsewhere a good deal of the power and pain of implementing a Wiki with Mediawiki is in the use and deployment of extensions.

At the moment I am suffering from the interaction of two extensions

I am using the Ldap authentication extension to authenticate the users against the company's Microsoft Windows 2003 Active Directory. This is working very well and generated a great deal of kudos when it was deployed. Not actually single sign on, but a small step in that direction for us. Actually, after slogging through the documentation, it wasn't so difficult to install and configure. Not withstanding anything else I may have written elsewhere on this blog.

I am using the "My Blog" extension to allow the users to create simple blog pages. The blog pages are simply ordinary wiki pages, which are aggregated very much in the manner of templates.

Although users login successfully, they receive a message which suggests that they do not because their cookie settings are incorrect and they are presented with another login box. However, if they just traverse away from the login page everything is fine. But everyone had to be told that they could.


OK, I was wrong above. After further investigation, it appears that the problem originates in the PasswordProtected extension. Having checked and re-checked the source code it is far from clear exactly why it should be causing the problem. Luckily, having reviewed the functionality, it doesn't work in quite the way we'd like so I was able to remove the extension from the LocalSettings.php file.

No sooner was I was congratulating myself on the implementation of the Ldap authentication extension against 3 internal Microsoft Active Directory Domains, than I was asked to add authentication against the Ldap service of our Lotus Notes installation. Having already configured the MS AD Domain authentication this was actually quite simple. In fact I've been able to comparing the two, the ldap authentication against Lotus Notes is simpler than MS AD Domains! In fact, I've re-ordered the Domain login list, so that Lotus Notes is the first option, and left the MS AD domain login option for only a restricted group of users.

There was one thing that surprised me about the user management side of Mediawiki. And this may be a result of our using the Ldap authentication plugin, especially against multiple domains. When A N User from Domain A logged in and a local User called A n user was created. If later on A N User from Domain B logged in, then the Domain B user would be mapped to the same local A n user account as the Domain A user.

Now, in my environment, this is exactly what I want. In fact, if it didn't I'd probably have to be scrabbling through the source code to try and mangle the usernames to try and achieve that result.