Aarrgghh! The sky is falling!!

We had an old Windows 2000 Active Directory domain - formerly an NT 4.0 domain. It had been limping on for quite a while past its sell by date.

Finally something had to go.
And it did.
Big time!

We had only kept it for a bunch of developers who had been very resistant to change. Through it they accessed ClearCase VOBs resident on a Solaris server. We were lucky we had this architecture.

The Domain Controllers stopped replicating with each other. And nothing, no how was going to get them back to being happy with each other. Perhaps it sounds like I'm making light of the situation, but a couple of days ago everything seemed like a source of stress.

Because it was only a small group using this domain, we had a solution that could be quite quickly and easily rolled out.

Essentially, these developers stopped logging into the domain and starting using local accounts on their PCs. This is how we set things up.

For each developer's PCs:

1. create a local user for clearcase_albd


2. create a local clearcase group


3. add clearcase_albd to clearcase group


4. create a local user for the engineer


5. create a local group for the engineer to match their UNIX group


6. change the Atria Location Broker service to use local clearcase_albd account


7. edit the HKEY_LOCAL_MACHINE -> SOFTWARE -> Atria -> ClearCase -> CurrentVersion -> ClearCaseGroupName registry value to point to the local clearcase group


8. logon as clearcase_albd and set CLEARCASE_PRIMARY_GROUP EV to clearcase


9. logon using engineer's local user and set CLEARCASE_PRIMARY_GROUP EV to the new local group matching the UNIX group
   


10. Loaded client for NFS from SFU v3.5


11. Configure client for NFS to map local user to UNIX user and to mount the VOB storage partition automatically.


12. Create new views or fix_prot the old views.

As views are meant to be temporary structures, even where views were migrated with fix_prot, those views were only actually used to check objects in and then removed. New views were created for on-going work.

Longer term this team is going into Windows 2003 Active Directory Domain that is used by the rest of the development teams.

ClearCase replica naming and replication

Over the years I've seen various conventions used for the naming of ClearCase VOB replicas. Some of the more popular, especially from when I started at my current company, some 12 years or so ago were:

• <site name>
  
• <division name>
  
• <division name>_<site name>
  

Recently, the admins at a remote site I've been working with have adopted a naming convention for their VOB replicas of <vob tag>_<site name>! I couldn't work out why you would want to include the vob tag as part of a replica name until I realized that they were using the supplied ClearCase MultiSite replication scripts.

Eleven or twelve years ago when using ClearCase v2.2 or possibly v3.0.1 I had written my own Perl script to manage replication. I used an earlier shell script as a starting point. Over the next 4 or 5 years I slowly modified the script to take advantage of new features, i.e [ls|ch]replica -actual I haven't touched it since.

This script automatically produced a filename: sync_<vob tag>_<source replica>_<target replica>_<Time stamp>
e.g. if a VOB had a vob tag of /vobs/src and replicas of Austin and Healey, then a pcaket sent from Austin to Healey might have a packet name of sync_vobs_src_Austin_Healey_21072008-101010 with a suffix _<num> if more than one packet was created.

But to get the vob tag to appear with the supplied scripts you need to include the vob tag as part of the replica names. Even then you only see the source replica listed not the target.

Changing the Domain Name of a WordPress Mu Site

I downloaded the Multi-Site Manager plugin from http://wpmudev.org/project/Multi-Site-Manager

This plugin makes it incredibly easy to create a new site. You can clone the original site to a new site, and then transfer blogs between those sites.

Unfortunately, in my experience, it didn't quite work perfectly. Probably a rookie mistake on my part. It is probably best not to transfer the primary blog from the original site. That caused some very strange behaviour from WordPress.

Anyway I ended up hacking through the MySQL database manually editing all the URL references.

Aarrgghh!!

This was made worse by the extra half dozen or more users who had appeared overnight. Each user causes an additional 8 tables to be created in the MySQL database. Truly, everyone has their own playpen!

137 tables later all the references had been updated.

There was a problem with tags and references or more specifically Permalinks. I found I had to go in and reselect the permalinks options. As none of the existing users have complained about tags and categories I’m presuming that they have successfully followed the instructions I emailed through to them or haven't yet re-logged in.

ldap services

One of the prime requirements of any service that is added within my company is authentication.

Within my company, there are two main sources of authentication:

• Microsoft Active Directory
• Lotus Notes ldap service

Now it is possible with varying degrees of difficulty to "persuade" most tools/services to use LDAP as an authentication source. However, there are assumptions written into most of these tools that if you are seeking to use LDAP you are either using OpenLDAP (or similar) or Microsoft's Active Directory.

In some ways it is quite encouraging to see how many other people are looking to authenticate against Active Directory. In other ways, it is deeply depressing that with so many years head start, the various UNIX vendors couldn't agree upon a common naming services standard that would be an improvement upon Active Directory.

I suspect that some will point to Kerberos and LDAP themselves as collaborative triumphs, which Microsoft had to use within Active Directory itself. However, whilst those are compelling technologies, they are not themselves individually a compelling solution. Collectively, they can be induced to become a solution, but depending upon the implementor, they may not be a compelling solution.

Restore Vizioncore vRanger backups via the OS

The Development VMware ESX server at my company is an IBM 366 with a SCSI attached EXP400 external disk pack. This system arrived in the UK from a company site on the west coast of the US via a stop at the company HQ on the east coast of the US. To say that the hardware had been shunted from pillar to post would be a minor understatement.

We are using Vizioncore vRanger to back up the VMs on both this Development box and the Production VMware ESX server.

Just recently, the RAID 5 array on the EXP400 dropped 6 disks(out of 9)! Why it did this is a different story. Here I'll recount how we recovered from this.

There were a couple of VMs we had to get back online quickly. No problem, we had vRanger backing them up.

Ah! Now, well there was a problem. The backups were very successful, but for some reason we're still investigating, vRanger refused point blank to recover to the Production system.

However, we had access to the backup via the Windows OS. A quick google discovered this thread on the Vizioncore support site.

Restoring a VMWare machine from the '.tvzc' files of vRanger :
1.) Download FileZipper : http://www.vizioncore.com/Downloads/ProductSupport/vcbrestore.zip
2.) Download BSDTar: http://www.vizioncore.com/Downloads/ProductSupport/bsdtar.zip
3.) Install BSDTar (the zipfile contains an installable .exe)
4.) Extract the desired files :
FileZipper.exe -D -I "filename.tvzc" -O - | "c:\Program Files\GnuWin32\bin\bsdtar.exe" xzvf -
5.) Ensure file permissions are correct - I have cygwin installed on my PC, which can be invaluable!
6.) remove the .vzsnp extension from the end of files.
7) Within Virtual Center Server:

• use Browse datastore on the Production Server's storage
• Upload the restored files to the Production ESX server
• select the .vmf file and Add to Inventory

8.) Start up the VM.

Bob's your parental sibling, of the usually male variety.

Blogging needs Wordpress not Wiki kludges

So our COO decided that we were going to provide a blogging solution in addition to a wiki. The "My blog" add-on to Mediawiki doesn't really cut it, although as a quick and dirty workaround it has a place.

I downloaded WordPress as a Jumpbox appliance. Quick, easy, restrictive. For a small company, it would be really good solution. For a larger company with an infrastructure to tie into, it is lacking. However, I'm really only talking about the free download version. I briefly considered registering the appliance, but didn't want any delay. So perhaps I am being slightly unfair. But hey its my blog!

I downloaded v2.5.1 of the WordPress application, created a CentOS v5.2 Linux VM configured as a Web & MySQL server and rolled my own! As a standalone application that you can install plugins into, its pretty straightforward and looks pretty good too.

I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually a bit tricky to set up. I remember it taking a number of hours to accomplish. Events since have wiped out quite a bit of my recollection of the event. It was quite cool after I had configured everything properly, though.

At this point, I realized that what the COO really wanted wasn't a single blog, but the ability for many VPs to have a blog.

Back to the drawing board?

Not completely. At this point, I downloaded v1.5.1 of the WordPress MultiUser software. A default installation is just as simple as the single user version of the application.

Again I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually very tricky to set up.

If you follow the above links to the Ldap plugins you'll discover that they are completely different. The wpmu-ldap plugin is different from the WordPress ldap plugin, written by different people.

The writer of the ldap plugin for WordPress MU has a blog here where he announced the release of the latest version. The maillists Aaron refers to at the bottom of his blog are an invaluable source of information, because to say the documentation is sparse is like saying that I'm an overweight bearded slaphead, i.e. a completely accurate and unbiased statement of fact.


Things I discovered whilst deploying WordPress MU and the ldap plugin are:
All the ldap files have to be owned by the httpd/apache/web server process owner. Otherwise the plugin isn't even seen. This is a file permissions problem, so not serious, but it can take an embarrassingly long time to track down. Or at least it did for me.

If after the WPMU ldap plug-in is enabled one of the files is edited by the root user and becomes owned by root, then the result is the infamous "White Screen of Death". Again, not something I immediately recognized. It took an embarrassingly long seeming hour to work it out!

The most obvious difference is that the single user wordpress plugin lets you specify the attribute to filter against, whereas the multiuser plugin lets you choose between linux LDAP and windows LDAP. Now the wpmu-ldap plugin maps linux to uid and windows to sAMAccountName. I was authenticating against Lotus Notes and needed cn! My only immediate option was to hack the source code.

Now its working its quite cool, but I did pick up some scars and a few more white hairs.

phpBB installation

This is yet another case where my mileage hasn't actually varied. But I had to write about the phpBB installation as it is just so damn slick.

It must be roughly 6 or7 years ago when I first installed and configured a phpBB site on a Solaris 8 server with MySQL v3.23.42, Apache v1.3.26 and php v4.0.6. Even then the install was pretty good, although it left enough techie stuff to be done that you felt you'd undertaken a "real man's job"! Afterall, I had to compile the Apache, MySQL and PHP distributions.

In this case, I created:

• a new VM on my development ESX server,
• loaded up CentOS v5.2 configured as a Web & MySQL server
• loaded up some additional php libraries
• started the httpd & mysqld services
• download and installed the latest phpMyAdmin
• created a DB
• created a DB User with appropriate priviledges
• started and finished the phpBB configuration very quickly

The phpBB configuration is performed via a web page. It recognises the current state of the installation and just steps you through it. When it has acquired all the relevant configuration details, it creates the tables in the database and sets up the initial admin account. And then you are in a position to start using the system.

Bob's your parental sibling, usually of the male variety!


This does slightly simplify the process, but only in terms of creating/deploying a new machine. I had to add a new server into the company QIP (now called VitalQIP) system and push that out to DNS.

After deploying a server, there's also planning that has to be undertaken for administration, usage policies, backup and restoration for the system.

VMware Training

After 18 months of using ESX starting with v2.5.4 and upgrading through v3.0.2 to v3.5, where the only training I'd had was to watch the DVD training "Virtualize it With: VMware ESX Server 3.0" from the elias khnaser company, I've finally taken the VMware training course "VMware Infrastructure 3: Install and Configure".

After such a long time, was it worth it?

Absolutely!

Didn't I know most of it?

Yes, perhaps 85%. I must admit to gaining a certain satisfaction at realizing how much I already knew.

But then again its hard to really know DRS and HA from the manuals and training DVD when your environment is two sites each with a development (sort of - some of those VMs seem to be in production!) server and a single production server.So 4 ESX servers in all! I have more servers coming, so the timing made sense. I'll soon be running my production servers in a DRS cluster on each site. I'll be considering making them HA clusters too, although that isn't so clear cut.

The ability to discuss issues and ask questions in a class-room environment, where the answer can be "I do not know, but lets just try that..." and there is no fall out in terms of production system downtime, can be really useful.

Also you pick up the odd bit of wisdom such as that soak testing your memory beyond that performed by the bios startup is well worth the effort. Unless you are under-utilizing your server, ESX will exercise all your RAM in a way that most other OS simply will not. So a memory fault that might never have been discovered by another OS can be exposed in very short order. Much better to perform a thorough soak test before deployment, than have your VMs do it.


I suppose the other reason for undertaking the training is that it is a requirement for the VCP exam.

Ticksy, that!

Although in VMware's position I would have done the same.

I have taken the VCP mock exam of 20 questions and passed with 80%. I'm still really annoyed by the 4 questions that I answered incorrectly. In my defence two of the questions weren't particularly practical, but in the actual exam that won't cut it.

Vizioncore vRanger Configuration Take 2

You should be thinking about backup even before you start creating Virtual Machines. This is perhaps obvious. Although, it is still possible to defer that decision by using traditional "in VM" solutions.

One of the features of ESX is that you can have a display name for a VM in the GUI which bares no relation to the names of the files. Now the name for the files of the VMs are taken at the time of creation from the XXX form. By default this is also the display name used within the GUI. The display name can be changed later. To change the filenames used, requires that the VM is stopped and all the files and the directory used are modified. And modified correctly!

Now it is possible to use a wide range of characters in the name of a VM, e.g. this is legal:


Legal, but not sensible.

Whilst ESX has no problem with filenames with non-alphanumeric characters, both vRanger and VCB do. They will both fail to back up a VM with the name indicated. That may suggest something about how both utilities are architected, or perhaps the APIs they are utilizing. It doesn't matter. You have to deal with it.

When you create a VM, give it a sensible, simple but meaningful alphanumeric name. Afterwards you can choose rename from the right mouse button menu and change the display name to include whitespace and other characters.


Ensure that you have modified the System Resource Reservation parameters, which reserves resources for the backup process to utilize.


Foreach ESX Server:
On the Configuration-> Ststem Resource Reservation->Simple Tab, set
CPU : 1500Mhz
Memory : 800MB


The Simple setting equates to the host->system setting under the Advanced tab!

After changing these settings, it is necessary to reboot the ESX server, before they take effect. Consequently, if you can it is sensible to set this all up before you start serving Virtual Machines.

Then

1. Verify that the ssh client service has been enabled on the ESX hosts to be backed up.
2. Enter all ESX hosts into Ranger by IP or FQDN.
3. Create a backup user on the ESX hosts. N.B. root ssh access is required for vmfs --> vmfs backups & restores.
4. To verify correct configuration, it is recommended that initial attempts should be undertaken using Ranger's legacy mode.

CD-ROMs in VMware

In my experience, after you have added enough NICs to a VMware ESX machine to be useful, i.e. at least 6 to 8, you start to get into a situation whereby ESX is unable to identify a physical CD-ROM that might be attached.

At that point whenever you try to start a Virtual Machine which is configured to try to attach to the physical CD-ROM, it will take an inordinate time to boot. Essentially hanging on the way up. and even after the machine has fully booted. It seems as though the VM is only getting 5 to 10 seconds of CPU every 2 minutes or so.

This can be immensely frustrating.

I know. Before I worked out what was happening, I became quite impatient.

The fix is quite straitforward. Simply change the CD-ROM over to a client device. For often than not, that is the most useful setting. It is still straitforward to map an iso to the drive as well, should you need that.