Solaris Virtualization

I'm doing quite a lot with virtualization at work. I've recently been looking at virtualization of our Sun Sparc servers. Sun has a very good story on virtualization. Sun has positioned itself to cover the entire virtualization market - with some very good software. Sun's strategy, which in fairness is similar to many other vendors, i.e. VMware & Microsoft, has been to make the hypervisor or "VM player" free and to charge for the management software.

Sun's virtualization software encompasses:

• LDoms
• Zones/Containers
• VirtualBox

LDom are similar to VMs under VMware ESX.
Zones/Containers are similar to VMs on VMServer
VirtualBox is, for me, more intuitive than VM player/workstation. At least in the current version.

There is a lot of information out there on the web about Sun's virtualization. The OpenSolaris and BigAdmin sites are good starting points.

Over the next few weeks I'll be creating a cookbook of procedures for using Sun's virtualization. Some of these I found on the Internet and am reproducing for my own benefit to have in a single place. Others are the result of my own perspiration.

Darik's Boot and Nuke

I came across Darik's Boot And Nuke (DBAN) a couple of months ago. I'm probably late to the party here, but I just wanted to say it is fantastic. It does exactly what it says on the tin!!

All I had to do was:

• Download the iso image.
• Burn the disk.
• Put the disk in a PC.
• Boot the PC.
• Accept the defaults.

Result: a wiped PC

It fulfilled my requirement and best of all it was free.

Linux Magazines

For the longest time I had preferred Linux Format. But the latest issue of Linux Magazine was really useful. At least for this month's edition I prefer it!

The Linux Magazine introduced me to new software that I had previously not heard of. Also, it seems to have caught up with the latest distributions. Previously it always seemed to be two or three months behind the times. But this month it has Ubuntu 8.10 at the same time as Linux Format!

The review of Virtual Machine management software was timely. I'm heavily involved in virtualization of Linux, Windows and Solaris (both x86 & Sparc) systems. I had been unaware of the software reviewed. As I've used UNIX for approx 25 years, the command line nature of the most interesting tool, MLN, won't worry me, but for SysAdmins who have only ever used the Windows interface of VMware Virtual Center server or the Virtual Infrastructure client (the "Click Next to continue" generation - or to be ruder lets call them SysAdmin Lites!) it will probably seem quite daunting.

The review of the O'Reilley book "Python for Unix and Linux System Administrators" was interesting. Although why anyone would want to use Python, when Perl is not only available but frequently seems designed for the job, is a mystery to me. Thats probably 13-ish years of prejudice coming to the fore!

The regular SysAdmin section is also usually very useful. This month's feature on Siege was particular interesting. I'll be looking into seeing if I can use that.

The websites for both magazines closely reflect the characters of the magazines themselves. The site of Linux Format is flashier, but the content of Linux Magazine is more detailed. Which sort of reminds me of a report I heard about about 15 or so years ago. The professors at an University examined the quality of thesis produced by their students. The best work from an academic point of view was produced by PC users. Mac users spent too much time chosing the just the right font or font size or slightly tweaking the layout and so didn't have enough time for "thinking".

Software configuration management software

I've been using ClearCase for approx. 13 years. I've been there, done that, and got quite a few Atria, Pure Atria, Rational and IBM T-Shirts over the years. Not to mention a few conference rucksacks and even a pint glass! Just recently I was reading Linux Magazine and read about another new software configuration management (SCM) tool, PureCM. Which interested because...

I was at an IBM seminar recently and saw some presentations about Rational Team Concert (RTC) for the first time. I was reminded of a bunch of presentations given 6, 7 or 8 years ago by Rational/IBM - my account manager at the time had told me that they were going to consolidate their tools so that there would only be a couple of tools left. Everything went quiet for such a long time I thought that the idea had been shelved or that someone had realized that having half a dozen to a dozen point tools at a slightly lower price would generate more revenue than a couple of more expensive tools. So it does now look like ClearCase, ClearQuest, ReqPro and a few other tools have now effectively been superseded by RTC. Although you can plugin ClearCase, ClearQuest and ReqPro to RTC the best reason I can think of for doing so is to ease the migration to the new tool.

So given that you are going to start again with a new tool, why not play the field and see what else is out there.

If you are going to pay for the tools then the two SCM tools that I would consider in addition to Team Concert would have been Accurev and PlasticSCM.

I've long wanted to actually put Accurev through its paces in anger. Its implementation of streams is far superior to that of ClearCase UCM, which can be considered some sort of half-*rsed after-thought. Although UCM has gotten smoother recently. From friends who have used Accurev in anger, I've heard that it does enable multi-site use of a single branch without resorting to some of the nasty hacks that have become second nature to ClearCase old-timers. The reduction of merges and regression testing builds that must provide might well be sufficient to consider or investigate a migration.

The feature of PlasticSCM that first caught my attention was the security model, which again was far superior to that of ClearCase. ClearCase, and DSEE before it, were designed by old time UNIX engineers and so utilized the octal user, group,others builtin permissions. PlasticSCM utilizes ACLs, like you'll find in Active Directory, which can be applied to almost any item: elements; streams/branches; individual versions; labels; etc. So much more flexibility.

But looking at the documentation for PureCM, it looks like it might be an acceptable alternative. Given that the tool apparently runs on Apples OS X, Linux, Solaris and Windows it is interesting that the feature comparison they chose to make was with VSS. I suspect that Windows may have been their most successful platform to date. The changeset functionality seems similar to that of Perforce. The builtin issue tracker is similar to Trac. The plugins for CruiseControl and FinalBuilder (a new tool for me - one I'm going to have to take a look at) are also a step in the right direction as RTC has built in build management - actually rather sophisiticated build management.

I do not have a conclusion. There isn't a prescription that all can take. Which is one reason there are so many SCM tools out there. This article is really another starting point for further blogs I'll be writing.

Dimdim

Very impressive software. Shame about the name. I mean, you go to the CEO and tell him to use dimdim! In the whole Web 2.0 naming scheme of things, Dimdim is one of the oddest.

Anyhow, I downloaded the free VMware version of the software, which had been created for VM player, so I had to recreate the disk for ESX. There is very little configuration that needs or even can be done. Just the name of an SMTP server, which is necessary to enable meeting invitations to be emailed out! Of course, you have have to disappear deep into the directory structure to find the dimdim.properties file and <FX: Shock, Horror> open an editor to change the value. </FX>

My company's requirement for collaboration software which Dimdim came very close to satisfying will almost certainly be mirrored by many other companies. In the current economic climate, travel budgets are restricted and the ability to "meet virtually" over the internet/internal WAN would be regarded as valuable. So a requirement to try and save money whilst still enabling staff in disparate geographies to communicate face-to-face

How did Dimdim fail, then? Simply put, it was insufficiently configurable. Perhaps this is a fault of the free version. If it is, it isn't demonstrated elsewhere on their website. There is no configuration into an LDAP or any other naming service. So your CEO (or more likely his PA) has to remember the email addresses of everyone he wants to invite.

Another fault is perhaps the lack of adequate documentation.

The purpose of downloading the free version was to assess whether the company should consider the Enterprise version of the software. The inability to integrate Dimdim into the company's infrastructure really did for it.

Which in a way I guess also goes to prove the point made int this infoworld article. A couple of years ago, I was certain that "cloud computing"/"software as a service" was about to take over the IT world. Two years later, I can still see its potential to be a real game changer. However, the rate of change seems to have slowed considerable. Google's roll-out of new features seems to have slowed. Perhaps they are concentrating these days on reliability, availability, security, uptime and scalability. Which can be no bad thing.

From the point of view of the company I work with, there is a distrust of hosting critical systems externally. My company has a large Chinese subsidiary. Everything, all transfers between the China and anyway else, has to be approved by Trade & Compliance and IT Security. The Chinese R&D department is on a completely isolated network - no access to the Internet at all. Consequently, despite the attractions of some of the cloud computing applications available, my company would almost certainly not be able to deploy them. Dimdim included.

Sendmail Relaying and Masquerading

The requirement was simple. Relay email from the CentOS Web Server in the DMZ back through the FireWall to the European corporate SMTP server for onward relay out to external customers.

Simple, huh?
Pah!

The default sendmail configuration that comes with CentOS is pretty good. But whilst the FireWall on the server would allow SMTP out the FireWall controlling the DMZ would only allow that SMTP traffic back throught the internal facing FireWall to a specific internal SMTP server. Also since the server is out in the DMZ, not only must the company FireWall "whitelist" every allowed port on each server, but each server must also only whitelist the bare minimum of required ports to function properly. So this server is only listening on http, https, SMTP and SSH.

With all that in mind, I sent a test email:
# echo "Hello, World" | mailx -s "Test" me@company.com

And that worked.

However
# echo "Is there anybody out there?" | mailx -s "Test" me@gmail.com
didn't work.

Although, I wasn't allowing DNS through the FireWall, /etc/resolv.conf contained
search emea.company.com
server 10.10.10.10

(All domain names and IPs are fictitious.)

Changing the resolv.conf file for an empty file will cause the email to me@company.com to fail to relay. It will simply be queued locally. Sendmail is trying to use DNS to look up the MX records of the email recipients. As this server is in the DMZ and we employ a split horizon DNS, this situation can't be resolved by just openning up port 53 on the server FireWall to talk to the external facing DNS server. This server isn't allowed to send email directly to the internet, and it wouldn't be able to relay email to the companies main SMTP servers as they are in a different DMZ and the network routing between the two DMZ is internal.

The DNS lookup needs to be turned off. Reading the documentation, you might think that just defining a SMART_HOST in the sendmail.mc, regenerating sendmail.cf and restarting the sendmail service would be sufficient. But it is not. DNS would still rear its ugly head.

In addition to adding
define(`SMART_HOST',`mailhost.emea.company.com') dnl
to sendmail.mc (and adding an entry for mailhost into /etc/hosts) it is also necessary to add
FEATURE(`accept_unresolvable_domains')dnl
FEATURE(`nocanonify')dnl

These two directives tells sendmail to accept email for domains that it cannot resolve and to not to canonify provided email addresses.

Command to generate the sendmail.cf file
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Command to restart Linux Sendmail service
service sendmail restart

Debugging
It is very useful to increase the log level temporarily for debugging purposes. This can be changed in sendmail.mc by changing the value of the following definition
define(`confLOG_LEVEL', `15')dnl
The default value is 9. The documentation lists 15 as the maximum for administration with the values of 16 up to 99 being of interest only to developers.

The logfile location is /var/log/maillog

Masquerading
There was an additional problem. An upstream SMTP server at our Data centre provider was performing a reserve lookup up on the originating relay server. Our SLA with the external company only allowed us to utilize specific sub-domains, and emea.company.com wasn't one of them. It was necessary to configure masquerading, too.

The following settings were added to sendmail.mc:
FEATURE(always_add_domain)dnl
MASQUERADE_AS(`company.com')dnl
MASQUERADE_DOMAIN(`company.com')dnl
FEATURE(masquerade_envelope)dnl
FEATURE(masquerade_entire_domain)dnl
FEATURE(`allmasquerade')dnl

The following feature was also commented out.
dnl EXPOSED_USER(`root')dnl
I was logged in as root when testing! D'Oh!

Resources
The following link provides a good description of sendmail on CentOS 5, but you really have to know a little bit about what you are doing first, otherwise it is confusing: linuxtopia
Another closely related link.

sendmail.org is also a good source of detail, especially on what all those options/FEATURES in the sendmail.mc file are for, and for Masquerading & Relaying.

An excellent HP website on how Sendmail works.

Sendmail nullclient configuration on CentOS v5.2

Sendmail is the work of the devil.

Here, however, is how to set up a nullclient, which will enable all mail from a server to be forwarded to a central mail hub.

[root@server1 mail]# rpm -qa | grep sendmail
sendmail-cf-8.13.8-2
sendmail-8.13.8-2
[root@server1 mail]# cat /etc/mail/sendmail.mc
divert(-1)dnl
dnl #
dnl # This is the sendmail macro config file for m4. If you make changes to
dnl # /etc/mail/sendmail.mc, you will need to regenerate the
dnl # /etc/mail/sendmail.cf file by confirming that the sendmail-cf package is
dnl # installed and then performing a
dnl #
dnl # make -C /etc/mail
dnl #
include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
VERSIONID(`Nullclient for Linux')dnl
OSTYPE(`linux')dnl
DOMAIN(`generic')dnl
FEATURE(`nullclient',`example.com')dnl
undefine(`ALIAS_FILE')dnl

[root@server1 mail]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.0.1 server1.example.com server1

[root@server1 mail]# make -C /etc/mail
make: Entering directory `/etc/mail'
make: Leaving directory `/etc/mail'
[root@server1 mail]# service sendmail restart.
Shutting down sm-client: [ OK ]
Shutting down sendmail: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
[root@server1 mail]#

Oviously, you could always add those lines into a file called something like null.mc and then create your sendmail.cf file with a command line like:

[root@server1 mail]# m4 null.mc > sendmail.cf


Just discovered that much of this is covered over at faqs.org.

Vista as a Virus #1

Among many other duties and responsibilities, I am also a Domain Admin of my company's Active Directory. Despite having a normal user account, I must confess to frequently logging into my desktop with my Domain Admin account. On one such occasion, I was trying to track down a DNS issue that our Sydney office was suffering, when I realised that I needed to flush my local DNS resolver cache. Pretty straight forward? Just open a Command Prompt:

C:\>"ipconfig /flushdns"

The requested operation requires elevation

C:\>

Oh! That didn't work! What the heck is "elevation"? Other than sounding like a U2 song!

Well, having googled around and found this thread on a Microsoft site, it appears that as an Active Directory Domain Admin I was insufficiently priviledged on my Desktop to perform that operation from a Command Prompt!

To be able to perform that sort of operation in a Command Prompt I should have started the Command Prompt with "Run as Administrator".

Some may argue that this is merely improving security, but I would not be one of them. Since then, I was started up FileZilla which informed me there was an update available and did I want to install it. I said yes. FileZilla downloaded the file successfully, and then failed. Guess what! Actually running the install program was an operation which required elevation. Grrr!

RVTools

I've just used RVTools for the first time.

What an absolutely excellent tool. It isn't graphical, but it's tabular presentation of information reveals information that I would have had to drill down into each VM's data to find. I thoroughly recommend it to anyone using VMware ESX.

And best of all, its free!

Suppliers' Websites

If ever an application crashes on Windows, I never hit the button to send information about it to Microsoft. I guess I was conditioned in the futility of attempting to engage Microsoft Support over 15 years ago. Ever since I haven't bothered with them. There is only so much hitting your head against a brick wall that is good for you after all. That said in this age of the Internet the resources provided on Microsoft's website are pretty good. Even the things that I might want to see might be there. If only I could find them.

I remember possibly 10 years ago, a colleague slamming the phone receiver down in frustration after talking to IBM when trying to get a licence for some software we had purchased. I took over and finally got a licence. I wasn't completely sure it was "our" licence, but it was a licence and it worked and we were able to move on. Even after its recent re-vamp - actually I'm sure that it is probably a continuous process in play here - it is still damn difficult to find what you really want. Try and use the IBM search for the bios update for a specific server, e.g. a x346. The results will list just about any IBM server.

The solution?

Just use Google. We all know it makes sense. I just wish for a higher signal to noise ratio. But no matter how bad it is, its still better than trying to use these two Vendors own search engines.

VMware VDM Agent - Access is Denied

When trying to RDP onto the VM that had been set up, the RDP screen comes up and then a box with a red cross saying "VMware VDM Agent - Access is Denied".

What's Up?

By default, VDM 2.1 blocks non-VDM RDP connections. This can be disabled by Group Policy or a registry setting on the VMs.

The group policy file is included on the VDM connection server install under the ADM subfolder.

The registry key that should be set is "AllowDirectRDP"="true" which can be found in either of the following two locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\VMware, Inc\VMware VDM\Agent\Configuration
or
HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc\VMware VDM\Agent\Configuration

If there is no VMware, Inc tree under Policies or if the key does not exist under the latter subtree, just create it under the latter tree.