Some good links

I came across a blog just today by following a link on Scott Lowe's bookmarks: Linux Performance Tuning

This is really useful information. All pulled together in one place. Some of it might seem common sense, but as has frequently been said it is surprising how uncommon common sense can be.


As I wrote in my previous post about recovering from a disaster, documentation on your environment is vital. And these two links below provide that.


I've only just noticed that RVTools has been considerably updated. This is an excellent tool and the more so for being free. I feel a bit guilty about  not being in a position to donate some of my employers' money via paypal.

Actually, the new feature that I'm most excited by is one that has been there for several versions now. Specifically, the ability to export all the information in csv format. And to do that from the command line!


I was browsing sunfreeware and came across sys_diag, a script generated by Todd Jobson. For doumenting the state of a Solaris server, it looks damn comprehensive. I intend to use it on the Solaris systems at my work and save the results into our database of server information.

Sendmail Relaying and Masquerading

The requirement was simple. Relay email from the CentOS Web Server in the DMZ back through the FireWall to the European corporate SMTP server for onward relay out to external customers.

Simple, huh?
Pah!

The default sendmail configuration that comes with CentOS is pretty good. But whilst the FireWall on the server would allow SMTP out the FireWall controlling the DMZ would only allow that SMTP traffic back throught the internal facing FireWall to a specific internal SMTP server. Also since the server is out in the DMZ, not only must the company FireWall "whitelist" every allowed port on each server, but each server must also only whitelist the bare minimum of required ports to function properly. So this server is only listening on http, https, SMTP and SSH.

With all that in mind, I sent a test email:
# echo "Hello, World" | mailx -s "Test" me@company.com

And that worked.

However
# echo "Is there anybody out there?" | mailx -s "Test" me@gmail.com
didn't work.

Although, I wasn't allowing DNS through the FireWall, /etc/resolv.conf contained
search emea.company.com
server 10.10.10.10

(All domain names and IPs are fictitious.)

Changing the resolv.conf file for an empty file will cause the email to me@company.com to fail to relay. It will simply be queued locally. Sendmail is trying to use DNS to look up the MX records of the email recipients. As this server is in the DMZ and we employ a split horizon DNS, this situation can't be resolved by just openning up port 53 on the server FireWall to talk to the external facing DNS server. This server isn't allowed to send email directly to the internet, and it wouldn't be able to relay email to the companies main SMTP servers as they are in a different DMZ and the network routing between the two DMZ is internal.

The DNS lookup needs to be turned off. Reading the documentation, you might think that just defining a SMART_HOST in the sendmail.mc, regenerating sendmail.cf and restarting the sendmail service would be sufficient. But it is not. DNS would still rear its ugly head.

In addition to adding
define(`SMART_HOST',`mailhost.emea.company.com') dnl
to sendmail.mc (and adding an entry for mailhost into /etc/hosts) it is also necessary to add
FEATURE(`accept_unresolvable_domains')dnl
FEATURE(`nocanonify')dnl

These two directives tells sendmail to accept email for domains that it cannot resolve and to not to canonify provided email addresses.

Command to generate the sendmail.cf file
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Command to restart Linux Sendmail service
service sendmail restart

Debugging
It is very useful to increase the log level temporarily for debugging purposes. This can be changed in sendmail.mc by changing the value of the following definition
define(`confLOG_LEVEL', `15')dnl
The default value is 9. The documentation lists 15 as the maximum for administration with the values of 16 up to 99 being of interest only to developers.

The logfile location is /var/log/maillog

Masquerading
There was an additional problem. An upstream SMTP server at our Data centre provider was performing a reserve lookup up on the originating relay server. Our SLA with the external company only allowed us to utilize specific sub-domains, and emea.company.com wasn't one of them. It was necessary to configure masquerading, too.

The following settings were added to sendmail.mc:
FEATURE(always_add_domain)dnl
MASQUERADE_AS(`company.com')dnl
MASQUERADE_DOMAIN(`company.com')dnl
FEATURE(masquerade_envelope)dnl
FEATURE(masquerade_entire_domain)dnl
FEATURE(`allmasquerade')dnl

The following feature was also commented out.
dnl EXPOSED_USER(`root')dnl
I was logged in as root when testing! D'Oh!

Resources
The following link provides a good description of sendmail on CentOS 5, but you really have to know a little bit about what you are doing first, otherwise it is confusing: linuxtopia
Another closely related link.

sendmail.org is also a good source of detail, especially on what all those options/FEATURES in the sendmail.mc file are for, and for Masquerading & Relaying.

An excellent HP website on how Sendmail works.

Sendmail nullclient configuration on CentOS v5.2

Sendmail is the work of the devil.

Here, however, is how to set up a nullclient, which will enable all mail from a server to be forwarded to a central mail hub.

[root@server1 mail]# rpm -qa | grep sendmail
sendmail-cf-8.13.8-2
sendmail-8.13.8-2
[root@server1 mail]# cat /etc/mail/sendmail.mc
divert(-1)dnl
dnl #
dnl # This is the sendmail macro config file for m4. If you make changes to
dnl # /etc/mail/sendmail.mc, you will need to regenerate the
dnl # /etc/mail/sendmail.cf file by confirming that the sendmail-cf package is
dnl # installed and then performing a
dnl #
dnl # make -C /etc/mail
dnl #
include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
VERSIONID(`Nullclient for Linux')dnl
OSTYPE(`linux')dnl
DOMAIN(`generic')dnl
FEATURE(`nullclient',`example.com')dnl
undefine(`ALIAS_FILE')dnl

[root@server1 mail]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.0.1 server1.example.com server1

[root@server1 mail]# make -C /etc/mail
make: Entering directory `/etc/mail'
make: Leaving directory `/etc/mail'
[root@server1 mail]# service sendmail restart.
Shutting down sm-client: [ OK ]
Shutting down sendmail: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
[root@server1 mail]#

Oviously, you could always add those lines into a file called something like null.mc and then create your sendmail.cf file with a command line like:

[root@server1 mail]# m4 null.mc > sendmail.cf


Just discovered that much of this is covered over at faqs.org.

Blogging needs Wordpress not Wiki kludges

So our COO decided that we were going to provide a blogging solution in addition to a wiki. The "My blog" add-on to Mediawiki doesn't really cut it, although as a quick and dirty workaround it has a place.

I downloaded WordPress as a Jumpbox appliance. Quick, easy, restrictive. For a small company, it would be really good solution. For a larger company with an infrastructure to tie into, it is lacking. However, I'm really only talking about the free download version. I briefly considered registering the appliance, but didn't want any delay. So perhaps I am being slightly unfair. But hey its my blog!

I downloaded v2.5.1 of the WordPress application, created a CentOS v5.2 Linux VM configured as a Web & MySQL server and rolled my own! As a standalone application that you can install plugins into, its pretty straightforward and looks pretty good too.

I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually a bit tricky to set up. I remember it taking a number of hours to accomplish. Events since have wiped out quite a bit of my recollection of the event. It was quite cool after I had configured everything properly, though.

At this point, I realized that what the COO really wanted wasn't a single blog, but the ability for many VPs to have a blog.

Back to the drawing board?

Not completely. At this point, I downloaded v1.5.1 of the WordPress MultiUser software. A default installation is just as simple as the single user version of the application.

Again I needed the Ldap plugin to enable integration with the Company's Lotus Notes LDAP service. This was actually very tricky to set up.

If you follow the above links to the Ldap plugins you'll discover that they are completely different. The wpmu-ldap plugin is different from the WordPress ldap plugin, written by different people.

The writer of the ldap plugin for WordPress MU has a blog here where he announced the release of the latest version. The maillists Aaron refers to at the bottom of his blog are an invaluable source of information, because to say the documentation is sparse is like saying that I'm an overweight bearded slaphead, i.e. a completely accurate and unbiased statement of fact.


Things I discovered whilst deploying WordPress MU and the ldap plugin are:
All the ldap files have to be owned by the httpd/apache/web server process owner. Otherwise the plugin isn't even seen. This is a file permissions problem, so not serious, but it can take an embarrassingly long time to track down. Or at least it did for me.

If after the WPMU ldap plug-in is enabled one of the files is edited by the root user and becomes owned by root, then the result is the infamous "White Screen of Death". Again, not something I immediately recognized. It took an embarrassingly long seeming hour to work it out!

The most obvious difference is that the single user wordpress plugin lets you specify the attribute to filter against, whereas the multiuser plugin lets you choose between linux LDAP and windows LDAP. Now the wpmu-ldap plugin maps linux to uid and windows to sAMAccountName. I was authenticating against Lotus Notes and needed cn! My only immediate option was to hack the source code.

Now its working its quite cool, but I did pick up some scars and a few more white hairs.

phpBB installation

This is yet another case where my mileage hasn't actually varied. But I had to write about the phpBB installation as it is just so damn slick.

It must be roughly 6 or7 years ago when I first installed and configured a phpBB site on a Solaris 8 server with MySQL v3.23.42, Apache v1.3.26 and php v4.0.6. Even then the install was pretty good, although it left enough techie stuff to be done that you felt you'd undertaken a "real man's job"! Afterall, I had to compile the Apache, MySQL and PHP distributions.

In this case, I created:

• a new VM on my development ESX server,
• loaded up CentOS v5.2 configured as a Web & MySQL server
• loaded up some additional php libraries
• started the httpd & mysqld services
• download and installed the latest phpMyAdmin
• created a DB
• created a DB User with appropriate priviledges
• started and finished the phpBB configuration very quickly

The phpBB configuration is performed via a web page. It recognises the current state of the installation and just steps you through it. When it has acquired all the relevant configuration details, it creates the tables in the database and sets up the initial admin account. And then you are in a position to start using the system.

Bob's your parental sibling, usually of the male variety!


This does slightly simplify the process, but only in terms of creating/deploying a new machine. I had to add a new server into the company QIP (now called VitalQIP) system and push that out to DNS.

After deploying a server, there's also planning that has to be undertaken for administration, usage policies, backup and restoration for the system.

Mediawiki VMware appliances

In general I really like downloading appliances from the VMware Community website. However, when I had to provision a Wiki for the entire company I quickly moved away from the appliances you can download.

The Jumpbox appliance is wonderful if you do not need or want anything further.

The rPath appliance is a bit is a bit more functional for an enterprise deployment. There is afterall a console that can be logged into. However the built-in OS upgrade didn't work. Via the website, it just hung. Via the commandline it ended up in an inconsistent state from which it could no longer be updated. D'Oh!

In the end I rolled my own from a CentOS Linux v5.1 Virtual Machine. I installed the latest MySQL, Apache, PHP (& loads of php libraries) and Mediawiki v1.12.0 - much further on than the v1.6.x version that the rPath appliance provided.

Part of the power and frustration of Mediawiki is the ability to extend the functionality using extensions.

I added a number of extensions to the instalation:

• Ldap Authentication (to the company MS 2003 AD)
• ImageMap
• WhosOnline
• etc

However, I battled the Ldap Authentication extension for most of a day and the ImageMap extension for most of an afternoon. The slightest mistake in the LocalSettings.php configuration file for Mediawiki or a missing library or misspelled filename would cause everything to fall apart. In contrast, WhosOnline was a dream to install. Its special page just loads and just works. Huzzah!

Trying to make the result pretty, like the Wikipedia homepage, is a whole different story. There is still knowledge required of the dhtml, css, etc.

The Initial Cost of VMware

I've written elsewhere abut the incremental cost of a VMware licence at the firm at which I work. But what about the initial cost?

There is the cost of:

• the new hardware
• training
• VMware software
• OS Licences
• additional Application Licences
• Administrator time!?!?

I try and use CentOS Linux for everything these days, but sometimes you need to use Windows. It takes just 5 to 10 minutes to create a new Windows VM from a template. However, just because you can doesn't mean you should. Each one of those Windows VMs will need a licence.

How do you factor in the cost of the mistakes you'll make with new technologies?

Some of these mistakes you will hope to avoid. Who makes mistakes? Or admits to them anyway?


Administrator time is a constant surely? I'm joking, but sometimes it does seem like there is a belief that a finite group can undertake infinite work.

There is also the opportunity cost of your Administrators not just improving and optimizing your existing infrastructure.

Training!? Choosing the right training first off for a new technology is difficult. For VMware, I'd recommend Elias Khnaser's Training DVDs. Having used VMware for nearly two years now, there are holes in his coverage. However, he also covers some topics I've yet to need to get involved with.

And new hardware! Well that wasn't a road we went down. To start with our development box was an IBM 366 with an EXP400 that had been forklifted from site to site to site. The downside, which we've had to explore extensively, is that as it originated in California IBM are unable supply a replacement motherboard in this country - a 366 in this country has different part numbers! In the long run, completely new hardware might have been the cheaper option, simply in Administrator time. Whilst we were getting the server back up, we weren't doing other more productive work!

So it goes.

Lotus Notes install on CentOS Linux

Actually this is a case where my mileage didn't vary at all.

I needed to install IBM's Lotus Domino server software onto a Linux machine - for once these days real iron not a virtual machine.

Obviously, I started with the latest version of CentOS - v5.1. I had no choice on the version of Lotus Domino - it had to be v6.5.5. It had to match our windows servers.

I ran into the problem of libstdc++-libc6.1.1.so.2 being missing which I resolved by loading the compat-libstdc++-296.2.96 rpm and linking the missing library to the later library provided by that rpm.

This still didn't resolve the problem. I still had a missing library: libXp.so! A quick yum install -y libXP resolved that. However, the Lotus Domino java installation program now just hung trying to create a session.

I ran out of time. A swift look at the IBM website for OS requirements for Lotus Domino installation and I conceded defeat. Next step I downloaded the CentOS v2.1 !! That was another 20 to 30 minutes to install and then it was back to the Lotus Domino install.

The only alternative would have been to use version 8 of the Lotus domino software. Something that just wasn't possible. Sigh!