Solaris Zone down

I made a minor configuration change to a Solaris zone and rebooted the zone.

On the way back up it hung with an error message about booting.

I entered zoneadm list -cv with the following results

root@server01 # zoneadm list -cv

ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 licence3-zone running /export/home/zones/licence3-zone native shared
2 licence2-zone running /export/home/zones/licence2-zone native shared
5 licence-zone running /export/home/zones/licence-zone native shared
7 licence1-zone running /export/home/zones/licence1-zone native shared
21 la-zone running /export/home/zones/la-zone native shared
23 build-machine down /export/home/zones/build-machine native shared
24 build-2 running /export/home/zones/build-2 native shared
- build-1 installed /export/home/zones/build-1 native shared
root@server01#
Its state was shown as down!

I entered
zoneadm -z build-machine halt
but it failed with a message saying the zone's /tmp directory couldn't be unmounted.

Entering
zoneadm -z build-machine reboot
failed with a similar message.

Entering
zoneadm -z build-machine boot
was the same.

So I traversed the zone's filesystem from the global zone. I'd downloaded some files to the zone's /tmp directory. I deleted the entire contents and entered
zoneadm list -cv
Its state was still shown as down!
I entered zoneadm -z build-macxhine halt, which returned without error. A zoneadm list -cv showed some good news.

root@server01 # zoneadm -z build-machine halt
root@server01 # zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 licence3-zone running /export/home/zones/licence3-zone native shared
2 licence2-zone running /export/home/zones/licence2-zone native shared
5 licence-zone running /export/home/zones/licence-zone native shared
7 licence1-zone running /export/home/zones/licence1-zone native shared
21 la-zone running /export/home/zones/la-zone native shared
24 build-2 running /export/home/zones/build-2 native shared
- build-machine installed /export/home/zones/build-machine native shared
- build-1 installed /export/home/zones/build-1 native shared
root@server01 #

And it booted successfully. Huzzah!

Well, that's that!

New Solaris Resource

Whilst searching for information on a specific feature of IP-Filter, I came across a new resource of Solaris Information.

My only worry, other than with Oracle taking SUN over Solaris will wither on the vine, is that a lot of this information may be quite old, if I was solely to judge the site from the Solaris logo!

I'm going to add the site to my useful links. And that's that!

Some good links

I came across a blog just today by following a link on Scott Lowe's bookmarks: Linux Performance Tuning

This is really useful information. All pulled together in one place. Some of it might seem common sense, but as has frequently been said it is surprising how uncommon common sense can be.


As I wrote in my previous post about recovering from a disaster, documentation on your environment is vital. And these two links below provide that.


I've only just noticed that RVTools has been considerably updated. This is an excellent tool and the more so for being free. I feel a bit guilty about  not being in a position to donate some of my employers' money via paypal.

Actually, the new feature that I'm most excited by is one that has been there for several versions now. Specifically, the ability to export all the information in csv format. And to do that from the command line!


I was browsing sunfreeware and came across sys_diag, a script generated by Todd Jobson. For doumenting the state of a Solaris server, it looks damn comprehensive. I intend to use it on the Solaris systems at my work and save the results into our database of server information.

An additional thought about whitespace and LDAP

I will have to test it, but it could be that the problem I have with using

ldapclient -v manual \
-a credentialLevel=self \
-a authenticationMethod=sasl/gssapi \
...

is that as all the users are in Active Directory OUs named things like "EMEA ENG" or "APR ENG".

As I reported earlier I had a number of problems with whitespace in the ldapclient command line. It could be that this was another problem with whitespace. However, this problem was masked by occurring at a stage where it appeared everything was working successfully.

Hopefully I can rename the OUs to be EMEA, APR, etc and resolve the issue.

Whitespace in Solaris 10 LDAP configuration

Spaces are allowed in the ldapclient command line if attribute is surrounded by "", i.e.
-a "proxyDN=cn=admin,cn=emea users,dc=example,dc=com"
or
-a "defaultServerList=123.123.123.1 123.123.123.2"

Failing to quote either attribute statement will cause the ldapclient command line to fail with a parsing error.

However, there are instances where quoting the attribute definition will cause the command line parsing to work and for the command to succeed but ldap lookups can still fail.
May 22 18:09:44 server1 nscd[4012]: [id 293258 user.error] libsldap: Status: 49 Mesg: openConnection: simple bind failed - Invalid credentials

The problem is resolved when the proxy user is replaced with another from an OU which doesn't contain a space, i.e.
-a "proxyDN=cn=admin,cn=users,dc=example,dc=com"

Netmask settings for Solaris 10 Zones

When the zone is created a large number of files are copied into the new zone. However, the netmask file seems to be generated as an empty file.

Consequently, when you enter ifconfig you end up seeing something like:
...
eri0:2: flags=1000843 mtu 1500 index 2
zone zone2
inet 123.123.123.11 netmask ffff0000 broadcast 123.123.123.255
...

Now obviously you can use ifconfig in the global zone to change the netmask setting interactively. That works. However, a reboot wipes that out. So you have to login to the console of the zone and amend the copied netmask file so that it contains a line like:
123.123.123.0 255.255.255.0

Alternatively, you can access the netmasks file via the global zone filesystem.

Legato Networker Configuration Duplication

I made a mistake whilst installing Legato Networker on a Solaris 10 box. The mistake prevented the backup server from backing up the Solaris 10 client.

I support multiple DNS domains and the backup server appears in many of those domains and as the backupserver is multihomed it appears in those domains with different IP addresses.

The Solaris 10 client was in Domain dom1.example.com, one of the few sub-domains the backup server actually isn't in!

When I installed NetWorker I specificed that the backupserver as backupserver.dom2.example.com. The backupserver was actually trying to communicate with the Solaris 10 client over the network where it was known as backupserver.dom3.example.com.

Networker didn't like the loop effect of this arrangement and the backup was failing.

In fact I should have just specified a simple server name, unqualified by a domain name and added the backupserver into hosts file.

D'Oh!

However, it took a little longer to resolve than I had expected because Networker not only records the backup server in the /nsr/res/servers file but also records it several times in the /nsr/res/nsrla.res file.

As I said above, I would have saved myself some confusion by entering the unqualified name of the backup server upon installation.

*nix & Windows integration software I'd like to use

Vintela have some pretty cool software for integrating Solaris & Linux systems into an AD environment.

Perhaps because it has to compete against Samba and other freeware solutions, their products aren't ridiculously expensive either. Quite a refreshing experience. Its a shame that the normal mode of operation for most software houses is to seek to soak their customers. No names! No packdrill! But we all know who I mean.

Perhaps the coolest feature is the ability to apply GPOs to *nix clients. Add in the inexpensive nature of the software, and it seems pretty compelling.

What's the downside?

For my company, its the fact that our AD Servers aren't running Windows 2003 R2, which is a requirement of the solution.

Solaris 10 authenticating against Active Directory

There a number of good blogs discussing this subject. I'd recommend Scott's and the OpenSolaris blogs.

I used Scott Lowe's blog for the instructions on how to get CentOS Linux machines to authenticate against Active Directory. It was right on the money. Especially as I needed my machines to run Samba to create an interoperability solution for a number of software development teams who use IBM Rational ClearCase.

However, his instructions for Solaris 10 servers never worked in my environment.

My environment is Windows 2003 Active Directory with all the servers have been patched with service pack 2 and the latest monthly patches. The Server for NIS and Password Synchronization modules of Services For Unix v3.5 has also been installed, which obviously had extended the schema. As Service Pack 2 had been installed, the hotfix that fixes passwd sync after the "upgrade" has also been applied. N.B. the servers are not running Windows 2003 R2 - that would make a big difference and from all accounts it would be much easier to interoperate with.

There is an article on BigAdmin on this subject. The method described almost worked for me. I'd say it went 90% of the way. The part that didn't work was the ldapclient command. Specifically, it was trying to usecredentialLevel=self with authenticationMethod= sasl/gssapi , i.e.

ldapclient -v manual \
-a credentialLevel=self \
-a authenticationMethod=sasl/gssapi \
...


I was able to get around this by changing the ldapclient command to:

ldapclient -v manual \
-a credentialLevel=proxy \
-a authenticationMethod=simple \
-a proxyDN=cn=proxy_user,cn=users,dc=example,dc=com \
-a proxyPassword=password \
...


I also had to change the serviceSearchDescriptor attributes from

-a serviceSearchDescriptor=passwd:cn=users,dc=example,dc=com?one \
-a serviceSearchDescriptor=group:cn=users,dc=example,dc=com?one

to

-a serviceSearchDescriptor=passwd:dc=example,dc=com?sub \
-a serviceSearchDescriptor=group:dc=example,dc=com?sub


That done and Bob was my parental Sibling of the usually male variety!

gcc 64bit compilation on Solaris 10

By default, if you were to build a shared library with gcc, you'd enter commands similar to

# gcc -fPIC -c file.c
# gcc -shared -o file.so file.o

The resultant shared library would be 32 bit, i.e.
# file file.so
file.so: ELF 32-bit MSB dynamic lib SPARC Version 1, dynamically linked, not stripped, no debugging information available
#

If you wish to build a 64 bit library you should amend the command above as follows:

# gcc -fPIC -m64 -mcpu=ultrasparc -c file.c
# gcc -shared -m64 -mcpu=ultrasparc -o file.so file.o

With the result:
# file file.so
file.so: ELF 64-bit MSB dynamic lib SPARCV9 Version 1, UltraSPARC1 Extensions Required, dynamically linked, not stripped, no debugging information available
#

Changing hostids for Solaris 10 Zones

I found 3 resources on the Internet which discuss changing the hostid of a Solaris instance.

Only two are specifically related to the case of Solaris Zones. The other is a more general "You can change the hostid..." type of resource.

In my experience with Solaris 10 Zones, only one of these methods succeeded.

Initially I attempted the "Dynamic Library Interposition method described by Julien Gabel on his Blog'o thnet. Initially this held promise.

1. I compiled the code
2. I set the environment variable
3. I ran the code

Success!!

I added the environment variable to an existing startup script. It failed!

The error indicated the library was the wrong type! As I had installed the 64-bit version of Solaris 10, I re-built the dynamic library as 64-bit. Received the same error!


At this point, I actually created my zones, following the outline provided by this Zones Tutorial. The tutorial on how to create a Solaris 8 Zone on a Solaris 10 Server describes how to set an attribute of the Zone as the hostid. Perhaps this works when you really do have a Solaris 8 Zone on your Solaris 10 Server, but it didn't work for my Solaris 10 zones on a Solaris 10 Server.


Finally I resorted to the other mechanism for altering hostids described upon Julien Gabel's Blog'o thnet - daemonizing a DTrace script. This did work. In fact it works very well. Much Kudos should be directed towards Brendan Gregg.