Replacing a disk in a Sun D2 Array

The Sun D2 Array is hot swap capable. As we control the D2s using Disksuite, a.k.a. SVM, it is possible to replace a disk in a D2 without requiring that the server be shutdown.

When you receive notification that a disk has died. First run metastat to determine with disk has been swapped out
# metastat
d0: Mirror
    Submirror 0: d10
      State: Okay        
    Submirror 1: d20
      State: Okay        
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 18876726 blocks


d10: Submirror of d0
    State: Okay        
    Size: 18876726 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t0d0s0                   0     No    Okay        




d20: Submirror of d0
    State: Okay        
    Size: 18876726 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t1d0s0                   0     No    Okay        




d1: Mirror
    Submirror 0: d11
      State: Okay        
    Submirror 1: d21
      State: Okay        
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 33555735 blocks


d11: Submirror of d1
    State: Okay        
    Size: 33555735 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t0d0s1                   0     No    Okay        




d21: Submirror of d1
    State: Okay        
    Size: 33555735 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t1d0s1                   0     No    Okay        




d3: Mirror
    Submirror 0: d30
      State: Okay        
    Submirror 1: d31
      State: Okay        
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 355604121 blocks


d30: Submirror of d3
    State: Okay        
    Hot spare pool: hsp000
    Size: 355604121 blocks
    Stripe 0: (interlace: 64 blocks)
        Device              Start Block  Dbase State        Hot Spare
        c2t0d0s0                   0     No    Okay        
        c2t1d0s0                2889     No    Okay        
        c2t2d0s0                2889     No    Okay        
        c2t3d0s0                2889     No    Okay        
        c2t4d0s0                2889     No    Okay        




d31: Submirror of d3
    State: Okay        
    Hot spare pool: hsp000
    Size: 355604121 blocks
    Stripe 0: (interlace: 64 blocks)
        Device              Start Block  Dbase State        Hot Spare
        c2t5d0s0                   0     No    Okay         c2t12d0s0
        c2t8d0s0                2889     No    Okay        
        c2t9d0s0                2889     No    Okay        
        c2t10d0s0               2889     No    Okay        
        c2t11d0s0               2889     No    Okay        




hsp000: 2 hot spares
        c2t12d0s0               In use          71124291 blocks
        c2t13d0s0               Available       71124291 blocks


#

OK, c2t5d0s0 is the sixth disk from the left in the array - or to put that another way it is the left one of the two middle disks!

You can run format->analyse->read which will determine whether the disk is really dead.
# format
Searching for disks...done




AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde452,0
       1. c1t1d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde367,0
       2. c2t0d0
          /pci@8,600000/pci@1/scsi@4/sd@0,0
       3. c2t1d0
          /pci@8,600000/pci@1/scsi@4/sd@1,0
       4. c2t2d0
          /pci@8,600000/pci@1/scsi@4/sd@2,0
       5. c2t3d0
          /pci@8,600000/pci@1/scsi@4/sd@3,0
       6. c2t4d0
          /pci@8,600000/pci@1/scsi@4/sd@4,0
       7. c2t5d0
          /pci@8,600000/pci@1/scsi@4/sd@5,0
       8. c2t8d0
          /pci@8,600000/pci@1/scsi@4/sd@8,0
       9. c2t9d0
          /pci@8,600000/pci@1/scsi@4/sd@9,0
      10. c2t10d0
          /pci@8,600000/pci@1/scsi@4/sd@a,0
      11. c2t11d0
          /pci@8,600000/pci@1/scsi@4/sd@b,0
      12. c2t12d0
          /pci@8,600000/pci@1/scsi@4/sd@c,0
      13. c2t13d0
          /pci@8,600000/pci@1/scsi@4/sd@d,0
Specify disk (enter its number): 7
selecting c2t5d0
[disk formatted]


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !      - execute , then return
        quit
format> anal


ANALYZE MENU:
        read     - read only test   (doesn't harm SunOS)
        refresh  - read then write  (doesn't harm data)
        test     - pattern testing  (doesn't harm data)
        write    - write then read      (corrupts data)
        compare  - write, read, compare (corrupts data)
        purge    - write, read, write   (corrupts data)
        verify   - write entire disk, then verify (corrupts data)
        print    - display data buffer
        setup    - set analysis parameters
        config   - show analysis parameters
        !    - execute , then return
        quit
analyze> read
Ready to analyze (won't harm SunOS). This takes a long time,
but is interruptable with CTRL-C. Continue? y

        pass 0
   24619/26/53 

        pass 1
   24619/26/53 

Total of 0 defective blocks repaired.
analyze> q


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !      - execute , then return
        quit
format> q
#
The excerpt above didn't show any repairs, but they may be some when you run the command.


Raise a call for the new disk under your support contract. Or failing that search the server room for a compatible disk - Good Luck!

As long as a hot spare has jumped in, the dead disk can just be removed from the array and the new one inserted. The disk will spin up immediately.

Wait for the green light to come on. And Bob's your uncle!

Run format to apply a disk label. You won't be able to format the disk until you do.
# format
Searching for disks...done

c2t5d0: configured with capacity of 33.92GB


AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde452,0
       1. c1t1d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde367,0
       2. c2t0d0
          /pci@8,600000/pci@1/scsi@4/sd@0,0
       3. c2t1d0
          /pci@8,600000/pci@1/scsi@4/sd@1,0
       4. c2t2d0
          /pci@8,600000/pci@1/scsi@4/sd@2,0
       5. c2t3d0
          /pci@8,600000/pci@1/scsi@4/sd@3,0
       6. c2t4d0
          /pci@8,600000/pci@1/scsi@4/sd@4,0
       7. c2t5d0
          /pci@8,600000/pci@1/scsi@4/sd@5,0
       8. c2t8d0
          /pci@8,600000/pci@1/scsi@4/sd@8,0
       9. c2t9d0
          /pci@8,600000/pci@1/scsi@4/sd@9,0
      10. c2t10d0
          /pci@8,600000/pci@1/scsi@4/sd@a,0
      11. c2t11d0
          /pci@8,600000/pci@1/scsi@4/sd@b,0
      12. c2t12d0
          /pci@8,600000/pci@1/scsi@4/sd@c,0
      13. c2t13d0
          /pci@8,600000/pci@1/scsi@4/sd@d,0
Specify disk (enter its number): 7
selecting c2t5d0
[disk formatted]
Disk not labeled.  Label it now? y


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !      - execute , then return
        quit
format> q
#

Enter a prtvtoc/fmthard command combination to ensure the disk has the same slices as the replaced disk. In the command below I use the disk that is in the equivalent position on the other side of the mirror as the source of the configuration.
# prtvtoc /dev/rdsk/c2t0d0s0 | fmthard -s - /dev/rdsk/c2t5d0s0
fmthard:  New volume table of contents now in place.
#


Check the disk format:
# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde452,0
       1. c1t1d0
          /pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cffde367,0
       2. c2t0d0
          /pci@8,600000/pci@1/scsi@4/sd@0,0
       3. c2t1d0
          /pci@8,600000/pci@1/scsi@4/sd@1,0
       4. c2t2d0
          /pci@8,600000/pci@1/scsi@4/sd@2,0
       5. c2t3d0
          /pci@8,600000/pci@1/scsi@4/sd@3,0
       6. c2t4d0
          /pci@8,600000/pci@1/scsi@4/sd@4,0
       7. c2t5d0
          /pci@8,600000/pci@1/scsi@4/sd@5,0
       8. c2t8d0
          /pci@8,600000/pci@1/scsi@4/sd@8,0
       9. c2t9d0
          /pci@8,600000/pci@1/scsi@4/sd@9,0
      10. c2t10d0
          /pci@8,600000/pci@1/scsi@4/sd@a,0
      11. c2t11d0
          /pci@8,600000/pci@1/scsi@4/sd@b,0
      12. c2t12d0
          /pci@8,600000/pci@1/scsi@4/sd@c,0
      13. c2t13d0
          /pci@8,600000/pci@1/scsi@4/sd@d,0
Specify disk (enter its number): 7
selecting c2t5d0
[disk formatted]


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !      - execute , then return
        quit
format> p


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 - 24618       33.91GB    (24619/0/0) 71124291
  1 unassigned    wu       0                0         (0/0/0)            0
  2     backup    wu       0 - 24619       33.92GB    (24620/0/0) 71127180
  3 unassigned    wu       0                0         (0/0/0)            0
  4 unassigned    wu       0                0         (0/0/0)            0
  5 unassigned    wu       0                0         (0/0/0)            0
  6 unassigned    wu       0                0         (0/0/0)            0
  7 unassigned    wm   24619 - 24619        1.41MB    (1/0/0)         2889

partition> q


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !      - execute , then return
        quit
format> q
#

Replace the hotspare disk with the original - the replacement command works on the mirror not the submirror which actually has the failed disk!:
# metareplace -e d3 c2t5d0s0
#

Use metastat | grep to check when the mirror has finished resync-ing
#  metareplace -e d3 c2t5d0s0
d3: device c2t5d0s0 is enabled
# metastat
d0: Mirror
    Submirror 0: d10
      State: Okay        
    Submirror 1: d20
      State: Okay        
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 18876726 blocks

d10: Submirror of d0
    State: Okay        
    Size: 18876726 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t0d0s0                   0     No    Okay        


d20: Submirror of d0
    State: Okay        
    Size: 18876726 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t1d0s0                   0     No    Okay        


d1: Mirror
    Submirror 0: d11
      State: Okay        
    Submirror 1: d21
      State: Okay        
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 33555735 blocks

d11: Submirror of d1
    State: Okay        
    Size: 33555735 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t0d0s1                   0     No    Okay        


d21: Submirror of d1
    State: Okay        
    Size: 33555735 blocks
    Stripe 0:
        Device              Start Block  Dbase State        Hot Spare
        c1t1d0s1                   0     No    Okay        


d3: Mirror
    Submirror 0: d30
      State: Okay        
    Submirror 1: d31
      State: Resyncing   
    Resync in progress: 0 % done
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 355604121 blocks

d30: Submirror of d3
    State: Okay        
    Hot spare pool: hsp000
    Size: 355604121 blocks
    Stripe 0: (interlace: 64 blocks)
        Device              Start Block  Dbase State        Hot Spare
        c2t0d0s0                   0     No    Okay        
        c2t1d0s0                2889     No    Okay        
        c2t2d0s0                2889     No    Okay        
        c2t3d0s0                2889     No    Okay        
        c2t4d0s0                2889     No    Okay        


d31: Submirror of d3
    State: Resyncing   
    Hot spare pool: hsp000
    Size: 355604121 blocks
    Stripe 0: (interlace: 64 blocks)
        Device              Start Block  Dbase State        Hot Spare
        c2t5d0s0                   0     No    Resyncing   
        c2t8d0s0                2889     No    Okay        
        c2t9d0s0                2889     No    Okay        
        c2t10d0s0               2889     No    Okay        
        c2t11d0s0               2889     No    Okay        


hsp000: 2 hot spares
        c2t12d0s0               Available       71124291 blocks
        c2t13d0s0               Available       71124291 blocks

# metastat d3 | grep "Resync in progress"
    Resync in progress: 5 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 5 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 6 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 6 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 8 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 51 % done
# metastat d3 | grep "Resync in progress"
    Resync in progress: 60 % done
#


Just repeat that last metastat command until you get no output and the resync-ing will have completed.


And that's that!

Further Good Links/Tools

In a previous post, I described some tools that I had found useful for documenting the system environment.

A pretty comprehensive system description of many versions of the Windows OS can be generated by SIW - Systems Information for Windows, which was written by Gabriel Topala. It is only free for personal use. The pricing model for Businesses is in the right sort of ball park.

If you are a Legato Networker user and you need to generate reports for either your own benefit or the benefit of others, then the Networker Reporting Utility is well worth a look.The download mechanism is rather painful, but it is worthwhile. There is a product from Legato which does a similar job, but for our environment we were looking at having to spend upwards of £30,000 to £40,000! That decision was pretty much a no-brainer!

And that's that for now!

What a PITA!!

This week I've been configuring a couple of new HP servers, ProLiant DL 360 G6s to be precise. They were configured to have 8 internal disks so the internal CD/DVD-ROM had to be sacrificed.

So to configure them, I temporarily attached a USB CD_ROM drive, Keyboard and Mouse and attached a monitor.

Having installed the latest CentOS x86_64 Linux from CD on the first one, I removed the CD-ROM and rebooted. It hung coming up configuring the USB storage driver!

So I spent a day googling for help. I re-installed Linux half a dozen or so times. I downloaded the DVD and followed the instructions to make that available via both HTTP and NFS and tried using the first CD to boot from and then installing across the network. Actually that was considerably faster than the CD method!


Nothing changed the hang during boot. Always on the USB storage driver installation.

As a punt with nothing else to lose, I unplugged the USB keyboard and mouse and rebooted.

The b*$#*^d box booted all the way.

I plugged the keyboard in and it was recognized without fuss and was immediately useful.

So it goes! Some days are just like that.

Disaster Recovery 101 Part 2

So in part 1, I promised to discuss some of the "soft" requirements to be considered when preparing for a recovery from disaster.


The following documentation should be held electronically. These days it is probably more difficult not to hold documentation electronically.


Documentation of your hardware maintenance contracts
The purpose of this is not to fix your lost equipment, but to immediately take the lost servers off the contract and to later update it with their replacements. This might not seem like a priority, but the larger the company the larger the cost saving that performing this action will save. Should you manage to recover some of your servers you'll need to know this information to raise support calls.

This action presupposes that you have negotiated your contract so that you can add and remove items during the life of the contract. If you haven't already, you should start doing so from your next renewal date.


Your policies & procedures.
Some might argue that a clean slate is the perfect opportunity to start again. As creating these documents isn't the most fun activity in the known universe, I have some sympathy with this idea. However it really should be resisted. It will have taken you some considerable time to develop those policies & procedures. Some policies might need tweaking, some might need obsoleting, but they are a gold mine of information about your environment.


Good Supplier Relationships
Another "soft" requirement. If it isn't obvious why this is a requirement for disaster recovery, consider that on that Sunday morning I mentioned in part 1 that by 11:00am one of our two main hardware suppliers had:

• opened their offices
• provided us with internet access
• provided us with hardware that had been purchased by and for someone else!
• provided us with lab space; phones; electricity; etc

and were arranging with their security company to allow us to stay through the night whilst we worked at building and recovering our backup system from the backup tapes and some installation media.

(Obviously, we did later pay for the hardware. Whether the original purchasers were ever told, I do not know.)

Of course, a good supplier relationship is not something that can't be magic-ed out of a hat first thing on the morning of your disaster. Good supplier relationships are an ongoing concern. That doesn't mean that you overpay for goods and services. That isn't a good relationship. That is being a doormat. It also doesn't mean screwing them over on every deal. It does mean being open with them. Working with them over a long time so that they understand your requirements; that sometimes every quote doesn't lead to a purchase; that company rules require that you get quotes from other suppliers too!


Offsite storage for the backup tapes and all the other documentation above
Offsite storage for your backup tapes is fairly standard. But how frequently do the tapes go offsite? It needs to be daily during the week! If your company is large enough to be able to afford weekend shifts, then you might also want to investigate weekend pickups as well.

Every month on the first day of the month my admin server sends me an email. That email reminds me to burn all the latest documentation from the intranet site down onto a DVD. That DVD then stays in my laptop bag until the next month.

Some of the documentation is stored in a number of Lotus Notes databases. These are even more ideal from a DR perspective. (It is a shame that IBM have made a hash of marketing Lotus Notes - some of its features are ideal for enterprises of any size. But that is a story for another blog post.) You can just make a local replica of the database onto your PC. Whatever the contents of that database. And it can be kept in step via replication, which can be as often as you like. Or never after the initial replication.

At a time when my company operated a campus of multiple building, and indeed multiple sites, a firesafe in one of the other buildings was considered offsite.


How to recover your environment.
Given your backup tapes and an empty room, would you know where to start/how to start to rebuild your environment? This is a question Joel Spolsky covered quite cogently in a post just before Christmas. Doing the backup is part of the bread and butter of the job. But so should be the restore.

Even with all the knowledge of your environment that you should have documented, the answer to the question of which servers to restore first will be similar to the the start-up order of your datacentre. Similar, but unlikely to be exactly the same.

Of course, the shutdown and startup orders will be part of the documentation listed under "Description of Inter-relationships" described in part 1.

Most "old lags" in IT will have a good idea of which systems need to be restored first and how to do so. Hopefully, there will have been an exercise in how to to restore individual servers and systems over time.

In part 1 of this series of posts on disaster recovery, I listed some/most of the information you should keep for each server. However, I missed some items out:
It is just about essential to list what is backed up & how to recover the server with that dataset.
And when you have recovered your server, how do you know it is recovered? How can you prove it has been recovered successfully?

Document a series of tests that will exercise the functionality of the server/system fully or at least to some acceptable (to you and justifiable to others) level of completeness. Generally this information is referred to as return to service information (RTS).


Knowledge of the company's insurance policy
This might not be regarded as an IT responsibility. In some companies this might be a site services or facilities management responsibility or a Financial or Legal Dept. concern. In smaller companies, the office manager might be responsible.
In fact, Id agree it wasn't an IT responsibility or shouldn't be. But if you are responsible for the company's IT infrastructure, you should make yourself aware of whether your company actually has Critical Incident Insurance or whether your company is large enough to carry the risk itself.

The answer will help you prepare. If the replacement cost of your infrastructure is US$2million and your comapany has no insurance, then the business should know that up to that amount will have to be found/provided in a disaster.

If the company does have insurance, then it is necessary to keep that policy up to date with the value of the company's infrastructure.


Multiple sites
In theory, having multiple sites should enable you to provide resilience through replication of  information to the other sites. It depends upon your level of risk and the budget available to you whether you implement replication.

But it is possible to mitigate a lot of risk through data replication between sites. At one stage, there was only the UNIX utilities: rdist and then rsync to accomplish the task. But they work at the file level. Then a lot of companies worked out how to accomplish this task at the block level. NetApp were possibly the first - the first I was aware of anyway. but it now appears to be a common facility in every venders' repertoire.

Both free and paid for dbms offer varieties of replication, master/slave and master/master. One of the best database replication mechanisms seems to be that used by Lotus Notes. But Lotus Notes isn't suitable for all applications. Plus IBM doesn't seem to have known how to market it. Actually IBM frequently doesn't appear to know how to market anything. Anyway, it should be possible to set up your database applications, to be location independent.


Well, that was part 2. After part 1, I stated there would be an additional two parts, Whilst finalising this part, I realised some of the issues I overlooked. There may well be a part 4. It depends upon how part 3 goes.

Downsides of Virtualisation?

This post possibly falls into the "stating the bleeding obvious " category! However,...

Reducing server count isn't just about reducing the amount of hardware. But, if you have virtualised a bunch of servers will you be so keen to look into combining their functions?
There still can be value in combining the functions of two virtual servers. There are the management benefits of fewer servers to manage and if you have software that is licensed on a per server basis, then there will be financial benefits too.

Eventually, you will develop an architectural dependence on servers that are virtualised. If your mail relay is virtualised and your payroll system isn't, then should your payroll system need to send email, a lot of the infrastructure needed for your virtualisation environment will need to be up and running too.
In most cases the virtual environment will be up and running. However, there are periods when you may be recovering from a disaster when that isn't the case.

Are these reasons not to virtualise?

Of course, not!
But they are factors to consider and factor in to your decision making.

Some good links

I came across a blog just today by following a link on Scott Lowe's bookmarks: Linux Performance Tuning

This is really useful information. All pulled together in one place. Some of it might seem common sense, but as has frequently been said it is surprising how uncommon common sense can be.


As I wrote in my previous post about recovering from a disaster, documentation on your environment is vital. And these two links below provide that.


I've only just noticed that RVTools has been considerably updated. This is an excellent tool and the more so for being free. I feel a bit guilty about  not being in a position to donate some of my employers' money via paypal.

Actually, the new feature that I'm most excited by is one that has been there for several versions now. Specifically, the ability to export all the information in csv format. And to do that from the command line!


I was browsing sunfreeware and came across sys_diag, a script generated by Todd Jobson. For doumenting the state of a Solaris server, it looks damn comprehensive. I intend to use it on the Solaris systems at my work and save the results into our database of server information.

Disaster Recovery 101 Part 1

Danger, Will Robinson! Danger...!

Ever since I have been involved in IT, management have been concerned with producing a disaster recovery plan. Inevitably, the hearts, morale and motivation of most of the staff involved has sunk deeper than the bottom of the Marianas Trench.

Management have inevitably sought a tome large enough for any doorstep - your basic shelf-ware, in fact.

However, having lived and worked through as big a disaster as any in the UK - I took a call from my boss at 7:00a.m. on a Sunday morning telling me "the building is gone", this is what I've learnt on the job. Both what we had done before and some things I wish we had done.

Some of the  requirements covered over the initial three posts I'll be making on this topic could be considered "soft" requirements, as in soft systems methodology, as they aren't focused on hardware, software or physical artifacts. And some of these sections could be the basis of a major article or blog posting. Heck! There are any number of books on Amazon on the subject of Disaster Recovery for IT systems.


Now, to start with, you need to know what you have and therefore what you might have lost.

So have a
Full Hardware Inventory
List your servers:

• make
• model
• number of cpus
• amount of RAM
• number and type of NICs
• number and type of HBAs
• internal storage
• external storage
• OS Version
• IP Addresses
• MAC Addresses
• MAC addresses
• hostid
• purchase order number
• purchase date
• maintenance contract number
• serial number
• asset tag
• etc

The internal and external storage descriptions should be stated both in terms of the disks used and the partitions built on top of them. It is also necessary to document the RAID schemes used and whether RAID is via the hardware or software.

List all the networking equipment: routers, switches, firewalls, load balancers, network appliances whether for web caching, spam filtering, mail relays, dns, etc. There will be a reason for making any configuration change to these systems. If you've opened a port through your firewall, there will be a very good business reason for doing so. In some geographies, e.g. the US, you will be required to record and keep that information for auditing and compliance control in a non-editable format.You can usually make comments against your firewall rules - at least you can with checkpoint - and it is good practise to give firewall change requests an unique reference number to include in the comment in the firewall rules. Then it will be possible to not only take a firewall change request and find out if rules exist for it, but it will also be possible to trace back from a rule to the request that caused its creation.

List all the storage networking equipment: NetApp, EMC, Sun StorageTek, etc
Again, record partition sizes, IP addresses, maintenance contracts and permissions

List all the Facilities "stuff", i.e. Air conditioning units, UPS, racking. Depending upon your company and the extent of your loss, then you might also list additional items like printers, scanners, photocopiers and multi-function devices. Perhaps also webcams. Although webcams aren't often allowed in the office anymore.

Basically, you want as much detail as possible. You aren't going to want to purchase the exact self same hardware, but this information will inform any purchasing decisions. You'll understand what your required processing and storage capacities are or were.

Additionally, you'll also need a
Full Server Listing
With virtualisation in whatever guise becoming almost mandatory, it will be clear what the difference is between a full Hardware Inventory and a Full Server Listing. It is still necessary to document  the information that is being listed in the previous section above for each server.


Full Software Inventory
If you have been following ITIL you should have a Definitive Software Library (DSL), which will contain all your required OS and Application software installation media and any significant updates. Even where a DSL exists, individual engineers will have installation media. Unless you have very fussy applications software, it is probably not necessary to define individual patch levels for various components of the OS. Although, documenting any  patches or software updates that should never be applied might well be valuable, e.g. the company I work for still has to use ie6 (*sigh*) allegedly because of SAP, so Microsoft updates for ie7 and ie8 are blocked from download. I like to cross-reference systems against software in addition to software against systems. A spreadsheet isn't always the best mechanism to maintain the referential integrity. A database which is designed to support such references and searching is a lot more useful.


Review of Inventory Requirements
The information listed above shouldn't just be kept for DR. It can/should be used to:

• generate your annual hardware maintenance requirements
• generate your annual software maintenance requirements.
• determine candidates for hardware upgrades
• determine candidates for OS and application software upgrades
• identify assets during audits
• monitor for capacity planning
• monitor the software for feature review, i.e. if you are about to renew the maintenance of a tool would you be better off with a new utility? e.g. we have used vRanger Pro for a couple of years but apparently  Veeam Backup and Recovery is now a better product.
• identify staff training requirements
• identify staff hiring requirements

Consequently, this information should always be gathered or generated and kept up to date. And stored both on and securely offsite.

Now your monitoring system will almost certainly be saving the system state for capacity analysis and planning. In any large enterprise, the realistic timeframes of interest are the last quarter and the last year. You have to plan your spending a year ahead. Capital and expense spending for each quarter will be reviewed quarterly. Your documentation, monitoring and planning should be reviewed over the same periods.


Description of Inter-relationships, i.e identification of systems
So you know you hardware, your servers, your aplications, now how do they hang together as systems? If this has been mismanaged in the past, then Tideway or someone similar will sell you some software and may even come in and perform a network discovery for a fee. However, up to a certain size, you should be able to accomplish most of the same yourself. Especially if you undertake this task as you go along. every time you add a new element to your infrastructure or you simplify something: document it. Thoroughly!

However, systems may extend further than you initially consider.
The IBM Rational software configuration management tool, ClearCase, can have the following components: VOB servers, view servers, build servers, registry servers and licence servers. Some servers may have more than one function. However, ClearCase is dependent upon an OS for security and id management. So, in a Windows environment it is dependent on the AD and in a UNIX environment, is dependent upon NIS, NIS+, LDAP or similar. And in a multi-platform development environment it is dependent on both. However, in many installations ClearCase will be teamed with ClearQuest, IBM Rational's Defect Tracking system. And sometimes it will also be teamed with a requirements management system like DOORS or Requisite Pro. These integrations and others will then extend the system to database and web servers as well as client programs running on engineer's desktops. All these interconnections have to be documented.


Networking Information
The company I work for has been allocated 8 Class B and a further 41 Class C networks.
I do not say that to gloat. Although,... I am aware of Google's work to popularise IPv6, where the main finding was that if each network node had its own unique address everything became easier to address. Well, with sufficient IPv4 IP addresses you can still do that!
With that many IP addresses, the use and disposition of those networks and the addresses within them must be documented and mapped. There are any number of network management tools available. Three FOSS choices are nagios, cacti or zenoss. There are many others. If you want to pay you have any number of choices.

The previous tools have been mostly about performance and alerting. There is also a requirement for network architecture, subnet mapping to sites and the actual use of IPs within those subnets. Again there is a choice of paid and "free" software on offer. VitalQIP is a very solid piece of software, but requires management and oversight. I have heard of it being used as a mechanism for enabling a helpdesk to allocate static IP addresses, and "freeing up valuable resources for other tasks". A free alternative might be IPPLAN, but there are many choices. If you are running a Microsoft Active Directory, then you will have subnet to site mapping information within the Sites and Servers section of the directory.

It is an historical curiosity of the company that the team that controlled the EMEA and APR regions arranged that the routers on all the subnets were always on IP address .20, i.e 10.10.10.20, 10.10.11.20 etc. Whereas in the US, it was always IP address .1, i.e. 10.10.10.1, 10.10.11.1, etc. Whilst recovering from a disaster, such decisions can be revisited. Standardisation of this kind of detail across an enterprise is always to your advantage.


OK, that's enough for now.

In Part 2, I'll cover some of the more "soft" requirements.

In Part 3, I wrap up considering the human element and make some recommendations.