Upgrading HP Firmware

Lately we bought a new HP blade chassis to replace a customer’s old database server. All it’s services run on ~15 blades, splitted cross two HP C7000 chassis.

The Proliant BL460 G6 we bought came with much newer firmware revisions than all the existing G1 – part of the infrastructure didn’t receive much sysadmin love over quite some time. :-)

Blades, ILO, chassis and controllers where all running way outdated firmware and upgrading was highly recommended. The arising firmware combinations haven’t been tested and the new blade wouldn’t even be detected, so HP. They offered us an upgrade for about $2000 and 6 hours of downtime per chassis.

Here are some handsome findings, to do the upgrade on your own:

HP Firmware Comapatibility Matrix

HP tested certain sets of firmware for compatibility. Take a look at their compatibility matrix and try to stay within the tested boundaries. This could mean to upgrade in more than one step, if you are running an older release.

(http://h18004.www1.hp.com/products/blades/components/c-class.html)

Hp-Firmware-Catalog

There is Christian Hofstedtlers great firmware upgrade script, which automatically downloads the latest and greatest HP firmware installation packages. Its even creating softlinks, to reference cryptic firmware package names to their corresponding hardware components.

(https://github.com/zeha/hp-firmware-catalog)

You can run them from your OS as an online upgrade. Certain components still might require rebooting, to finish the “delayed upgrade”.

I would love to see HP maintaining this, since the approach provides a good example of providing customers with a modern and automated way to upgrade and monitor firmware for more recent releases.

ILO Shell

When upgrading many machines it will save you a lot of time, if you just use the SSH shell for configuring a boot device and rebooting the server.

Connect to ILO using SSH

Make sure you send the right username, AFAIK it’s case sensitive on the ILO:

    ssh phx-vnode03.oob.ono.at -l Administrator

Set an ILO Advanced Licence key

    cd /map1
    set license=YOUR-LICENCE-KEY

The advanced licence key is required to enable virtual device firmware features. Eg. to make use of the remote console or a virtual disk boot drive.

Mount and configure a network hosted ISO image as boot device

    cd /map1
    vm cdrom insert http://10.0.10.21/FW920B.2010_1129.2.iso
    vm cdrom set boot_always

…be it a firmware upgrade or an OS installation disk. Make sure you run the following command to “eject” it again:

     cd /map1
     vm cdrom eject

Monitoring

To please your monitoring system as well, check out checkmk. They wrote a couple of good SNMP checks for your HP or IBM bladecenter.

In the end I can highly recommend to keep your hardware firmware up to date. At least HP, my vendor of choice, they add a lot of useful bug fixes.

HP currently informs customers by a e-mail newsletter about updates, I would love to see this in my monitoring system too, like all the other security upgrades.

Try to plan the upgrade a bit or use existing downtimes to boot the HP Firmware Maintenance image.