Ukko cluster down due to power outage, stays down for some upgrades - upgrades take longer than anticipated

01.11.2012 - 13:45 - 08.11.2012 - 16:00

As a power outage just brought down the entire Ukko cluster, this opportunity will be used to update the firmware on all nodes. 

Break report update @ Nov 1, 18:30: Most NFS filesystems inaccessible, some nodes in weird shape, cluster unusable.

Also network hardware will be upgraded, possibly operating system too.

So this will take a while longer than expected.

Update @ Nov 5, 16:10:

Unfortunately the firmware upgrades are taking much more time than was anticipated. However, we planning to do all applicable updates during this break so we can avoid many shorter breaks in the future.

Update @ Nov 8, 18:30:

Several nodes are still offline, some network hardware is still misbehaving, but what's online has been upgraded to newest firmware versions and newest Cubbli Linux (based on Ubuntu 12.04).

08.11.2012 - 19:29 Ville Hautakangas
01.11.2012 - 14:49 Ville Hautakangas