It was still possible to login and peek around, but anything that did anything with disk activity other then the root disk just seemed to die. It wasn't possible to reboot it nicely either.
But when I powercycled using the new fancy remote powercycler, it didn't come back up like it's done every time someone was actually *there* and powercycled it. Typically, neh?
So now it's still down. No idea when it will get back up again, I hope someone might pass there at least tomorrow.
I also saw a wierd, techy "kernel oops" in one of the logs, if any linux kernel guru happens to pass I wouldn't mind a hint:
scsi0:A:14:0): Locking max tag count at 64 ... (scsi0:A:6:0): Locking max tag count at 64 Unable to handle kernel paging request at virtual address 7073618f ... Oops: 0000 ...[lots of figures cut out]
My *guess* is that the SCSI subsystem shuts down for some reason (bad termination, bad disks, faulty controller), but it'd be neat to know which hardware we should replace...