I use:

Friday, November 30, 2007

@#$% Happens...

Sometimes, no matter how welll you plan and how much you've begged for the funds to provide better service and support to your customer base, things happen with such ferocity that you're left dumbfounded and bewildered while trying to find your way back to functional.

Yesterday our systems, located at a datacenter which will remain nameless, were delt a heavy blow, 5 dead drives, 1 dead production SAN, and a very long, slow process to recovery. Staff were not the cause of the failure, recovery was impaired by the presence of Police and fire services (preventing access) and our world came to a grinding halt.

Today, we are waiting... we are preparing for the worst. What is the worst? Well, we restore from backup and implement an alternate SAN (iSCSI) solution we were preparing for use in around Christmas.

What caused this? I dunno. We haven't been informed with any certainty of the cause.

No comments: