Power was lost for approximately 55 milliseconds, however, all of our gear was powered down. We have redundant UPS systems, however, we are now aware that they are wired in parallel and therefore vulnerable to power surges originating within our datacenter (we must concede that we had not anticipated this type of failure).

The majority of our systems were up within minutes of the power failure, though others took significantly longer to complete file system integrity checks.

New circuits are being deployed to our racks to ensure that we are on fully independent UPS systems wherever possible. Our provider will be adding additional independent UPS systems to connect to the racks shortly.

We may require some maintenance windows over the coming weeks to a very small subset of our customer base – all maintenance notifications will be sent with 48 hours’ notice per our policy.

Often when an outage of this magnitude occurs we find flaws in other systems and procedures: our primary sites were unavailable over the course of the outage as our failover sites had not been implemented, however, our Twitter stream remained active and in use to supply status updates.

We strive to learn from our mistakes and improve our platform – we will take this incident’s lessons and ensure that our Seattle power infrastructure is no longer vulnerable to neighboring networks’ activities.

We are approaching a holiday weekend with reduced staffing, however, if you have any questions, comments, or concerns regarding this incident (or if you would like to claim a Service Level Agreement credit) please let us know – we look forward to hearing from you and we will make every effort to address your concerns in a timely fashion.

We welcome discussion on the Spry Forums and, if you are not already subscribed, please take this opportunity to subscribe to the Spry Hosting Twitter Feed.