Shared serice outage 06/09/2012
We experienced an unplanned service outage between approximately10am and 11am this morning which affected our ‘shared services’ . 90% of services were restored by 11am with the remaining 10% of services restored subsequently.
The precise nature of the issue is being investigated and further details will be placed here within approximately 24 hours when we have concluded our investigations.
Please accept our sincere apologies for any incovenience. During the outage all inbound mail will have been stored on our ‘off site’ MX and sent to the mailbox servers shortly after they came back online. We engaged with the ‘remote hands’ service at the data centre within 5 minutes of our ‘off site’ monitoring system alerting us to visibility issues.
As stated all services should be operating normally but should you have any residual issues please do not hesitate to email support with details of the issue and we will investigate promptly.
Since restoring service we have been investigating to understand the root cause. Our investigations conclude the issue was triggered by a firmware bug impacting the blade enclosure due to the long uptime of the enclosure controllers.
We are currently planning to upgrade the firmware at the earliest opportunity in a controlled approach. In addition we are re-evaluating server placement to mitigate and/or reduce the impact of a similar rare event occurring.
Once again please accept our sincere apologies for the impact to service earlier today.