Power issues
We are experiencing power issues within the data centre, this is impacting primarily on email. We are working to resolve the problem as soon as possible.
4 thoughts on “Power issues”
Comments are closed.
Mail is now accessible, however access to email is classified as ‘at-risk’ until 17:00 to enable us to complete additional work to bring the mail back up to its normal performance levels. This will result in several short outages (approximately 5 minutes each) whilst we finalise the configuration.
Additional information will be added to outline the exact nature of the problem.
Cause of the outage
This morning our data centre suffered a loss of phase (L3) on the incoming utility electricity supply. The backup generator began to supply load to the facility, however 2 of the 7 UPS systems failed to correctly transfer load due to the incoming phase imbalance and resulting in the loss of a synchronisation reference point.
The result was a loss of supply to a section of the data centre for several seconds whist the UPSs found a new reference point from the generator supply.
Unfortunately the break in supply was sufficient to cause reboots of some but not all equipment hosted in the affected areas.
The data centre is now running on the main utility supply. The phase fault has been temporarily fixed by Central Networks (who maintain the supply network in the area), however a perminent resolution is yet to follow.
The data centre will be undertaking a review of the UPS problem with the manufacturer and we will advise of any further remedial work which needs to take place to cater for this unusual situation.
Result Email outage
As a result of the short loss of power our mail system suffered a severe software fault which prevented us from bringing it back online when powered was resumed.
As a result we began a ‘bare metal’ restore process to bring the mail system back online. This process was completed shortly before 13:00 today. The process involved rebuilding from hardware and restoring half a terrabyte of mail data. Due to our comprehensive backup routine no data was lost as a result.
We would like to thank you for your patience during this major outage.