Date of Incident: Thursday 09/11/2017
Summary:Due to a severe power outage on our hosting partner's infrastructure, we are currently experiencing problems across our worldwide platform.
From 7am UK time, our monitoring systems notified us of connection problems with our core traffic managers. We then discovered wider outages across our platform.
Our partner subsequently updated us with information regarding the power outages, which affected a number of their key systems including interconnects between their data centres. By 9am UK time, we established that much of our platform remained powered on, but could not connect out to the Internet, which affects the availability of our billing and messaging services.
Since 9am, our partner has successfully worked with their energy company to supply power to the majority of their infrastructure, but some remain out of action. We are currently advised that a further 1 to 2 hours will be required to restore remaining services. We can then fully evaluate the impact on our platform and ensure delivery of services.
This is a severe and rare issue over which we have little control, and we will review our procedures in light of the situation, working as required with our hosting partner to evaluate the reasons for this issue and how we can extend our redundancy and failover provision. It is not yet clear why our partner's backup power facilities did not operate as required and in line with their ISO certification or SLA.
We apologise sincerely to all clients affected by the current issue and we will keep you updated.
[ x ] WEB / WAP Billing / PayForIt
[ x ] Inbound SMS (MO)
[ x ] Outbound Premium SMS (MT)
[ x ] Direct Operator Billing
[ x ] Voice Services / IVR
[ x ] Inbound Longcode / Non-Premium SMS (MO)
[ x ] Outbound Non-Premium SMS (Bulk MT)
[ x ] Number Lookup / HLR
[ x ] Control Panel
[ ] Other
[ ] Total loss of service
[ x ] Partial loss of service for a period of time.
[ x ] Reduced availability with temporary connection losses
[ x ] Delayed response
[ x ] Other