Outage of the Public Photon Cloud in region US East on October 20th, 2018
Our team works very hard to achieve a very high QoS – even the shared Photon Public Cloud has an uptime of 99.9% and higher in all available regions. We are proud of this and serve more than 300 Million players every month.
On Saturday we hit a snag in one of our 13 regions: We apologize for any inconveniences that may have occurred and we have applied corrective measures to prevent similar issues in the future.
Affected Products: Public Cloud / Premium Cloud
Affected Regions: US East
Downtime (UTC): Sat 1:30am – Sat 08:00am // 6.5h
Root cause: Memory issue on master servers in the US East region.
Problem: An alert-endpoint was misconfigured and not triggered for the US East environment. Hence our 24×7 NOC team was not alerted and could not resolve the issue.
Resolution: The issue was resolved by the German team that immediately replaced the master server.
Our operations team took immediate actions
- have double checked all alert configurations to make sure they are correct
- installing fallback alerts to our 24×7 NOC
We are deeply sorry! We will do the best possible to keep highest standards and continue to invest into our monitoring, alerting and automation.
