Our Postgres master died and appeared to have lost its disk upon restarting. The caching layers we have in front of the database all continued to operate, allowing regular service to continue. Logins require validation with the database, however, which prevented new authentication tokens being given for dashboard logins. A spike of 500s was observed just after the new database came online, which we shall be investigating. No further API issues have been observed for the last 10 minutes but we are continuing to monitor.
Posted Apr 03, 2019 - 16:25 BST
Update
We are currently investigating errors connecting to Postgres which is affecting dashboard login sessions and a small number of API requests.