Irregular traffic appears to have resulted in a hot key in our database, which caused one of our database nodes to become unresponsive. We scaled up the number of nodes to limit the impact of that node. Our failover and retry logic was able to satisfy the majority of requests, significantly lowering the error rate by 19:34 BST, albeit with elevated response times. The error rates and response times returned to zero by 20:30. We are continuing to investigate the cause and mitigations for this performance issue.