We've identified the cause of the problem as a bug in logging that resulted in ever-increasing log message sizes, eventually causing slowdowns.
We've identified the problem components and rolled out versions without the logging issue. Latency has returned to baseline and we do not believe it will increase again.
Resolved
We've identified the cause of the problem as a bug in logging that resulted in ever-increasing log message sizes, eventually causing slowdowns.
We've identified the problem components and rolled out versions without the logging issue. Latency has returned to baseline and we do not believe it will increase again.
Investigating
We’ve seen gradually increasing latency over the last 24h that may result in increased timeouts. The latency has returned to baseline but we’re investigating what the issue was in case it recurs.