All Systems Operational
API Operational
pci.ravelin.com Operational
Dashboard Operational
www.ravelin.com Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Apr 17, 2021

No incidents reported today.

Apr 16, 2021

No incidents reported.

Apr 15, 2021

No incidents reported.

Apr 14, 2021

No incidents reported.

Apr 13, 2021

No incidents reported.

Apr 12, 2021
Resolved - This incident has been resolved.
Apr 12, 08:12 BST
Monitoring - Normal operation has resumed.

Additional BigTable load was generated due to the queue retry behaviour of a data replay that has been running today. Ravelin uses data replays to patch and upload data, and they are a part of normal operation. The data requests are batched per customer for ordering purposes. This evening we have encountered a large enough batch that the queue handler timed out because we are not pinging it mid-batch. This automatically puts the entire batch back on the queue to be processed again. We believe the BigTable CPU spike and subsequent API response errors to be a result of this loop.

We will resume the replay once there is enough capacity for it to succeed. Tomorrow we will investigate pinging the message queue to avoid timing out mid-batch, and only re-queuing the parts of a batch yet to be processed.
Apr 11, 20:12 BST
Investigating - With another small spike between 1941 and 1943 BST we are investigating the cause of BigTable CPU spikes correlating with the time these errors occurred. Scoring times remain elevated. We have paused background data cleaning operations to reduce load.
Apr 11, 19:47 BST
Monitoring - There was an elevated error rate on the API causing a spike in 500s between 1934 and 1938 BST and another beginning now. This appears to correlate with a spike in BigTable CPU usage which we are investigating.
Apr 11, 19:41 BST
Apr 11, 2021
Apr 10, 2021

No incidents reported.

Apr 9, 2021

No incidents reported.

Apr 8, 2021

No incidents reported.

Apr 7, 2021

No incidents reported.

Apr 6, 2021

No incidents reported.

Apr 5, 2021

No incidents reported.

Apr 4, 2021

No incidents reported.

Apr 3, 2021

No incidents reported.