Resolved
All data has now been updated. Searching within the dashboard should be up to date and running as normal. Thanks for your patience
Monitoring
We've fixed the elastic search database cluster issues, so we are no longer experiencing any errors or timeouts. However the cluster is not running completely up to date yet so there will be stale data returned on searches. We're running through the queues to fix that, and expect them to finish in ~2hrs by 5pm GMT
Monitoring
Index time has gone back to nominal latency. We are currently draining queues of backfilled data. Expect to have search data up to date shortly
Identified
Mitigation is still ongoing. Search times have improved and index latency is improving. Not recovered yet but going in the right direction.
Identified
We have done some mitigation that will improve search latency for the dashboard but data will be delayed. We are continuing to work on the core database issue.
Identified
We are still investigating and managing node scaling on our search database. We will update you further as things progress.
Identified
We are aware of a slow down in our search database. We are attempting to scale further nodes and investigate root cause.
Investigating
Searching in the dashboard will occasionally timeout or be slower than expected. We are currently investigating.