No components marked as affected
Resolved
This incident has been resolved.
Monitoring
We are continuing to monitor for any further issues.
Monitoring
We have identified that at 4:21am several machines in different zones simultaneously stopped responding. It appears that GCP attempted to restart each of those machines multiple times for about 2 hours. Some machines did eventually come back online. Deleting those flakey instances saw them replaced with fully functional instances.
Normal service has since been resumed. We will continue to monitor and have an incident report soon.
Investigating
We are experiencing elevated responses times and 500 rates from our API, affecting ~2% of requests. We are investigating to identify the cause. Please check back here for further updates.