Systems fully recovered

2026-06-11 Incident post-mortem

Summary

On June 11, 2026, part of our platform that stores the data produced by syncs began rejecting writes for a subset of syncs over a window of roughly 15 minutes. During this period, affected syncs were unable to save their fetched data and reported failures. The issue was caused by an internal change that was rolled out ahead of the component that still depended on the previous behavior.

We have fully mitigated the incident and have a concrete pla...

Post-Incident Summary

Date: 29 April 2026

Summary

A gradual increase in Records API response payload size eventually exceeded the memory available to the pods serving those requests, causing them to be terminated. As pods were terminated and replaced, the load balancer returned 502 errors for requests routed to them. The growth had been building over several days and was not caught by internal monitoring before a customer reported the issue.

Timeline (UTC)

• Issue began: 24 A...

Post-Incident Summary

Date: 12 April 2026

Impact: Degraded sync execution and delayed actions and webhook processing

Status: Resolved

Summary

A webhook flood originating from a single customer environment caused one of our databases to saturate, resulting in broad degradation of asynchronous job processing. Sync execution dropped to near zero, and a large portion of actions and webhook-driven work were delayed or unable to run. A secondary bug in the scheduling system amplified t...