POSTMORTEM:
A recent change caused the execution of all functions (actions, webhooks, syncs) to stop on Friday October 2nd between 17:22 and 17:54 UTC.
We identified a performance regression introduced by a recent query refactor in our task scheduler. The change caused one database query to scale poorly when the number of fresh execution increased significantly.
Under normal load, the query remained fast, which is why the issue was not immediately visible after deployment. 8 hours later, a ...