POSTMORTEM:

A recent change caused the execution of all functions (actions, webhooks, syncs) to stop on Friday October 2nd between 17:22 and 17:54 UTC.

We identified a performance regression introduced by a recent query refactor in our task scheduler. The change caused one database query to scale poorly when the number of fresh execution increased significantly.
Under normal load, the query remained fast, which is why the issue was not immediately visible after deployment. 8 hours later, a spike in queued tasks triggered the inefficient execution path, increasing database work and causing the dequeuing query to time out.

We have since corrected the query and are adding additional safeguards, including additional monitoring to reduce the likelihood of similar regressions in the future.

This issue has been mitigated. We are now investigating the root cause and will update here when complete.

We are seeing some recovery in task processing.

We are still investigating. All nango tasks (actions, syncs, and webhooks) are currently not running.

Actions and syncs are failing to execute. We are investigating and will update here as soon as we have more information.