Previous incidents

Records saving is now fully operational

Slack Notifications are now enabled again. Thank you for your patience

We have identified the root cause of the issue. The sharding for our log database was too small, creating a bottleneck. During the incident, syncs and actions were slower, but logging was not on the critical path. Only some log messages have been lost.

Downtime: resolved.

Root cause: a table from the main database had grown very large. A database client used for Business Intelligence performed an exact count on this table, locking the database.

Mitigation: the database client now points to a read replica. The large table is being cleaned up.

Downtime: resolved.

Root cause: Our cloud provider restarted Redis during a scheduled maintenance operation. A race condition bug in Nango's Redis reconnection logic led to the creation of orphans Redis clients that eventually used all available connections, preventing any logic that depends on Redis to run successfully.

Mitigation: We fixed our Redis disconnection logic. We're investigating how to recover better from Redis's unavailability. We plan to migrate to a different cloud provider,...

Occasional errors: resolved.

Root cause: As we scaled our services horizontally, we exhausted our database connection pool.

Mitigation: use a database connection pooler + optimize our database client code to consume less connections.