2026-08-03 Missing logs

Summary

On August 3, 2026, Nango's log ingestion pipeline began rejecting writes because the index for that day had already transitioned to a read-only state.

Our investigation indicates that several Connect UI events were received from a browser whose system clock was approximately six days ahead. We accepted the client-provided timestamp without validation and used it to select the storage index, causing the August 3 index to be created prematurely. After 48 hours, our lifecycle policy moved the index to the warm tier, shrank it, and made it read-only—before August 3 had arrived.

Customer-facing function executions were not affected. We mitigated the incident by re-enabling writes to the affected index.

Customer impact

• What: Nango logs could not be written while the affected index was read-only. Function executions and other customer workloads were not affected.
• When: August 3, 2026, from approximately 00:00 UTC to 01:26 UTC.
• Data integrity: All log events received during this period were lost.

What happened

Log indices are selected using the createdAt timestamp attached to each event.

On July 28, a Connect UI session from a Windows/Chrome client emitted several events. The client’s clock appears to have been approximately six days ahead, so these events carried August 3 timestamps. The telemetry endpoint accepted the client-supplied timestamp and used it directly as createdAt. The first event therefore created the August 3 index on July 28.

Our index lifecycle policy moves indices to the warm tier after 48 hours, where they are shrunk and made read-only. Around July 30, the policy applied this transition based on the index’s actual age.

When normal August 3 traffic began writing to the same index, those writes were rejected because the index had already been shrunk and locked.

The lifecycle policy behaved as configured. The underlying issue was that an unvalidated client timestamp could control index selection and create a future-dated index.

Resolution

We restored log ingestion by re-enabling writes to the shrunk index.

What we’re doing to prevent recurrence

• Validate clock skew: We will normalize client-provided timestamps that fall unreasonably far in the past or future.

We apologize for the disruption. If you have questions about how this affected you, please reach out.

Nango logs are not being ingested