API instability on May 19-20, 2026
Resolved
May 19, 2026 at 11:44pm UTC
On May 19-20, 2026, we experienced intermittent instability in the Carbon Query API. During this period, some requests experienced elevated latency, timeouts, or intermittent failures.
There was no data loss and no security impact.
Impact
The incident affected API availability and latency globally. Some clients may have seen slower API responses, intermittent request timeouts, or intermittent server errors. Retrying affected requests may have succeeded.
Root Cause
The issue was caused by an inefficient database lookup path in part of our flight emissions calculation flow which size grew significantly after last update. Under certain request patterns, this caused backend capacity to degrade and led to intermittent API instability.
Although the issue originated in one calculation path, it affected the shared API infrastructure and therefore had broader impact.
Resolution
We identified the problematic lookup path and deployed a fix to ensure those requests are handled efficiently. We also increased service capacity during recovery and adjusted autoscaling settings to improve resilience.
After the fix was deployed, API latency and availability returned to normal.
Prevention
We are taking the following actions to reduce the likelihood of recurrence:
- Expanding test coverage for edge-case request patterns.
- Testing critical lookup paths against production-sized datasets.
- Improving monitoring for slow backend operations and service health degradation.
- Reviewing similar lookup paths for performance risks.
- Continuing to monitor API latency, error rates, and capacity after the fix.
We apologize for the disruption and appreciate your patience while we investigated and resolved the issue.
Affected services