Issue resolved

What happened

The API was unreachable, or rather requests ended in a 500 error. The incident started on Thursday, 3rd of August, at 3.30 pm UTC, and by 3.55 pm UTC, most traffic from affected servers was redirected to a healthy cluster. The error was finally resolved by 4.45 pm UTC. Within this ~1h interval, about 35% of all requests failed. The main impact was on traffic originating in the US.

Why this happened

A new feature was launched that lets users see which API key was used for purchases. This required updating account configurations. However, right after the release, the authentication layer failed due to a stale cache of the updated configuration.

Estimated costs

For some customers, we were down for 25 to 75 minutes. Please reach out to us if there is anything we can do to help you, e.g., re-run some calculations or look into specific requests.

How to prevent this in the future

We take every incident seriously, meaning we do a thorough analysis and discuss with the team steps we can do to improve and learn from any incident. The resulting actions are: We are improving our gradual rollouts to avoid such issues in the future.