API unreachable

Summary

Earlier today, our platform experienced a period of degraded API performance. The issue was triggered by an unusually large burst of automated traffic that exceeded our expected operational thresholds and highlighted limits in one of our protection layers.

Impact

• Some customers observed elevated error rates or timeouts when calling the API.
• The impact was intermittent and limited to a subset of traffic.

What Happened

Our systems received an extremely high volume of repeated requests. While our application includes rate limiting, this surge occurred at a level that prevented the application-layer limiter from fully initializing and applying its rules effectively.

As part of our mitigation process, we blocked the traffic at the network level. Once the excessive traffic was filtered out, API performance stabilized and returned to normal.

What We’re Doing to Prevent This Going Forward

To strengthen our resilience, we are implementing the following improvements:

• Restoring and Verifying Alerts: We are investigating why automated alerts were not propagated and will ensure the alerting pipeline works as intended.
• Adding Ingress-Level Rate Limiting: A new traffic-shaping layer will be introduced at the ingress point to ensure that unusual spikes are controlled before they reach application components. This will protect the system even in scenarios where application-level rate limiting cannot fully activate.

Current Status

All services are operating normally, and we continue to monitor the platform closely.

‌

Thank You
We appreciate your patience and understanding as we further strengthen the reliability and stability of our platform.