Approximately 4.2% of card charge attempts are failing with HTTP 502 as the payment provider connection pool saturates. Checkout is partially degraded; existing subscriptions are unaffected.
7 entries · newest last
Alert fired: billing-service error rate 4.21% > 1.00% (5m window)
Acknowledged. Confirming impact and pulling request traces for /v1/charges.
Checked upstream dependency. Stripe API p99 normal at 184ms — not the source.
Hypothesis: DB connection pool exhaustion after pool size change. Inspecting pg_stat_activity.
Escalated to SEV-2 and paged the Payments on-call. Customer support notified.
Confirmed 184/200 pool connections held; HikariCP reporting acquisition timeouts at 3000ms.
Mitigation prepared: revert pool size 200 -> 120 and restart canary pods.
Visible to every responder on the incident channel
Customer and budget impact
Approximately 4.2% of card charge attempts are failing with HTTP 502 as the payment provider connection pool saturates. Checkout is partially degraded; existing subscriptions are unaffected.
Users affected
1,840
Error budget burn
7.4%
3 engaged
Not yet identified.
Incident is still open.
7 events
Error rate above 1%
Database connections above 180
P95 latency above 750ms
Error rate above 1%
P95 latency above 750ms
Near the incident window
No deployments nearby