HEY, I'M
PUSHPANDER.

WHAT HAPPENED

Our reconciliation job stopped marking ~3% of successful Easebuzz transactions as settled. Payments completed on gateway, but our system kept them as pending and users received duplicate reminders.

IMPACT

~75 transactions/day were misclassified for 3 days before alerts. Around Rs.4.2L looked unreconciled, support tickets rose by ~40%, and user trust took a hit even though no money was actually lost.

ROOT CAUSE

Easebuzz changed webhook field casing in a minor SDK update (`paymentStatus` to `payment_status`). Our parser lacked fallback mapping; failures were logged but not alerted, so signal got buried.

WHAT I DID

1. Cut off auto-retry reminders to stop spamming users.
2. Ran a one-off backfill script over raw webhook payloads from S3 archive and reconciled against Postgres.
3. Added PagerDuty schema-validation alert if parse failures exceed 0.5% in any 5-minute window.
4. Published customer communication through support within 2 hours of resolution.

WHAT WE LEARNED

Parse-failure logs without alerting create fake confidence. Every external integration now has a parse-failure SLI with a hard threshold.

WHAT HAPPENED

A downstream partner API went down for 11 minutes. Our unbounded retry logic stacked retries and then burst 12K queued attempts in 90 seconds when service recovered.

IMPACT

Partner-dependent flows saw 18 minutes of effective downtime. RabbitMQ queue depth spiked 40x; there was no data loss but recovery extended due to self-inflicted throttling.

ROOT CAUSE

Retry strategy had exponential backoff but no jitter, no max attempts, and no circuit breaker. Failures kept amplifying traffic while dependency remained down.

WHAT I DID

1. Paused webhook consumers in RabbitMQ to stop storm expansion.
2. Drained dead-letter queue to a holding table for controlled replay.
3. Restored traffic gradually (10% to 50% to 100% over 15 minutes).
4. Shipped circuit breaker with bulkhead + jitter; capped retries at 5.

WHAT WE LEARNED

Retries without circuit breakers amplify dependency outages. Bulkheads and bounded retries are now defaults for partner integrations.

WHAT HAPPENED

A routing refactor in our GMS inventory adapter caused some `getInventoryByLocation` reads to resolve against the wrong tenant schema. Tenant A could briefly see line items from Tenant B.

IMPACT

6 tenants were affected and around 200-300 line items were exposed read-only. No write corruption occurred, but disclosures were required for enterprise clients plus a security review.

ROOT CAUSE

Tenant context moved from request scope to a singleton cache of last seen tenant for perceived performance gains. Under concurrency, stale tenant IDs leaked across requests on the same Node worker.

WHAT I DID

1. Reverted the offending commit immediately after pattern confirmation in logs.
2. Extracted affected request IDs and mapped exposure per tenant.
3. Partnered with compliance to draft disclosures for impacted enterprise tenants.
4. Replaced singleton helper with AsyncLocalStorage request scoping.
5. Added integration test: 200 concurrent requests across 5 tenants with zero cross-contamination assertion.

WHAT WE LEARNED

Tenant context is security-critical state and must remain request-scoped. Any performance optimization touching tenant scoping now goes through security-sensitive review.

WHAT HAPPENED

A scheduled bulk fleet sync started during a manual backfill. Producers outran consumers and telemetry queue depth climbed from ~500 to ~480K in about 25 minutes.

IMPACT

No data loss due to durable RabbitMQ queues, but consumer lag reached 22 minutes and broker disk usage rose from 18% to 71%. Real-time dashboards lagged for roughly 90 minutes.

ROOT CAUSE

Two heavy workloads collided against consumers sized for steady-state traffic. Producer-side backpressure and queue-depth autoscaling were missing.

WHAT I DID

1. Scaled consumer deployment 3x to process backlog faster.
2. Paused backfill job until spike passed.
3. Monitored broker disk closely and prepared failover if usage crossed 85%.
4. Added producer-side rate limit for bulk sync and queue-depth autoscaler for consumers.

WHAT WE LEARNED

Steady-state capacity planning is not enough for bursty systems. Backfills now avoid peak fleet-write windows and rate limits protect queue health.