stash pagesPostmortem: the retry storm of June 3rdMD2mo agoRaw

This page is public — anyone with the link can see it. Sign up for Stash to make it private.

Create your own page →

Postmortem: the retry storm of June 3rd

Severity: SEV-2 · Duration: 47 minutes · User impact: delayed syncs, no data loss · Author: on-call

Summary

A malformed pagination cursor from one connector caused its extraction job to fail after side effects but before checkpointing. The retry policy did exactly what it was configured to do, which was the problem: 8 retries × 200 affected objects × one enqueued webhook each ≈ a self-inflicted thundering herd that saturated the worker pool for everyone.

Timeline (UTC)

  • 14:02 — Vendor API deploys a change: cursors now expire after 60s instead of 10min.
  • 14:07 — First InvalidCursor failures. Jobs retry from the start of pagination, re-emitting events for pages already processed.
  • 14:11 — Queue depth alert. On-call assumes vendor outage, waits.
  • 14:26 — Worker pool at 100% on one connector's jobs; other sources' freshness SLOs breached. SEV-2 declared.
  • 14:38 — Mitigation: connector paused, queue drained of its events (12k), pool recovers.
  • 14:49 — Patched cursor refresh shipped; connector resumed; backlog cleared by 15:20.

What went wrong, in order of importance

  1. Retries restarted the whole pagination loop instead of resuming. The checkpoint lived at job level, not page level. One bad page at position 190 meant re-fetching 189 good pages — and re-emitting their events.
  2. Emitted events weren't tied to checkpoint success. We produced side effects for work we then declared failed. Replay-safety assumed consumers dedup; the dedup window (30s) was tuned for webhook storms, not multi-minute retry loops.
  3. No per-source circuit breaker. A connector failing 100% of jobs for 20 minutes should isolate itself. Fairness buckets cap throughput per source; they don't cap failure spend.

What went right

  • Pausing one connector was a one-command mitigation and affected only that source.
  • No data loss or duplication reached users — downstream idempotency held even though we leaned on it harder than designed.

Action items

  • Page-level checkpoints; retries resume, never restart. (shipped June 5)
  • Events emit only after checkpoint commit — outbox pattern. (shipped June 9)
  • Circuit breaker: 10 consecutive failures on a source → auto-pause + alert, owner: platform.
  • Dedup window becomes per-source and adaptive to retry depth.
  • Synthetic canary that exercises cursor expiry against each vendor weekly.

The lesson we keep relearning

Retry policies encode an assumption about why things fail. Ours assumed transient network blips. When the failure is deterministic — same input, same error — retries are just amplification. Classify before retrying.

Discussion welcome. Select any line to comment.