# Design a Webhook Delivery System > Design a webhook delivery system: fan-out queues, retries with backoff and jitter, HMAC signing, circuit breakers, and dead-letter redelivery at Stripe scale. Canonical: https://ironclad.academy/system-design/articles/design-webhook-system | Difficulty: medium | Topics: Interview, Messaging, Queues, Reliability, Security Asked at: Stripe, Svix, Hookdeck ## Key takeaways - At-least-once over HTTP means duplicates are part of the contract: providers never promise exactly-once, and consumers dedupe on the event id before acting. - Concurrency = rate x latency is the sizing law that breaks naive designs: 5% of traffic timing out at 10 s can need 2x the worker slots of the healthy 95%, which is why per-endpoint isolation and circuit breakers are non-negotiable. - Stripe retries with exponential backoff for up to 3 days then may disable the endpoint; Svix makes 8 attempts over about a day and auto-disables after 5 days of continuous failure — copy one of these schedules, do not invent your own. - Sign HMAC-SHA256 over '{timestamp}.{raw_body}' with a per-endpoint secret and reject timestamps older than ~5 minutes; putting the timestamp inside the signed string is what defeats replay. - Do not promise ordering: retries plus parallel delivery make FIFO incompatible with throughput. Ship timestamps and object versions, and tell consumers to fetch current state. - The self-serve delivery dashboard is load-bearing, not cosmetic — failed-webhook debugging is one of the biggest support-ticket categories any API platform has, and logs plus one-click redelivery are what keep it off your support queue. > **In one line:** A webhook delivery system persists every event through a transactional outbox, fans it out through per-endpoint queues to a signing worker pool, retries with exponential backoff and jitter for up to ~3 days, trips a per-endpoint circuit breaker when a consumer dies, and gives customers a self-serve delivery log with redelivery — promising at-least-once, unordered delivery and nothing more. **The problem.** Your API platform needs to tell customer systems that something happened — a payment succeeded, a model run finished — by POSTing to URLs the customers control. Those endpoints are the most hostile dependency you will ever integrate: misconfigured, slow, down for the weekend, or gone forever. You must deliver reliably to all of them without letting the worst one degrade the rest. **Requirements that drive the design.** - Customers register HTTPS endpoints with event-type filters; you deliver matching events as signed POSTs. - At-least-once delivery: an accepted event is never silently dropped, even across provider crashes. - Retries with exponential backoff + jitter; dead-letter after the schedule is exhausted; customer-visible logs and manual redelivery. - One dead endpoint must not delay anyone else's deliveries (multi-tenant isolation). - HMAC signing with replay protection; no ordering guarantee — say so out loud. **Scale (illustrative, back-of-the-envelope).** - 500M events/day × 1.2 endpoints matched → 600M deliveries/day ≈ 7k/s average, ~28k/s peak. - ~90% first-attempt success, failures average 5 attempts → ~840M HTTP attempts/day (~1.4× amplification). - ~2 KB per attempt log → ~1.7 TB/day; 30-day retention ≈ 50 TB. **The architecture in one diagram.** ```mermaid flowchart LR SVC[Product services] --> OB[(Outbox)] OB --> K[Event log] K --> FAN[Subscription matcher] FAN --> PQ["Per-endpoint queues
+ token buckets"] PQ --> W["Workers
sign + POST"] W --> EP[Customer endpoints] W --> RS[Retry scheduler] RS --> PQ W --> DLQ[(DLQ + delivery log)] style OB fill:#0e7490,color:#fff style PQ fill:#a855f7,color:#fff style W fill:#15803d,color:#fff style DLQ fill:#9f1239,color:#fff ``` **Key design decisions.** | Decision | Choice | Why | | --- | --- | --- | | Event capture | Transactional outbox, relayed to a log | Event and business write commit atomically — no ghost or lost webhooks | | Delivery semantics | At-least-once, consumer dedupes on event id | Exactly-once over HTTP is not achievable; pretending otherwise breaks trust | | Ordering | None guaranteed; timestamps + object versions in payload | FIFO + retries + parallelism = head-of-line blocking | | Isolation | Per-endpoint queues + token buckets | One dead consumer can't occupy the shared worker pool | | Retry schedule | Exponential backoff + full jitter, ~8 attempts over 1–3 days | Matches Stripe (3 days) and Svix (8 attempts) practice | | Dead consumers | Circuit breaker per endpoint, auto-disable after days | Fail fast instead of burning sockets on a corpse | | Security | HMAC-SHA256 over `{timestamp}.{body}`, 5-min tolerance | Authenticates payloads and defeats replay | | Customer UX | Delivery log + manual/bulk redelivery portal | Webhook debugging is a top support-ticket driver | **If you have 60 seconds, say this.** "Product services write events to a transactional outbox in the same DB transaction as the business change; a relay publishes them to a durable log. A matcher fans each event out to subscribed endpoints and enqueues a delivery job per endpoint — into per-endpoint queues with token buckets, so one slow consumer can't cause head-of-line blocking for the rest. Workers sign each payload with a per-endpoint HMAC-SHA256 over timestamp-dot-body and POST with a ~10-second budget; only a 2xx counts. Failures retry on an exponential backoff with jitter — roughly 8 attempts over a day to three days, like Svix and Stripe — then dead-letter. A per-endpoint circuit breaker fails fast on dead consumers, and after days of continuous failure the endpoint is disabled and the customer notified. Delivery is at-least-once and unordered; consumers dedupe on event id and fetch current state. Customers get a self-serve log with redelivery." At 02:14 the pager goes off: webhook delivery p95 has crossed ten minutes. Nothing is down. Kafka is healthy, the workers are healthy, CPU is bored. The cause is dumber than any of your dashboards suggest: your largest customer deployed a broken ingress at midnight, and their endpoint now swallows every POST for 60 seconds before returning a 504. Your delivery workers — one shared pool — are all parked inside that customer's timeout. Every other customer's webhooks are queued behind a wall of sockets held open by a machine that will never answer. One consumer broke; your product's entire asynchronous surface broke with it. That incident, in some form, is why "design a webhook delivery system" has become a standard interview question at API-first companies, from Stripe-scale fintechs to model providers. It looks like "make an HTTP POST." It is actually a study in multi-tenant isolation, retry theory, and the art of promising exactly as much as you can deliver. This article builds the system the way the interview should go: requirements, capacity, then an architecture where every component exists because something broke without it. ## What a webhook delivery system must guarantee Functionally: customers register HTTPS endpoint URLs, each with a filter over event types (`payment.succeeded`, `run.completed`). When a matching event occurs, you POST a signed JSON payload to every subscribed endpoint. Customers can list past deliveries, see request and response details, and redeliver failed or missed events themselves. The non-functional requirements are where the design actually lives. **At-least-once delivery**: once you accept an event, it is never silently dropped — not by a worker crash, not by a customer outage, not by a deploy. Latency: seconds at p95 *to healthy endpoints*; an event that takes 30 seconds to arrive is a support ticket at a fintech. Isolation: endpoint A's outage must not delay endpoint B by even one percentile. Security: payloads are authenticated and replay-resistant, because "POST from the open internet" is otherwise an unauthenticated write path into your customer's billing logic. Two honest non-goals, and saying them explicitly is worth more in an interview than any diagram. **Exactly-once is off the table**: if the consumer's 200 response is lost in transit, you must retry, and the consumer sees the event twice. Every real provider documents this — Stripe's docs tell consumers to log event ids and skip duplicates. And **ordering is off the table** too; we'll get to why, because interviewers push on it. Webhooks are one of three ways to get data out of a platform — the others being polling and persistent connections; the [realtime communication patterns article](/system-design/articles/realtime-communication-patterns) covers that trade-off. Here we assume webhooks won and design the delivery side. ## Capacity: the numbers that shape the architecture ``` Illustrative Stripe-scale target (not Stripe's actual internals): Events: 500M events/day Fan-out: x1.2 endpoints matched per event -> 600M deliveries/day Average rate: 600M / 86,400 s ≈ 7,000 deliveries/s Peak (4x): ≈ 28,000 deliveries/s Retry amplification: ~90% succeed on attempt 1; the rest average ~5 attempts 600M x (0.9 x 1 + 0.1 x 5) ≈ 840M HTTP attempts/day (~1.4x) Worker concurrency (Little's law: in-flight = rate x latency): All healthy at ~250 ms: 28k/s x 0.25 s ≈ 7,000 in-flight Now 5% hit 10 s timeouts: 1,400/s x 10 s = 14,000 in-flight -> 5% of traffic needs 2x the slots of the other 95% Attempt log: 840M x ~2 KB ≈ 1.7 TB/day -> 30-day retention ≈ 50 TB ``` Read the middle block twice, because it is the whole article in four lines. Throughput is not the hard part — 28k requests/sec is a boring number for a fleet of async HTTP workers. **Latency variance is the hard part.** Your p50 consumer answers in 200 ms; your worst consumer answers never, and you only find out after your timeout budget. Since concurrency is rate times latency, the slowest consumers dominate your capacity even when they are a sliver of traffic. Everything from here on — per-endpoint queues, token buckets, circuit breakers — exists to stop that sliver from setting your fleet size. ## Building up to the design This is a sibling of the [notification system problem](/system-design/articles/design-notification-system), with one inversion that changes everything: there, you deliver through a handful of professional providers (APNS, Twilio) with published SLAs. Here, you deliver to thousands of endpoints run by whoever configured them, with no SLA at all. Walk the evolution. ### V1: POST inline from application code ```python def on_payment_succeeded(payment): db.commit(payment) for hook in subscriptions.matching("payment.succeeded", payment.account): requests.post(hook.url, json=payment.as_event(), timeout=10) ``` This demos fine and fails four ways at once. A slow endpoint adds its latency to your payment path. A crash between the commit and the POST loses the webhook forever. Flip the order and a rollback after the POST announces a payment that never happened. And there is no retry, so any consumer blip is a permanently lost event. The deeper bug is structural: delivery shares a fate with the business transaction, and they need to fail independently. ### V2: transactional outbox, queue, worker pool Write the event to an **outbox table** in the same database transaction as the business change — atomic by construction. A relay process tails the outbox (or uses CDC) and publishes to a durable log; Kafka fits because you want replay and multi-day retention, though the [message queue article](/system-design/articles/design-distributed-message-queue) covers when SQS is the more honest choice. A **subscription matcher** consumes events, looks up which endpoints subscribe to that event type for that account, and emits one delivery job per (event, endpoint) pair. A worker pool executes the POSTs. Now the caller never blocks, crashes lose nothing, and failed deliveries can retry from the queue. This is a genuinely decent system — right up until the 02:14 page from the opening. One shared queue plus one shared worker pool means one 60-second-timeout endpoint occupies workers 240× longer per delivery than a healthy 250 ms endpoint. That is textbook **head-of-line blocking**, and no amount of "add more workers" fixes a ratio. ### V3: per-endpoint queues and token buckets Give each endpoint its own logical queue and its own delivery budget. Physically this can be thousands of Redis lists, a partitioned "delivery jobs" table, or a Kafka topic keyed by endpoint with a fair scheduler on top — the mechanism matters less than the invariant: **workers pull from endpoints round-robin (or weighted-fair), and a per-endpoint token bucket caps in-flight requests and requests/sec to any single destination.** A struggling endpoint now queues its own backlog without consuming anyone else's worker slots, and you stop DDoSing customers who are already down — if an endpoint returns 429, the bucket drains to match. This is the same [rate limiter machinery](/system-design/articles/design-rate-limiter) you'd build for ingress, pointed outward, and it is [backpressure](/system-design/articles/backpressure-and-flow-control) applied per tenant. ### V4: circuit breakers for dead consumers Token buckets protect throughput but still waste real sockets probing a corpse. A **per-endpoint circuit breaker** stops even trying: track failure rate over a sliding window; past a threshold the breaker opens and deliveries park immediately in the retry schedule without an HTTP call. After a cooldown, a half-open probe sends one real delivery; success closes the breaker and the backlog drains (with jitter — see below), failure re-opens it. ### V5: dead letters, redelivery, and the portal When the retry schedule is exhausted, the delivery moves to a **dead-letter state** — never deleted, visible to the customer, bulk-redeliverable once they fix their endpoint. Add the customer-facing delivery log and you have the production system. ```mermaid flowchart LR V1["V1: inline POST
shared fate with business txn"] --> V2["V2: outbox + queue + workers
durable, decoupled"] V2 --> V3["V3: per-endpoint queues
no head-of-line blocking"] V3 --> V4["V4: circuit breakers
dead endpoints fail fast"] V4 --> V5["V5: DLQ + redelivery portal
production"] style V1 fill:#0e7490,color:#fff style V3 fill:#a855f7,color:#fff style V5 fill:#15803d,color:#fff ``` ## The full architecture ```mermaid flowchart TD SVC1[Billing service] --> OB[(Transactional outbox)] SVC2[Platform services] --> OB OB --> REL[Outbox relay / CDC] REL --> K["Kafka: webhook_events"] K --> MATCH[Subscription matcher] MATCH --> REG[(Endpoint registry)] MATCH --> PQ["Per-endpoint queues
+ token buckets"] PQ --> POOL[Delivery worker pool] POOL --> PROX["Egress proxy
SSRF filter, static IPs"] PROX --> EPS[Customer endpoints] POOL --> RS["Retry scheduler
(delay queue)"] RS --> PQ POOL --> ALOG[(Attempt log)] POOL --> DLQ[(Dead letters)] style OB fill:#0e7490,color:#fff style PQ fill:#a855f7,color:#fff style POOL fill:#15803d,color:#fff style PROX fill:#374151,color:#fff style DLQ fill:#9f1239,color:#fff ``` The retry scheduler deserves a note: Kafka has no native delayed delivery, so future attempts live in a store indexed by due time — a Redis sorted set scored by epoch, or tiered delay queues — and a poller republishes due jobs back into the per-endpoint queues. The [distributed job scheduler article](/system-design/articles/design-distributed-job-scheduler) covers the timer-wheel machinery; webhook retries are its most common production customer. ## The delivery attempt One attempt is a short, disciplined transaction with a hostile counterparty. ```mermaid sequenceDiagram participant W as Delivery worker participant P as Egress proxy participant C as Customer endpoint W->>W: build payload — event id, type, created_at W->>W: sign HMAC-SHA256 over "timestamp.body" W->>P: POST with signature headers P->>P: resolve DNS — reject private IP ranges P->>C: HTTPS POST, ~10 s total budget C-->>P: 200 OK in 180 ms P-->>W: success W->>W: write attempt log — status, latency, response snippet ``` The rules, all learned from production: only a **2xx** counts as delivered. Redirects count as failures — Stripe treats 3xx as failure, and following redirects from customer-controlled URLs is an SSRF invitation. Keep a hard total budget of ~10 seconds; a consumer that needs longer should ack fast and process async. TLS is mandatory (Stripe requires TLS 1.2+ in live mode). And every request leaves through an **egress proxy** that resolves DNS itself, rejects private and link-local ranges (a customer registering `http://169.254.169.254/` is either an attacker probing your cloud metadata service or a very confusing bug report), and provides the static egress IPs enterprise customers will demand for their firewalls. Svix's architecture writeup calls out both the SSRF proxy and static IPs as day-one requirements, and they're right. ### Signing and replay protection Sign the concatenation `{timestamp}.{raw_body}` with a per-endpoint secret and send timestamp plus signature in a header. Stripe's `Stripe-Signature` header carries `t=` and `v1=` fields; verification recomputes the HMAC over the exact raw bytes and compares constant-time: ```python def verify(raw_body: bytes, header: str, secret: str, tolerance=300): t, sig = parse_header(header) # t=1721286000, v1=5257a8... if abs(now() - int(t)) > tolerance: # Stripe default: 5 minutes raise ReplayError expected = hmac_sha256(secret, f"{t}.".encode() + raw_body) if not hmac.compare_digest(expected, sig): raise SignatureError ``` Two details carry the security weight. The timestamp lives *inside* the signed string, so a captured request cannot be replayed later with a fresh timestamp — any edit invalidates the signature. And the secret is per-endpoint, so one leaked secret compromises one integration, not your fleet. Rotation works by sending multiple signatures during an overlap window; Stripe lets you keep the previous secret valid for up to 24 hours. The classic consumer-side bug: verifying against a parsed-then-reserialized body instead of the raw bytes. Key order shifts, whitespace changes, verification fails, and three days later the endpoint gets disabled for returning 400s. ## Retries: backoff, jitter, and knowing when to stop A single retry policy, applied per delivery: exponential backoff with **full jitter** (sleep a uniform random value in `[0, backoff]`), capped attempts, then dead-letter. The production schedules are public, so anchor to them rather than inventing numbers: | Provider | Schedule | After exhaustion | | --- | --- | --- | | Stripe (live mode) | Exponential backoff, retried for up to 3 days | May disable the endpoint and notify the owner | | Svix | 8 attempts: now, 5 s, 5 m, 30 m, 2 h, 5 h, 10 h, 10 h (~28 h total) | Marked `Failed`, fires a `message.attempt.exhausted` operational webhook | | A sane default | now, 1 m, 5 m, 30 m, 2 h, 8 h, 24 h, + full jitter | Dead-letter + notify | Why jitter is non-optional at fan-out scale: when a big customer's endpoint comes back after two hours down, every parked delivery for that endpoint becomes due at once. Without jitter you greet their recovery with a synchronized battering ram — the **thundering herd** — and knock them straight back over. With full jitter plus the token bucket from V3, the backlog drains at a rate the endpoint can survive. Same reasoning caps the far end of the schedule: retrying past ~3 days is theater. The endpoint is not "about to recover"; its owner is on vacation, and what they need on return is a dead-letter queue and a redeliver button, not attempt number forty. One consequence worth stating in an interview: retries are why amplification exists (our 1.4× above) and why your attempt log, not your event log, is the biggest table you own. ## Ordering: the honest answer Interviewers love this one because the correct answer is a refusal. Stripe's docs are blunt: delivery order is not guaranteed — your endpoint can see `invoice.paid` before the `invoice.created` that logically precedes it. That is not laziness. At-least-once delivery plus parallel workers plus per-delivery retries makes ordering structurally unaffordable: to guarantee that event 2 never lands before event 1, you must not send 2 until 1 is *acknowledged* — so one failing delivery stalls everything behind it for the length of the retry schedule. Ordering converts every retry into head-of-line blocking, the exact disease V3 cured. | Option | What you promise | What it costs | | --- | --- | --- | | Global FIFO | Total order across all events | Serial delivery per endpoint; a 3-day retry stalls 3 days of events. Nobody ships this | | Per-key FIFO (per object/customer) | Order within one key | HOL blocking scoped to the key; real but bounded damage. Offer as opt-in, if at all | | Best-effort + sequence hints | Nothing, but payloads carry `created_at` and object versions | Consumer logic to discard stale updates | | Thin events (fetch-before-process) | Webhook is a doorbell, not a package | One API read per event; consumer burns rate limit | The design that survives contact with reality is the last two combined: deliver unordered, include timestamps and version numbers, and teach consumers to treat the webhook as a trigger — dedupe on event id, then fetch the object's current state from your API. Hookdeck's consumer guides push the same fetch-before-process pattern, and it quietly solves duplicates too: processing the same doorbell twice is harmless when the action is "read the truth and reconcile." The consumer-side machinery — dedupe keys, idempotent handlers — is exactly the [idempotency article's](/system-design/articles/idempotency-and-exactly-once) territory. ## Circuit breakers and endpoint lifecycle ```mermaid flowchart LR CL["Closed — deliver normally"] -->|"failure rate over threshold"| OP["Open — fail fast, park deliveries"] OP -->|"cooldown elapsed"| HO["Half-open — one probe delivery"] HO -->|"2xx"| CL HO -->|"failure"| OP OP -->|"days of continuous failure"| DIS["Disabled — notify customer"] style CL fill:#15803d,color:#fff style OP fill:#dc2626,color:#fff style HO fill:#ffaa00,color:#111 style DIS fill:#374151,color:#fff ``` The breaker's job is to make failure cheap: an open breaker turns a 10-second timeout into a 0 ms decision, which — per the capacity math — is the difference between 5% of traffic costing 2× your fleet and costing nothing. The terminal state matters just as much. An endpoint that fails for days is dead, and continuing to accrue parked deliveries for it wastes storage and floods the eventual recovery. Svix auto-disables endpoints after all attempts fail continuously for 5 days; Stripe stops retrying after 3 days and may disable a persistently failing endpoint, notifying the owner. Disable, notify loudly through a channel that is not the broken endpoint, and keep the dead letters ready for redelivery. ## Observability customers can self-serve Here is the operational truth that justifies the biggest chunk of engineering time: **failed webhooks generate support tickets, and webhook support tickets are miserable** — "we never got the event" arrives with no request id, no timestamp, and an accusatory tone, and without tooling it turns into an engineer grepping delivery logs on both sides of the fence. The fix is to give customers the logs. Per delivery: attempt history, response codes, response-body snippets, latency, next scheduled retry. Per endpoint: health, breaker state, failure-rate trend. Plus the two recovery verbs Svix ships that every system should copy — *recover failed* (bulk-redeliver everything since a timestamp) and *replay missing* (send events that predate the endpoint's registration). Add an operational meta-webhook for "your endpoint was disabled"… delivered to a different, working channel, because there is a special irony in notifying a dead endpoint about its own death. This is also your monitoring: the same attempt log, aggregated, gives you fleet-wide delivery success rate, p95 delivery latency to healthy endpoints, queue depth per endpoint, and breaker-open counts — the four numbers that belong on the on-call dashboard. ## What breaks **The retry storm you built yourself.** A popular endpoint recovers after an hours-long outage and every parked retry fires in the same minute. Cause: backoff without jitter, or a DLQ redelivery button with no rate limit. The fix is full jitter plus draining through the same per-endpoint token bucket as live traffic — recovery traffic gets no special lane. **The secret-rotation self-own.** A customer rotates the signing secret in their code but not in your dashboard (or vice versa). Every delivery now fails verification with a 400; after the retry window, the endpoint gets disabled — and because the failure looked like *their* bug, nobody paged. This exact sequence is one of the most common causes of "Stripe silently stopped sending us events." Mitigation: overlap windows for rotation, and endpoint-health alerting on sustained 4xx, which is almost always a config bug rather than an outage. **The inline-processing consumer.** The consumer does real work — DB writes, third-party calls — before responding. Under a burst, their processing slows past your timeout; you retry; the retries add more load; their p99 gets worse; more timeouts. A retry death spiral against a healthy-but-slow consumer. Your 429-respecting token bucket dampens it, but the real fix lives in their code: return 200 immediately, enqueue internally, process async. Say it in your docs, then say it again. **Fan-out amplification from one tenant.** A single customer runs a bulk import that emits 10M events in an hour, each fanning to 3 endpoints. Without per-tenant quotas at the matcher, their burst occupies the pipeline ahead of everyone's real-time traffic. Per-endpoint queues protect *destinations*; you also need per-tenant fairness at the *source* — quotas or weighted-fair scheduling at the matcher. **Duplicates hitting a non-idempotent consumer.** Your at-least-once retry, their `charge_customer()` on every POST, one lost 200 response: a double charge with your logo on the timeline. You cannot fix consumer code, but you can make the safe path the obvious one — event ids prominent in payloads and docs, dedupe snippets in every SDK, thin events as the recommended pattern. **Payload schema drift.** You rename a field; ten thousand consumer parsers break simultaneously — a breaking API change shipped by push. Stripe pins each event's shape to the API version configured on the account when the event was created, which is the right model: version the contract, never mutate it retroactively. ## Build vs. buy The honest line, as of mid-2026. **Svix** sells the sending side of this article as an API — retries, signing, the customer portal, breaker logic — and is what "webhooks as a service" mostly means; you trade per-message pricing and a third party in your data path for skipping roughly this entire design. **Hookdeck** covers the opposite direction — receiving infrastructure that ingests, verifies, queues, and replays inbound webhooks for consumers. Building is the right call in three cases: webhooks are a core product surface at hundreds of millions of deliveries a day and vendor unit economics stop working; compliance or data-residency rules forbid third-party egress; or you already run the Kafka-plus-workers substrate and the marginal cost is genuinely small. Be honest about the shape of the work, though: the queue, backoff, and signing core is a week or two of engineering. The delivery portal, redelivery tooling, breaker tuning, SSRF-safe egress, and the long tail of customer debugging is the multi-quarter part — and it's precisely the part vendors amortize across their customer base. ## The decision in practice If you take one design position out of this article, take the refusal: promise **at-least-once, unordered delivery with signed payloads and a visible retry schedule** — and nothing more. Every system that promised more (ordering, exactly-once) either quietly broke the promise or throttled itself into irrelevance defending it. The build order mirrors the failure order. Day one: outbox, queue, workers, backoff with jitter, HMAC — the V2 system, honest and durable, fine for the first tens of thousands of events per day. The day you sign a customer whose endpoint is bad — and that day comes with the customer, not with traffic — you need V3's per-endpoint isolation, because the math is unforgiving: concurrency is rate times latency, and their latency is now your fleet size. Breakers and the disable pathway follow as soon as endpoints start dying permanently, which is to say, immediately. And the delivery portal is not polish to defer; it is the difference between webhook debugging being your customer's self-serve task or your support queue's biggest category. In the interview, walking that ordering — with the 5%-of-traffic-needs-2×-the-fleet arithmetic to justify it — is the difference between "knows the boxes" and "has been paged for this." ## Further reading - [Stripe webhooks documentation](https://docs.stripe.com/webhooks) — the reference consumer contract: signatures, tolerances, retry windows, and the explicit no-ordering disclaimer. - [Svix: webhook architecture design](https://www.svix.com/blog/webhook-architecture-design/) — a vendor writeup that is unusually candid about SSRF proxies, static egress IPs, and per-endpoint rate limiting. - [Svix retry schedule docs](https://docs.svix.com/retries) — the concrete 8-attempt schedule and endpoint-disable semantics quoted above. - [Svix on retry best practices](https://www.svix.com/resources/webhook-best-practices/retries/) — backoff, jitter, and the thundering-herd argument. - [Hookdeck webhook guides](https://hookdeck.com/webhooks/guides) — the consumer's side of the contract: verification, idempotency, async processing. ## Frequently asked questions Q: How long should a webhook system retry failed deliveries? A: The production reference points: Stripe retries with exponential backoff for up to 3 days in live mode, then may disable the endpoint and email the owner. Svix makes 8 attempts — immediately, then 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, 10 hours — a bit over a day in total, then marks the message Failed and fires an operational webhook. Anything past roughly 3 days is theater: an endpoint that has failed for that long needs a human, not a ninth retry. Q: How do HMAC webhook signatures and replay protection work? A: The provider computes HMAC-SHA256 over the string '{timestamp}.{raw_body}' using a per-endpoint secret and sends both timestamp and signature in a header (Stripe's Stripe-Signature carries t= and v1= fields). The consumer recomputes the HMAC over the raw bytes it received, compares in constant time, and rejects timestamps older than a tolerance window — Stripe's default is 5 minutes. Because the timestamp is inside the signed string, an attacker cannot replay an old captured request with a fresh timestamp. Q: Do webhooks guarantee ordering? A: No — and providers say so explicitly: Stripe's documentation states event delivery order is not guaranteed, so an endpoint can see invoice.paid before the invoice.created that logically precedes it. Ordering is structurally incompatible with at-least-once delivery plus parallel workers plus retries: one failed delivery would stall everything behind it. Consumers should treat webhooks as triggers, dedupe on the event id, and fetch current object state from the API rather than replaying event payloads as state transitions. Q: Why does a webhook system need per-endpoint circuit breakers? A: Because worker concurrency equals delivery rate times response time, a small fraction of dead endpoints eats a disproportionate share of the fleet: at 28k deliveries/sec, 5% of traffic timing out at 10 seconds needs 14,000 in-flight slots — double what the healthy 95% needs at ~250 ms. A per-endpoint breaker opens after a failure-rate threshold, fails fast without making HTTP calls, and probes with a single request after a cooldown. Svix also auto-disables endpoints that fail continuously for 5 days. Q: What scale should you design a webhook system for in an interview? A: A defensible illustrative target: 500M events/day with an average fan-out of 1.2 matched endpoints gives 600M deliveries/day — about 7,000/sec average and 28,000/sec at a 4x peak. With ~90% first-attempt success and failed deliveries averaging 5 attempts, that is roughly 840M HTTP attempts/day (1.4x amplification), and at ~2 KB per attempt log entry, about 1.7 TB/day of delivery logs — 50 TB at 30-day retention. Q: Should you build webhook infrastructure or buy it? A: Buy when webhooks are a feature: Svix sells the whole sending pipeline (retries, signing, customer portal) as an API, and Hookdeck covers the receiving side (ingest, verify, queue, replay). Build when webhooks are a core product surface at hundreds of millions of deliveries a day, when per-message vendor pricing stops making sense, or when compliance forbids routing customer data through a third party. The queue-and-retry core is about a week of work; the expensive part is the customer-facing delivery log, redelivery UX, and endpoint debugging tools.