Design a Webhook Delivery System

At 02:14 the pager goes off: webhook delivery p95 has crossed ten minutes. Nothing is down. Kafka is healthy, the workers are healthy, CPU is bored. The cause is dumber than any of your dashboards suggest: your largest customer deployed a broken ingress at midnight, and their endpoint now swallows every POST for 60 seconds before returning a 504. Your delivery workers — one shared pool — are all parked inside that customer's timeout. Every other customer's webhooks are queued behind a wall of sockets held open by a machine that will never answer. One consumer broke; your product's entire asynchronous surface broke with it.

That incident, in some form, is why "design a webhook delivery system" has become a standard interview question at API-first companies, from Stripe-scale fintechs to model providers. It looks like "make an HTTP POST." It is actually a study in multi-tenant isolation, retry theory, and the art of promising exactly as much as you can deliver. This article builds the system the way the interview should go: requirements, capacity, then an architecture where every component exists because something broke without it.

What a webhook delivery system must guarantee

Functionally: customers register HTTPS endpoint URLs, each with a filter over event types (payment.succeeded, run.completed). When a matching event occurs, you POST a signed JSON payload to every subscribed endpoint. Customers can list past deliveries, see request and response details, and redeliver failed or missed events themselves.

The non-functional requirements are where the design actually lives. At-least-once delivery: once you accept an event, it is never silently dropped — not by a worker crash, not by a customer outage, not by a deploy. Latency: seconds at p95 to healthy endpoints; an event that takes 30 seconds to arrive is a support ticket at a fintech. Isolation: endpoint A's outage must not delay endpoint B by even one percentile. Security: payloads are authenticated and replay-resistant, because "POST from the open internet" is otherwise an unauthenticated write path into your customer's billing logic.

Two honest non-goals, and saying them explicitly is worth more in an interview than any diagram. Exactly-once is off the table: if the consumer's 200 response is lost in transit, you must retry, and the consumer sees the event twice. Every real provider documents this — Stripe's docs tell consumers to log event ids and skip duplicates. And ordering is off the table too; we'll get to why, because interviewers push on it.

Webhooks are one of three ways to get data out of a platform — the others being polling and persistent connections; the realtime communication patterns article covers that trade-off. Here we assume webhooks won and design the delivery side.

Capacity: the numbers that shape the architecture

Illustrative Stripe-scale target (not Stripe's actual internals):

Events:        500M events/day
Fan-out:       x1.2 endpoints matched per event  -> 600M deliveries/day
Average rate:  600M / 86,400 s                   ≈ 7,000 deliveries/s
Peak (4x):                                       ≈ 28,000 deliveries/s

Retry amplification:
  ~90% succeed on attempt 1; the rest average ~5 attempts
  600M x (0.9 x 1 + 0.1 x 5) ≈ 840M HTTP attempts/day  (~1.4x)

Worker concurrency (Little's law: in-flight = rate x latency):
  All healthy at ~250 ms:      28k/s x 0.25 s  ≈  7,000 in-flight
  Now 5% hit 10 s timeouts:    1,400/s x 10 s  =  14,000 in-flight
  -> 5% of traffic needs 2x the slots of the other 95%

Attempt log: 840M x ~2 KB ≈ 1.7 TB/day -> 30-day retention ≈ 50 TB

Read the middle block twice, because it is the whole article in four lines. Throughput is not the hard part — 28k requests/sec is a boring number for a fleet of async HTTP workers. Latency variance is the hard part. Your p50 consumer answers in 200 ms; your worst consumer answers never, and you only find out after your timeout budget. Since concurrency is rate times latency, the slowest consumers dominate your capacity even when they are a sliver of traffic. Everything from here on — per-endpoint queues, token buckets, circuit breakers — exists to stop that sliver from setting your fleet size.

Building up to the design

This is a sibling of the notification system problem, with one inversion that changes everything: there, you deliver through a handful of professional providers (APNS, Twilio) with published SLAs. Here, you deliver to thousands of endpoints run by whoever configured them, with no SLA at all. Walk the evolution.

V1: POST inline from application code

def on_payment_succeeded(payment):
    db.commit(payment)
    for hook in subscriptions.matching("payment.succeeded", payment.account):
        requests.post(hook.url, json=payment.as_event(), timeout=10)

This demos fine and fails four ways at once. A slow endpoint adds its latency to your payment path. A crash between the commit and the POST loses the webhook forever. Flip the order and a rollback after the POST announces a payment that never happened. And there is no retry, so any consumer blip is a permanently lost event. The deeper bug is structural: delivery shares a fate with the business transaction, and they need to fail independently.

V2: transactional outbox, queue, worker pool

Write the event to an outbox table in the same database transaction as the business change — atomic by construction. A relay process tails the outbox (or uses CDC) and publishes to a durable log; Kafka fits because you want replay and multi-day retention, though the message queue article covers when SQS is the more honest choice. A subscription matcher consumes events, looks up which endpoints subscribe to that event type for that account, and emits one delivery job per (event, endpoint) pair. A worker pool executes the POSTs.

Now the caller never blocks, crashes lose nothing, and failed deliveries can retry from the queue. This is a genuinely decent system — right up until the 02:14 page from the opening. One shared queue plus one shared worker pool means one 60-second-timeout endpoint occupies workers 240× longer per delivery than a healthy 250 ms endpoint. That is textbook head-of-line blocking, and no amount of "add more workers" fixes a ratio.

V3: per-endpoint queues and token buckets

Give each endpoint its own logical queue and its own delivery budget. Physically this can be thousands of Redis lists, a partitioned "delivery jobs" table, or a Kafka topic keyed by endpoint with a fair scheduler on top — the mechanism matters less than the invariant: workers pull from endpoints round-robin (or weighted-fair), and a per-endpoint token bucket caps in-flight requests and requests/sec to any single destination. A struggling endpoint now queues its own backlog without consuming anyone else's worker slots, and you stop DDoSing customers who are already down — if an endpoint returns 429, the bucket drains to match. This is the same rate limiter machinery you'd build for ingress, pointed outward, and it is backpressure applied per tenant.

V4: circuit breakers for dead consumers

Token buckets protect throughput but still waste real sockets probing a corpse. A per-endpoint circuit breaker stops even trying: track failure rate over a sliding window; past a threshold the breaker opens and deliveries park immediately in the retry schedule without an HTTP call. After a cooldown, a half-open probe sends one real delivery; success closes the breaker and the backlog drains (with jitter — see below), failure re-opens it.

V5: dead letters, redelivery, and the portal

When the retry schedule is exhausted, the delivery moves to a dead-letter state — never deleted, visible to the customer, bulk-redeliverable once they fix their endpoint. Add the customer-facing delivery log and you have the production system.

flowchart LR
    V1["V1: inline POST<br/>shared fate with business txn"] --> V2["V2: outbox + queue + workers<br/>durable, decoupled"]
    V2 --> V3["V3: per-endpoint queues<br/>no head-of-line blocking"]
    V3 --> V4["V4: circuit breakers<br/>dead endpoints fail fast"]
    V4 --> V5["V5: DLQ + redelivery portal<br/>production"]
    style V1 fill:#0e7490,color:#fff
    style V3 fill:#a855f7,color:#fff
    style V5 fill:#15803d,color:#fff

The full architecture

flowchart TD
    SVC1[Billing service] --> OB[(Transactional outbox)]
    SVC2[Platform services] --> OB
    OB --> REL[Outbox relay / CDC]
    REL --> K["Kafka: webhook_events"]
    K --> MATCH[Subscription matcher]
    MATCH --> REG[(Endpoint registry)]
    MATCH --> PQ["Per-endpoint queues<br/>+ token buckets"]
    PQ --> POOL[Delivery worker pool]
    POOL --> PROX["Egress proxy<br/>SSRF filter, static IPs"]
    PROX --> EPS[Customer endpoints]
    POOL --> RS["Retry scheduler<br/>(delay queue)"]
    RS --> PQ
    POOL --> ALOG[(Attempt log)]
    POOL --> DLQ[(Dead letters)]
    style OB fill:#0e7490,color:#fff
    style PQ fill:#a855f7,color:#fff
    style POOL fill:#15803d,color:#fff
    style PROX fill:#374151,color:#fff
    style DLQ fill:#9f1239,color:#fff

The retry scheduler deserves a note: Kafka has no native delayed delivery, so future attempts live in a store indexed by due time — a Redis sorted set scored by epoch, or tiered delay queues — and a poller republishes due jobs back into the per-endpoint queues. The distributed job scheduler article covers the timer-wheel machinery; webhook retries are its most common production customer.

The delivery attempt

One attempt is a short, disciplined transaction with a hostile counterparty.

sequenceDiagram
    participant W as Delivery worker
    participant P as Egress proxy
    participant C as Customer endpoint
    W->>W: build payload — event id, type, created_at
    W->>W: sign HMAC-SHA256 over "timestamp.body"
    W->>P: POST with signature headers
    P->>P: resolve DNS — reject private IP ranges
    P->>C: HTTPS POST, ~10 s total budget
    C-->>P: 200 OK in 180 ms
    P-->>W: success
    W->>W: write attempt log — status, latency, response snippet

The rules, all learned from production: only a 2xx counts as delivered. Redirects count as failures — Stripe treats 3xx as failure, and following redirects from customer-controlled URLs is an SSRF invitation. Keep a hard total budget of ~10 seconds; a consumer that needs longer should ack fast and process async. TLS is mandatory (Stripe requires TLS 1.2+ in live mode). And every request leaves through an egress proxy that resolves DNS itself, rejects private and link-local ranges (a customer registering http://169.254.169.254/ is either an attacker probing your cloud metadata service or a very confusing bug report), and provides the static egress IPs enterprise customers will demand for their firewalls. Svix's architecture writeup calls out both the SSRF proxy and static IPs as day-one requirements, and they're right.

Signing and replay protection

Sign the concatenation {timestamp}.{raw_body} with a per-endpoint secret and send timestamp plus signature in a header. Stripe's Stripe-Signature header carries t= and v1= fields; verification recomputes the HMAC over the exact raw bytes and compares constant-time:

def verify(raw_body: bytes, header: str, secret: str, tolerance=300):
    t, sig = parse_header(header)          # t=1721286000, v1=5257a8...
    if abs(now() - int(t)) > tolerance:    # Stripe default: 5 minutes
        raise ReplayError
    expected = hmac_sha256(secret, f"{t}.".encode() + raw_body)
    if not hmac.compare_digest(expected, sig):
        raise SignatureError

Two details carry the security weight. The timestamp lives inside the signed string, so a captured request cannot be replayed later with a fresh timestamp — any edit invalidates the signature. And the secret is per-endpoint, so one leaked secret compromises one integration, not your fleet. Rotation works by sending multiple signatures during an overlap window; Stripe lets you keep the previous secret valid for up to 24 hours. The classic consumer-side bug: verifying against a parsed-then-reserialized body instead of the raw bytes. Key order shifts, whitespace changes, verification fails, and three days later the endpoint gets disabled for returning 400s.

Retries: backoff, jitter, and knowing when to stop

A single retry policy, applied per delivery: exponential backoff with full jitter (sleep a uniform random value in [0, backoff]), capped attempts, then dead-letter. The production schedules are public, so anchor to them rather than inventing numbers:

Provider	Schedule	After exhaustion
Stripe (live mode)	Exponential backoff, retried for up to 3 days	May disable the endpoint and notify the owner
Svix	8 attempts: now, 5 s, 5 m, 30 m, 2 h, 5 h, 10 h, 10 h (~28 h total)	Marked Failed, fires a message.attempt.exhausted operational webhook
A sane default	now, 1 m, 5 m, 30 m, 2 h, 8 h, 24 h, + full jitter	Dead-letter + notify

Why jitter is non-optional at fan-out scale: when a big customer's endpoint comes back after two hours down, every parked delivery for that endpoint becomes due at once. Without jitter you greet their recovery with a synchronized battering ram — the thundering herd — and knock them straight back over. With full jitter plus the token bucket from V3, the backlog drains at a rate the endpoint can survive. Same reasoning caps the far end of the schedule: retrying past ~3 days is theater. The endpoint is not "about to recover"; its owner is on vacation, and what they need on return is a dead-letter queue and a redeliver button, not attempt number forty.

One consequence worth stating in an interview: retries are why amplification exists (our 1.4× above) and why your attempt log, not your event log, is the biggest table you own.

Ordering: the honest answer

Interviewers love this one because the correct answer is a refusal. Stripe's docs are blunt: delivery order is not guaranteed — your endpoint can see invoice.paid before the invoice.created that logically precedes it. That is not laziness. At-least-once delivery plus parallel workers plus per-delivery retries makes ordering structurally unaffordable: to guarantee that event 2 never lands before event 1, you must not send 2 until 1 is acknowledged — so one failing delivery stalls everything behind it for the length of the retry schedule. Ordering converts every retry into head-of-line blocking, the exact disease V3 cured.

Option	What you promise	What it costs
Global FIFO	Total order across all events	Serial delivery per endpoint; a 3-day retry stalls 3 days of events. Nobody ships this
Per-key FIFO (per object/customer)	Order within one key	HOL blocking scoped to the key; real but bounded damage. Offer as opt-in, if at all
Best-effort + sequence hints	Nothing, but payloads carry created_at and object versions	Consumer logic to discard stale updates
Thin events (fetch-before-process)	Webhook is a doorbell, not a package	One API read per event; consumer burns rate limit

The design that survives contact with reality is the last two combined: deliver unordered, include timestamps and version numbers, and teach consumers to treat the webhook as a trigger — dedupe on event id, then fetch the object's current state from your API. Hookdeck's consumer guides push the same fetch-before-process pattern, and it quietly solves duplicates too: processing the same doorbell twice is harmless when the action is "read the truth and reconcile." The consumer-side machinery — dedupe keys, idempotent handlers — is exactly the idempotency article's territory.

Circuit breakers and endpoint lifecycle

flowchart LR
    CL["Closed — deliver normally"] -->|"failure rate over threshold"| OP["Open — fail fast, park deliveries"]
    OP -->|"cooldown elapsed"| HO["Half-open — one probe delivery"]
    HO -->|"2xx"| CL
    HO -->|"failure"| OP
    OP -->|"days of continuous failure"| DIS["Disabled — notify customer"]
    style CL fill:#15803d,color:#fff
    style OP fill:#dc2626,color:#fff
    style HO fill:#ffaa00,color:#111
    style DIS fill:#374151,color:#fff

The breaker's job is to make failure cheap: an open breaker turns a 10-second timeout into a 0 ms decision, which — per the capacity math — is the difference between 5% of traffic costing 2× your fleet and costing nothing. The terminal state matters just as much. An endpoint that fails for days is dead, and continuing to accrue parked deliveries for it wastes storage and floods the eventual recovery. Svix auto-disables endpoints after all attempts fail continuously for 5 days; Stripe stops retrying after 3 days and may disable a persistently failing endpoint, notifying the owner. Disable, notify loudly through a channel that is not the broken endpoint, and keep the dead letters ready for redelivery.

Observability customers can self-serve

Here is the operational truth that justifies the biggest chunk of engineering time: failed webhooks generate support tickets, and webhook support tickets are miserable — "we never got the event" arrives with no request id, no timestamp, and an accusatory tone, and without tooling it turns into an engineer grepping delivery logs on both sides of the fence. The fix is to give customers the logs. Per delivery: attempt history, response codes, response-body snippets, latency, next scheduled retry. Per endpoint: health, breaker state, failure-rate trend. Plus the two recovery verbs Svix ships that every system should copy — recover failed (bulk-redeliver everything since a timestamp) and replay missing (send events that predate the endpoint's registration). Add an operational meta-webhook for "your endpoint was disabled"… delivered to a different, working channel, because there is a special irony in notifying a dead endpoint about its own death.

This is also your monitoring: the same attempt log, aggregated, gives you fleet-wide delivery success rate, p95 delivery latency to healthy endpoints, queue depth per endpoint, and breaker-open counts — the four numbers that belong on the on-call dashboard.

What breaks

The retry storm you built yourself. A popular endpoint recovers after an hours-long outage and every parked retry fires in the same minute. Cause: backoff without jitter, or a DLQ redelivery button with no rate limit. The fix is full jitter plus draining through the same per-endpoint token bucket as live traffic — recovery traffic gets no special lane.

The secret-rotation self-own. A customer rotates the signing secret in their code but not in your dashboard (or vice versa). Every delivery now fails verification with a 400; after the retry window, the endpoint gets disabled — and because the failure looked like their bug, nobody paged. This exact sequence is one of the most common causes of "Stripe silently stopped sending us events." Mitigation: overlap windows for rotation, and endpoint-health alerting on sustained 4xx, which is almost always a config bug rather than an outage.

The inline-processing consumer. The consumer does real work — DB writes, third-party calls — before responding. Under a burst, their processing slows past your timeout; you retry; the retries add more load; their p99 gets worse; more timeouts. A retry death spiral against a healthy-but-slow consumer. Your 429-respecting token bucket dampens it, but the real fix lives in their code: return 200 immediately, enqueue internally, process async. Say it in your docs, then say it again.

Fan-out amplification from one tenant. A single customer runs a bulk import that emits 10M events in an hour, each fanning to 3 endpoints. Without per-tenant quotas at the matcher, their burst occupies the pipeline ahead of everyone's real-time traffic. Per-endpoint queues protect destinations; you also need per-tenant fairness at the source — quotas or weighted-fair scheduling at the matcher.

Duplicates hitting a non-idempotent consumer. Your at-least-once retry, their charge_customer() on every POST, one lost 200 response: a double charge with your logo on the timeline. You cannot fix consumer code, but you can make the safe path the obvious one — event ids prominent in payloads and docs, dedupe snippets in every SDK, thin events as the recommended pattern.

Payload schema drift. You rename a field; ten thousand consumer parsers break simultaneously — a breaking API change shipped by push. Stripe pins each event's shape to the API version configured on the account when the event was created, which is the right model: version the contract, never mutate it retroactively.

Build vs. buy

The honest line, as of mid-2026. Svix sells the sending side of this article as an API — retries, signing, the customer portal, breaker logic — and is what "webhooks as a service" mostly means; you trade per-message pricing and a third party in your data path for skipping roughly this entire design. Hookdeck covers the opposite direction — receiving infrastructure that ingests, verifies, queues, and replays inbound webhooks for consumers. Building is the right call in three cases: webhooks are a core product surface at hundreds of millions of deliveries a day and vendor unit economics stop working; compliance or data-residency rules forbid third-party egress; or you already run the Kafka-plus-workers substrate and the marginal cost is genuinely small. Be honest about the shape of the work, though: the queue, backoff, and signing core is a week or two of engineering. The delivery portal, redelivery tooling, breaker tuning, SSRF-safe egress, and the long tail of customer debugging is the multi-quarter part — and it's precisely the part vendors amortize across their customer base.

The decision in practice

If you take one design position out of this article, take the refusal: promise at-least-once, unordered delivery with signed payloads and a visible retry schedule — and nothing more. Every system that promised more (ordering, exactly-once) either quietly broke the promise or throttled itself into irrelevance defending it.

The build order mirrors the failure order. Day one: outbox, queue, workers, backoff with jitter, HMAC — the V2 system, honest and durable, fine for the first tens of thousands of events per day. The day you sign a customer whose endpoint is bad — and that day comes with the customer, not with traffic — you need V3's per-endpoint isolation, because the math is unforgiving: concurrency is rate times latency, and their latency is now your fleet size. Breakers and the disable pathway follow as soon as endpoints start dying permanently, which is to say, immediately. And the delivery portal is not polish to defer; it is the difference between webhook debugging being your customer's self-serve task or your support queue's biggest category. In the interview, walking that ordering — with the 5%-of-traffic-needs-2×-the-fleet arithmetic to justify it — is the difference between "knows the boxes" and "has been paged for this."

Further reading

• Stripe webhooks documentation — the reference consumer contract: signatures, tolerances, retry windows, and the explicit no-ordering disclaimer.
• Svix: webhook architecture design — a vendor writeup that is unusually candid about SSRF proxies, static egress IPs, and per-endpoint rate limiting.
• Svix retry schedule docs — the concrete 8-attempt schedule and endpoint-disable semantics quoted above.
• Svix on retry best practices — backoff, jitter, and the thundering-herd argument.
• Hookdeck webhook guides — the consumer's side of the contract: verification, idempotency, async processing.