Design a Notification System (Push, Email, SMS)

The problem

Slack sends you a push when a thread you're in goes active at 2 AM. Amazon fires an email the moment your package is out for delivery. WhatsApp pings your phone, your laptop, and your watch simultaneously. These look like trivial "call APNS and move on" operations — until you're responsible for one. At 1 billion notifications a day across push, email, and SMS, the surface area for failure is enormous.

The core of this system is a multi-channel fan-out: a single upstream event (order shipped, payment received, new follower) must be routed to the right channels, for the right user, through the right external provider, with the right rendered copy in the right language. That routing pipeline has to respect opt-outs, honor quiet hours, and survive flaky provider APIs — all without ever double-sending the same message.

Two engineering tensions dominate the design. First, at-least-once delivery vs. exactly-once user experience: Kafka and retry logic guarantee you'll attempt every notification, but that means duplicates are a constant threat. Every piece of the pipeline needs idempotency, or users start filing bugs like "I got the same confirmation email six times." Second, fan-out amplification: a single celebrity-account event on a social platform can fan out to millions of followers instantly. The queueing and routing layer has to absorb that burst without the pipeline backing up or the providers rate-limiting you into a blacklist.

Functional requirements

• Send notifications via push (mobile), email, SMS, in-app.
• Support templates with variables (Hi {name}, your order #{id} shipped).
• User preferences: opt-in/out per channel, per category.
• Scheduled and triggered notifications.
• Track delivery status and engagement.

Non-functional

• 1B notifications/day at peak.
• Delivery within 30 seconds for transactional events.
• 99.9% delivery success on healthy paths.
• Idempotent — never double-send.
• Auditable.

Capacity

Dimension	Estimate	How we got there
Volume	1B notifications/day	Given
Throughput (avg)	~12k/sec	1B ÷ 86,400 s
Throughput (peak)	~50k/sec	~4× average burst
Channel split	70% push · 20% email · 5% SMS · 5% in-app	Typical app ratio
Per-event payload	~2 KB	Template + variables + metadata
Retention	30 days delivery audit	SLA / compliance
Audit log storage	60 TB	1B × 2 KB × 30 days

Takeaway: the bottleneck is not storage (60 TB is manageable) — it is throughput. At 50k/sec peak across four external providers (APNS, FCM, SES/SendGrid, Twilio), the system lives or dies on queue depth management and per-provider rate limiting.

Building up to the design

A notification system is the "looks simple, fails in horrifying ways" problem. The architecture below has half a dozen pieces; each appears the moment you hit a wall. Walk forward.

V1: Synchronous send from the calling service

def on_order_shipped(order):
    user = users.get(order.user_id)
    apns.push(user.device_token, "Your order has shipped!")
    db.save_order(order)

This works for tiny apps and gets you a demo. The problem is that APNS is a third-party network call — it's fast on a good day and completely unreachable on a bad one. When it's slow, your order shipped endpoint times out. When the caller retries, you send the notification twice. You've tightly coupled your order flow to a flaky external dependency, and there's no seam to pull them apart.

V2: Enqueue via a queue, send in a worker

order shipped → push a NotificationEvent to a queue (Kafka, SQS, RabbitMQ). A separate worker reads the queue and calls APNS.

Now the caller returns instantly. APNS slowness no longer blocks business logic, and failures are retryable because the event is sitting safely in the queue. The problem is that one queue plus one worker can't handle 50k notifications per second across multiple channels. And if a user opts out of marketing notifications after the event is already queued, you'll still send it — because you haven't separated the routing decision from the enqueue step.

V3: Channel-specific workers + user preference check

Split into channel-specific workers — a push worker, an email worker, an SMS worker. Each pulls from the queue, checks whether this user wants this kind of notification on this channel, then sends. Each channel now scales independently, and opt-outs are honored at delivery time, not just at enqueue. The next crack appears when you try to change notification copy: every string is hard-coded in the worker, so a wording change means a redeploy.

V4: Template engine + render at send

A separate template service stores templates by (template_id, channel, locale). The worker fetches the template, renders it with the event's variables, then sends. Now you can edit a template and all queued messages pick up the change. You also get safe variable substitution that prevents HTML or SMS injection. The next thing that bites you: services retry on network blips. You'll start sending duplicate notifications.

V5: Idempotency + dedup

Every event carries an idempotency_key (e.g. order:42:shipped). Workers check a recent-deliveries cache (Redis with a 24-hour TTL) before sending — same key in that window means skip. This handles Kafka redelivery, API retries, and the occasional provider that returns success twice.

V6: Rate limiting + provider failover

Two more things you need before calling this production-ready. First, per-user rate caps: never send the same user more than N notifications per hour for the same category. Second, per-provider rate limiting so you don't blow your Twilio quota. And when a provider goes down — SendGrid having an incident, say — you want to route email through SES automatically until it recovers.

V7: Audit + delivery tracking

Every send, skip, and failure writes to an audit log. Failures requeue with exponential backoff up to N retries; after that, the event lands in a dead-letter queue and pages the on-call. This is the production design — every box in the diagram below earned its place by fixing something that broke.

flowchart LR
    V1[V1: sync send<br/>caller blocked] --> V2[V2: + queue + worker<br/>decoupled]
    V2 --> V3[V3: + channels + prefs<br/>scaled, polite]
    V3 --> V4[V4: + templates<br/>no-deploy changes]
    V4 --> V5[V5: + idempotency<br/>no dupes]
    V5 --> V6[V6: + rate limit + failover<br/>resilient]
    V6 --> V7[V7: + audit + retries<br/>production]
    style V1 fill:#0e7490,color:#fff
    style V3 fill:#15803d,color:#fff
    style V5 fill:#ff6b1a,color:#0a0a0f
    style V7 fill:#a855f7,color:#fff

High-level architecture

flowchart TD
    SRC1[Order service] --> ENQ[Notification API]
    SRC2[User service] --> ENQ
    SRC3[Background jobs] --> ENQ

    ENQ --> KAFKA[Kafka: notify_events]

    KAFKA --> ROUTER[Router]
    ROUTER --> TPL[Template Engine]
    TPL --> ROUTER
    ROUTER --> PREFS[(User Preferences)]
    PREFS --> ROUTER

    ROUTER --> CHN_PUSH[Push Worker]
    ROUTER --> CHN_EMAIL[Email Worker]
    ROUTER --> CHN_SMS[SMS Worker]
    ROUTER --> CHN_IN[In-app Worker]

    CHN_PUSH --> APNS[APNS]
    CHN_PUSH --> FCM[FCM]
    CHN_EMAIL --> SES[SES / SendGrid]
    CHN_SMS --> TWIL[Twilio]
    CHN_IN --> WS[WebSocket Gateway]

    CHN_PUSH --> AUDIT[(Audit Log)]
    CHN_EMAIL --> AUDIT
    CHN_SMS --> AUDIT
    CHN_IN --> AUDIT

    style ENQ fill:#ff6b1a,color:#0a0a0f
    style KAFKA fill:#a855f7,color:#fff
    style ROUTER fill:#15803d,color:#fff

API

POST /notifications/send
{
  "user_id": 12345,
  "template_id": "order_shipped",
  "data": { "order_id": 99, "tracking_url": "..." },
  "channels": ["push", "email"],            // or auto from prefs
  "idempotency_key": "order_99_shipped",
  "send_at": "2026-02-15T14:00:00Z"          // optional, default now
}

Returns 202 Accepted with a notification_id.

Why a queue is mandatory

Provider API calls are fast in the happy path but highly variable and periodically unavailable. SendGrid's API accepts a message in ~22 ms p50 but spikes to hundreds of milliseconds under load, and then your email still has to be delivered to the recipient's mailbox — total end-to-end latency is seconds to minutes. Twilio's SMS API similarly accepts requests in ~65 ms p50, but actual delivery to the handset adds carrier-dependent latency (seconds to tens of seconds). APNS round-trip is typically fast (tens of milliseconds per message on a warm HTTP/2 connection), but a single persistent connection can saturate or the provider can be temporarily unreachable. None of these are reliable enough to call synchronously on the critical path.

A request hits the API → enqueue → return 202. Workers consume the queue, deliver, retry on failure.

Kafka is a strong choice: high throughput, partitioning by user_id (in-order per user), retention for replay. If a worker crashes, Kafka redelivers from the last committed offset, which is exactly why idempotency is non-negotiable.

But "use Kafka" is a real decision, not a reflex — the canonical follow-up is why not SQS or RabbitMQ.

Property	Kafka	SQS	RabbitMQ
Per-user ordering	Yes, via partition key	FIFO queues only, low throughput cap	Per-queue only, awkward per-user
Replay after a bug	Yes — reset offset, reprocess days	No — a consumed message is gone	No — ack removes it
Native delayed delivery	No	Yes (up to 15 min)	Via plugin / TTL + DLX
Throughput	Very high	Managed, high, but per-op billed	Moderate; degrades with deep queues
Ops burden	You run (or pay for) a cluster	Zero — fully managed	You run a cluster

Kafka wins here because per-user ordering and multi-day replay both matter — replay is what lets you re-drain a DLQ after fixing a template bug, and ordering keeps a "shipped" notification from landing before "order confirmed". SQS is the better answer at smaller scale or when you want zero ops and can live without replay; its native per-message delay would even cover the send_at case for free. RabbitMQ fits when you need rich routing topologies more than raw throughput. See the message queue design article for the trade-offs in depth.

Fan-out and traffic isolation

The second tension from the opening — fan-out amplification — lives here. A celebrity account with 40 million followers posts once, and that single upstream event has to become 40 million per-user notifications. You don't want the API expanding that inline; you want a fan-out service that consumes the high-follower event, pages through the follower list in batches (say 10k recipients each), and emits per-user NotificationEvents back onto Kafka. Expansion becomes just another consumer that can lag without blocking anything upstream.

The real danger is what that burst does to other traffic. Drop 40 million marketing fan-out events onto the same topic as a password-reset and the reset waits behind them, blowing the 30-second transactional SLA. Isolate the two: separate Kafka topics (or a priority partition scheme) for transactional vs. bulk, with transactional consumers scaled to drain their topic in seconds and bulk consumers allowed to run minutes behind. A marketing blast then starves only itself. The math makes the point — 40 million pushes at APNS's ~4k/sec per-client ceiling is nearly three hours on a single client, and even a pool of a few dozen clients is minutes of steady sending. Fine for a digest, unacceptable for a 2FA code. The topic split is what keeps the 2FA code out of that line.

Templates

Templates separate what to say from how to render it for each channel.

template: order_shipped
versions:
  push:    "Your order #{order_id} shipped! 📦"
  email:
    subject: "Your order is on the way"
    html:    "Hi {name}, your order shipped. Track it at {tracking_url}."
  sms:     "Order #{order_id} shipped. Track: {tracking_url}"

Stored in a templates DB; rendered just-in-time with the event payload. Supports per-language variants (order_shipped:en, order_shipped:fr).

Idempotency

Every notification has an idempotency_key. Before sending, the worker checks whether we already processed that key.

def process(event):
    if redis.set(f"idem:{event.idempotency_key}", 1, nx=True, ex=86400):
        send(event)
    else:
        log("duplicate, skipped")

SET NX returns True on the first call and None on every retry — both are handled correctly by the if conditional (the redis-py library translates the raw Redis "OK" response to True; on a no-op it returns None, which is falsy). This matters for two scenarios: Kafka redelivers from the last committed offset after a worker crash, and a caller retries the API after a network timeout. The 24-hour TTL covers both.

There's a trade-off buried in the ordering. Claiming the key before the provider call means that if the worker crashes after the SET succeeds but before the send completes, the Kafka redelivery sees the key and skips — the notification is silently lost for the 24-hour window, turning at-least-once into at-most-once for that one event, in tension with the 99.9% delivery target. Two ways out: set the key with a short in-flight state and finalize it to done only after the send returns, so a crash leaves the key reclaimable; or accept the rare loss and reconcile against the audit log. Neither is free, so pick deliberately.

Note also what this key does not cover: a provider call that times out after the provider already delivered, where the worker then retries in-process. The same worker still holds the key, so SET NX can't see the retry — that duplicate has to be deduped provider-side (APNS apns-collapse-id, an idempotency header on the SES or Twilio call) rather than in Redis.

flowchart LR
    EVT[Event arrives] --> CHK{"Redis SET NX<br/>idempotency_key"}
    CHK -->|"OK — first time"| SEND[Send to provider]
    CHK -->|"nil — seen before"| SKIP[Skip + log]
    SEND --> LOG[Write audit log]
    SKIP --> LOG
    style CHK fill:#ff6b1a,color:#0a0a0f
    style SEND fill:#15803d,color:#fff
    style SKIP fill:#ffaa00,color:#0a0a0f
    style LOG fill:#0e7490,color:#fff

User preferences

Stored in a fast-access KV store:

user:42:notif_prefs = {
  "push":  ["transactional", "social"],
  "email": ["transactional"],
  "sms":   ["transactional"]
}

Router enforces: an event with category social and only-email user → drop the email send (user opted out).

GDPR / regulatory: every notification must respect "do not contact" lists; transactional vs. marketing categories matter for legal compliance.

Push notifications: APNS / FCM

APNS (Apple Push Notification service): persistent HTTP/2 connection from your servers to Apple's servers. Apple delivers to the device. A single client instance (multiplexing concurrent HTTP/2 streams) can sustain roughly 4,000+ pushes/sec in community benchmarks (Apple does not publish a formal limit); run one instance per CPU core to scale throughput further.

FCM (Firebase Cloud Messaging): the canonical push rail for Android (successor to the deprecated GCM). FCM also delivers to iOS by tunneling through APNS and to the web via the Web Push protocol — meaning for cross-platform apps, FCM can be a single provider abstraction in front of both Android and APNS delivery.

sequenceDiagram
    participant App as Push Worker
    participant APNS
    participant Device
    App->>APNS: send(token, payload)
    APNS->>Device: deliver
    APNS-->>App: 200 OK / 4xx (bad token, etc.)
    Device-->>APNS: feedback (token revoked, etc.)

Each app has a device token per install. Tokens become invalid when a user reinstalls the app, the OS rotates the token, or the app is deleted. APNS returns HTTP 410 (Unregistered) for an inactive token, and 400 (DeviceTokenNotForTopic) if the token's topic (bundle ID) mismatches the certificate. On a 410, remove the token from your DB immediately.

One operational note: APNS deliberately delays returning 410 on a sliding, undocumented schedule — this is a privacy-protecting design so providers cannot track app installs and uninstalls in real time. You may send to a stale token for minutes or longer before APNS starts returning 410. Your audit log and a periodic token-validity sweep help detect stale tokens over time. Do not infer an app is still installed just because APNS hasn't returned 410 yet.

Delivery guarantees

Three things worth being explicit about here. Kafka gives you at-least-once delivery, which means duplicates are possible if you don't dedup with idempotency keys. Channels have no ordering guarantees relative to each other — push may arrive before email for the same event, or after. And provider acceptance is not the same as device delivery: even after APNS or FCM returns 200 OK, a notification may never reach the user's screen if their device is offline, the app is uninstalled, or a carrier filters the SMS. Build your audit log accordingly — "accepted by provider" and "delivered to user" are two different states.

Rate limiting and quiet hours

You don't want to spam users. The Router enforces two layers of policy.

The first is a per-user rate cap: for example, max 5 marketing notifications per hour per user. This is a Redis sliding window keyed by (user_id, category) — cheap to check, no DB write per send. See the rate limiter article for the algorithm.

The second is quiet hours: 10 PM – 7 AM in the user's local timezone, suppress non-critical notifications. Quiet-hours logic must use the user's current timezone, not the timezone at enqueue time — always recompute at dispatch, because users travel.

flowchart LR
    EVT2[Event ready to send] --> RC{"Rate cap OK?<br/>(sliding window)"}
    RC -->|"over limit"| DROP[Drop or delay]
    RC -->|"within limit"| QH{"Quiet hours<br/>in user TZ?"}
    QH -->|"yes + non-critical"| SCHED[Schedule for morning]
    QH -->|"yes + critical"| DISPATCH[Dispatch to worker]
    QH -->|"no"| DISPATCH
    style RC fill:#ff6b1a,color:#0a0a0f
    style QH fill:#a855f7,color:#fff
    style DISPATCH fill:#15803d,color:#fff
    style SCHED fill:#ffaa00,color:#0a0a0f

Scheduled and deferred sends

Both send_at in the API and the "schedule for morning" branch of quiet hours need the same machinery, and Kafka doesn't provide it. Kafka has no native delayed delivery — you can't publish an event stamped "release at 9 AM" and expect the broker to hold it. Put scheduled events in a durable store indexed by due time: a Postgres table with an index on send_at, or a Redis sorted set scored by the due epoch. A scheduler polls the near-term slice every few seconds (ZRANGEBYSCORE now-ε now, or a WHERE send_at <= now query) and republishes each due event onto notify_events, where it rejoins the normal Router path. Quiet-hours deferral is the same operation with the due time set to the local-morning boundary.

Keep the poll window tight and bucket the store by due-time so the scan stays cheap; the classic way this component falls over at volume is a single unindexed "scan the whole table every minute". For the general pattern — timer wheels, tiered delay queues, exactly-once firing — see the distributed job scheduler article.

Aggregation / digesting

Aggregation deserves a closer look. The naive system fires one notification per event; a popular post can generate hundreds of likes in minutes. We collapse them:

flowchart LR
    E1[Like by Alice] --> AGG[Aggregator window]
    E2[Like by Bob] --> AGG
    E3[Like by Carol] --> AGG
    E4[5 min timer] --> AGG
    AGG --> NOTIF["3 people liked your post"]
    style AGG fill:#ff6b1a,color:#0a0a0f

Implementation: Redis sorted set keyed by (user_id, post_id, action), members are actor IDs. After a quiet window or N events, flush as a digest.

Tracking and engagement

For each notification, track:

• created_at
• dispatched_at (sent to provider)
• delivered_at (provider confirmed delivery)
• opened_at (user tapped/opened — best-effort, varies by channel)
• clicked_at (user clicked a link)

Stored in an event store (Kafka → BigQuery / Snowflake) for analytics.

A/B test templates by varying template_version and measuring open rates. Bad templates (high opt-out, low CTR) get retired.

Failures

Provider outages

When APNS, FCM, or Twilio returns 5xx or times out, the worker retries with exponential backoff. After N failed attempts, the event goes to a dead-letter queue — visible to ops, inspectable, and either re-drainable once the provider recovers or forwarded to a fallback channel (APNS down → try email). The key is that the event is never silently dropped; it's always somewhere in the system with a status you can query.

flowchart LR
    SEND2[Send to provider] --> OK{"200 OK?"}
    OK -->|yes| DONE[Write audit: accepted]
    OK -->|no| RETRY{"Retries < N?"}
    RETRY -->|yes| BACKOFF["Wait 2^n seconds<br/>retry"]
    RETRY -->|no| DLQ[Dead-letter queue<br/>+ alert]
    BACKOFF --> SEND2
    style SEND2 fill:#0e7490,color:#fff
    style OK fill:#ff6b1a,color:#0a0a0f
    style DONE fill:#15803d,color:#fff
    style DLQ fill:#ff2e88,color:#fff

Invalid tokens

When a provider returns 410 or a similar "bad token" status, mark the device token as dead in your DB and don't try again. Retrying bad tokens wastes quota and can get your sending IP flagged.

User opts out mid-flight

A notification was queued before the opt-out arrived. Because the Router checks preferences at dispatch time — not at enqueue time — the opt-out applies even to messages already sitting in Kafka. This is the reason preference enforcement happens late in the pipeline.

Timezone changes

If a user is mid-flight between timezones, quiet-hours logic must use their current timezone at the moment of dispatch. Storing the timezone at enqueue time would suppress or send notifications at the wrong local hour. Recompute at dispatch.

Storage

Data	Store
Notification event	Kafka (transient) → S3 (long-term audit)
User preferences	DynamoDB / Redis (read-heavy)
Device tokens	Postgres (small per user)
Templates	Postgres + cache
Delivery status	DynamoDB or Cassandra (write-heavy, append)

Hot path: send a notification

sequenceDiagram
    participant Caller
    participant API
    participant Kafka
    participant Router
    participant Push Worker
    participant APNS
    Caller->>API: POST /send
    API->>Kafka: enqueue
    API-->>Caller: 202
    Kafka->>Router: deliver
    Router->>Router: check prefs, render template
    Router->>Push Worker: dispatch
    Push Worker->>APNS: send
    APNS-->>Push Worker: 200
    Push Worker->>Audit: delivery_log

End-to-end ~1–5 seconds for transactional. Marketing / digests can be minutes to hours.

Things to discuss in an interview

• Why a queue (Kafka) between API and workers — provider latency.
• Idempotency with SET NX keys.
• Multi-channel routing based on prefs.
• Aggregation to avoid spam.
• Template engine with per-channel renderers.
• Provider quirks — token expiry, rate limits, regional outages.

Things you should now be able to answer

• Why is the API 202 Accepted instead of 200 OK?
• What does idempotency mean here, and how do you achieve it?
• How do you avoid spamming a user with 50 notifications in a minute?
• What happens when APNS goes down for an hour?
• How do you ensure quiet hours work across timezones?

Further reading

• "Notifications at Scale" — Slack engineering blog
• APNS HTTP/2 protocol docs
• Pinterest's notification system blog post