#reliability
9 articles
Durable Execution: How Temporal-Style Replay Works
Durable execution is event sourcing plus deterministic replay applied to your code. How Temporal-style engines journal effects, replay crashes, and break.
Design a Webhook Delivery System
Design a webhook delivery system: fan-out queues, retries with backoff and jitter, HMAC signing, circuit breakers, and dead-letter redelivery at Stripe scale.
Model Context Protocol (MCP) and Tool-Use Infrastructure
How MCP lets any LLM host discover and call any tool server — collapsing N×M integrations to N+M, with the security, reliability, and token math that matters.
Design an LLM Gateway (AI Gateway & Model Router)
Design an LLM gateway: one proxy in front of every model provider, with automatic failover, semantic caching, per-team budgets, and SSE streaming under 50 ms.
Design an AI Agent Platform
Design an AI agent platform: durable checkpointed runs, idempotent tool calls, tiered memory, and per-tenant safety guardrails around an unreliable LLM core.
Design a Distributed Job Scheduler (cron at scale)
Run millions of scheduled and recurring jobs reliably — at-least-once execution, leader election, sharded time-wheels, and exactly-once side effects via idempotency.
Backpressure & Flow Control
What happens when a fast producer overwhelms a slow consumer? Backpressure, bounded buffers, load shedding, and why unbounded queues are a trap.
Idempotency & Exactly-Once Semantics
Networks retry, so your operations will run twice. Idempotency keys, dedup, and why "exactly-once delivery" is a myth but "exactly-once effect" is achievable.
Reliability and Failure Patterns
Timeouts, retries with backoff, circuit breakers, bulkheads, deadlines, hedged requests, and graceful degradation — the patterns that keep distributed systems standing.