#architecture
19 articles
Sandboxing AI Agents: Containers, gVisor, and MicroVMs
Sandboxing AI agents with containers, gVisor, Firecracker microVMs, and Wasm: honest boot and overhead numbers, and why egress filtering matters most.
Voice Agent Frameworks: Vapi vs LiveKit vs Pipecat
Compare Vapi, LiveKit Agents, and Pipecat across latency, cost, telephony, and control — with the exact break-even math to pick the right abstraction.
Streaming LLM Responses: The Engineering Nobody Talks About
SSE vs WebSockets, partial-JSON parsing for streamed tool calls, backpressure, cancellation, and rendering streamed markdown without flicker — the complete engineering guide.
Concurrency in Agent Systems: Parallel Tools, Fan-Out, and Rate Limits
How to run agent tool calls in parallel, design fan-out/join patterns, schedule around provider rate limits, and cancel in-flight work without corrupting state.
From Prototype to Production: Scaling an AI Product
The concrete engineering decisions — model routing, caching strategy, async batching, context engineering, and cost governance — that turn a demo into a system handling real load.
Build vs Buy and Small vs Large: The Framework for Model and Infrastructure Decisions
Self-hosting beats APIs only above ~$4,200/month. Small models win on latency and cost. Learn the decision framework that prevents both premature optimization and missed savings.
Agent Memory Architectures: Episodic, Semantic, Procedural
How production agents store and recall knowledge across sessions — episodic logs, semantic stores, procedural caches, and sane forgetting policies.
Cascaded pipeline vs native speech-to-speech: when to use each
Chain STT, LLM, and TTS for control — or use one speech-native model for latency. The real tradeoffs, with numbers, and when each architecture wins.
The Voice Pipeline: STT to LLM to TTS End-to-End
Build the STT to LLM to TTS voice pipeline end-to-end — streaming each stage, budgeting latency, and hitting sub-second time-to-first-audio.
Fallbacks, Retries, and Circuit Breakers: Building Reliable LLM Pipelines
How to handle 429s, provider outages, and transient failures in production LLM systems: exponential backoff, fallback chains, and circuit breakers explained with real code.
Model Context Protocol: The USB-C for AI Agents
How MCP standardizes tool integration for AI agents — the client-server protocol, why OpenAI and Microsoft adopted it, tool schema pitfalls, and security risks you cannot ignore.
GraphRAG and Structured Retrieval for Multi-Hop Reasoning
How knowledge graphs replace flat vector indexes to answer multi-hop and global queries flat retrieval cannot handle — with real costs, when to use it, and when not to.
Multi-Agent Orchestration: Patterns That Survive Production
Six orchestration topologies for multi-agent systems — from orchestrator-worker to swarm and adaptive planning — with real failure rates and cost math.
Document AI and OCR in 2026: When to Use VLMs vs Classic Engines
VLMs now beat classical OCR on noisy documents, but the right choice depends on layout complexity, volume, and cost. A decision framework with real benchmarks.
Model routing and cascades: sending the right query to the right model
How to build rule-based routers, intent classifiers, and confidence-based cascades that cut your LLM API bill 60–85% while preserving 95% of frontier quality.
Agent Memory: From Context Window to Persistent Knowledge
How agents remember things across tool calls, sessions, and deployments — in-context working memory, episodic stores, semantic vector DBs, and when each breaks.
The AI Gateway Pattern
How a thin proxy between your app and every LLM provider gives you routing, fallback, cost tracking, and semantic caching without touching application code.
ReAct, Reflexion, and Planning Patterns
How ReAct, Reflexion, Plan-and-Execute, and LATS differ as agent planning strategies — with honest benchmarks, failure modes, and a decision framework for production.
Reference architecture for a production LLM application
The seven-layer blueprint for production LLM apps: ingress, AI gateway, context assembly, model execution, output validation, observability, and feedback store.