#cost-optimization
36 articles
Mixture of Experts: why every frontier model went sparse
How mixture of experts works — routers, top-k selection, shared experts — and the VRAM and batching math that decides when sparse models save you money.
Long Context vs RAG: The Million-Token Question
When does stuffing a million tokens beat retrieval? The empirics on lost-in-the-middle accuracy, context-stuffing costs, and when RAG still wins at 1M tokens.
From Prototype to Production: Scaling an AI Product
The concrete engineering decisions — model routing, caching strategy, async batching, context engineering, and cost governance — that turn a demo into a system handling real load.
Build vs Buy and Small vs Large: The Framework for Model and Infrastructure Decisions
Self-hosting beats APIs only above ~$4,200/month. Small models win on latency and cost. Learn the decision framework that prevents both premature optimization and missed savings.
Throughput vs Latency: The GPU Economics of LLM Serving
How tokens/sec, cost per token, TTFT SLAs, and GPU utilization interact — and the engineering decisions that separate a profitable inference deployment from a money pit.
Limits and failure modes: overthinking, underthinking, and when reasoning breaks
The empirically-documented ways reasoning models fail in production — overthinking, underthinking, sycophantic traces, benchmark gap — and how to detect and route around them.
RAG evaluation, failure modes, and when to use long context or fine-tuning instead
Measure a RAG pipeline with RAGAS, spot the failure modes that kill production quality, and decide between RAG, long context, or fine-tuning.
Metadata Filtering, Scaling, and Production Cost Control
How metadata filtering interacts with ANN indexes, multi-tenancy patterns, quantization tradeoffs, and why your vector DB bill runs 2.5–4x over forecast at scale.
The LLM Gateway Pattern: Rate Limiting, Cost Attribution, and Observability
How a centralized AI gateway gives you virtual keys, dual TPM/RPM rate limiting, per-request cost tagging, and provider fallback — and when to build vs buy.
Choosing a serving engine: vLLM, SGLang, TensorRT-LLM, and Ollama in 2026
A decision framework for picking the right LLM inference engine based on workload, hardware, and ops complexity — grounded in 2025-2026 benchmarks.
Fallbacks, Retries, and Circuit Breakers: Building Reliable LLM Pipelines
How to handle 429s, provider outages, and transient failures in production LLM systems: exponential backoff, fallback chains, and circuit breakers explained with real code.
Context window management for agents: select, compress, isolate
How to keep agents from degrading when the context fills up — retrieval-based selection, compression strategies, multi-agent isolation, and the numbers behind each choice.
Context Windows, KV Cache, and the Reality Behind 1M Token Claims
What context windows actually cost in GPU memory, why effective recall degrades before the advertised limit, and how to make smart architectural choices about long context.
Open vs. closed models: a decision framework, not a religion
A practical framework for choosing between open-weight and closed frontier models — covering cost math, data governance, performance gaps, and operational burden as of mid-2026.
LLM Observability: Monitoring Quality, Cost, and Drift in Production
Traditional APM tells you the HTTP call succeeded. LLM observability tells you whether the answer was any good — covering traces, quality scores, cost anomalies, and drift.
Streaming, Batching, and Throughput: Latency vs Cost Trade-offs in Production
How streaming, continuous batching, and async batch APIs interact with latency and cost — and how to configure each without destroying p99 for interactive users.
Quantization Demystified: GPTQ, AWQ, GGUF, and FP8 With Real Tradeoffs
A practical comparison of GPTQ, AWQ, GGUF, and FP8 quantization formats — memory math, quality loss, hardware targets, and when each format actually wins in production.
Cost, latency, and the model-selection decision tree for reasoning
A practical framework for choosing between reasoning and standard models, with real cost and latency numbers from 2025-2026 deployments.
Prompt Caching: Cutting Costs 50-90% Without Changing Model Behavior
How Anthropic and OpenAI prompt caching works, what breaks the cache, and how to structure prompts to hit 80%+ cache rates in production.
Document AI and OCR in 2026: When to Use VLMs vs Classic Engines
VLMs now beat classical OCR on noisy documents, but the right choice depends on layout complexity, volume, and cost. A decision framework with real benchmarks.
Model routing and cascades: sending the right query to the right model
How to build rule-based routers, intent classifiers, and confidence-based cascades that cut your LLM API bill 60–85% while preserving 95% of frontier quality.
Continuous Batching and Scheduling: Why Static Batching Wastes 70% of Your GPU
How iteration-level scheduling replaced static batching and became the single biggest serving optimization available without any model changes — with real numbers.
Test-Time Compute Scaling: Sequential, Parallel, and Tree-Based Strategies
How reasoning models spend more compute at inference time via longer chains, Best-of-N sampling, and tree search — and the cost, latency, and accuracy tradeoffs of each.
Choosing an embedding model in 2026: OpenAI, Cohere, Gemini, Voyage, and open-source
A decision framework for picking an embedding model: MTEB scores, latency, cost, multimodal needs, and when to self-host — with 2026 benchmarks.
Calling Vision APIs in Production: GPT-4o, Claude, and Gemini Compared
Calling vision APIs in production: image tokens and cost math, resolution tradeoffs, OCR accuracy, and how GPT-4o, Claude, and Gemini actually compare.
The AI Gateway Pattern
How a thin proxy between your app and every LLM provider gives you routing, fallback, cost tracking, and semantic caching without touching application code.
Prompt caching deep dive: prefix caching, semantic caching, and when each wins
How provider-side prefix caching and application-side semantic caching work, what they cost, where each breaks, and the decision framework for combining them in production.
The KV Cache: GPU Memory Math Every Serving Engineer Must Know
A concrete formula for how much GPU memory your KV cache consumes, why it scales brutally with context length, and the PagedAttention trick that fixes it.
Prompting reasoning models: what changes and what breaks
The counter-intuitive rules for prompting o3, DeepSeek-R1, and Claude extended thinking — simpler prompts often outperform elaborate ones, and common techniques backfire.
Token Economics: Where Your API Bill Actually Comes From
Dissect the full LLM API cost structure: input vs output token pricing, context bloat in multi-turn sessions, and the multiplier effect of long system prompts.
The Decision Ladder: Fine-Tuning vs RAG vs Prompting
A concrete decision framework for when prompting, RAG, or fine-tuning pays off — grounded in real cost, latency, and data trade-offs, not vendor marketing.
Tokens, Costs, and the Context Window
Why text becomes tokens, how input/output pricing asymmetry drives your bill, what the context window actually limits, and how prompt caching cuts costs by up to 90%.
The Model Landscape: Choosing Without Getting Played
A decision framework for picking between closed frontier models, open-weight, and reasoning tiers — plus benchmark literacy so leaderboard scores stop misleading you.
Context Engineering: Filling the Window Right
The shift from prompt to context engineering: how to write, select, compress, and isolate what goes in the window — and the silent failure modes that ship to production.
Reasoning Models: When and How to Use Them
Reasoning models spend inference tokens thinking before they answer. Learn when that's worth it, when it's a waste, how prompting changes, and how to control cost.
Cost Control and Reliability Engineering
Cut your LLM API bill ~4x and eliminate 429 cascades with prompt caching, model routing, semantic caching, and circuit breakers. Worked math from $60k to $14k/month.