Embeddings and Vector Databases That Scale

You've tuned chunk size, adjusted your retrieval prompt, and tested on a hundred examples. The system looks good. Then a colleague runs a query with a product SKU and the top result is a shipping FAQ from 2022. The embedding distance scores are all above 0.85. Everything looks healthy. The results are just wrong.

This is an embeddings problem. Not a prompt problem, not a chunking problem, not a model problem. The retrieval layer is the foundation everything else sits on. Get it wrong and no amount of prompt engineering fixes it downstream.

This module covers what embeddings actually encode, how the indexes that search them work, which database to put them in at your scale, and — most importantly — where the whole thing breaks silently in production.

Meaning as geometry

An embedding is a dense numerical vector that represents a piece of text (or an image, or code, or audio). The vector has hundreds or thousands of dimensions. The trick is that the model is trained so that semantically similar inputs produce numerically similar vectors — meaning that the spatial distance between two points in that high-dimensional space approximates the conceptual distance between two ideas.

This isn't metaphor. "Paris" and "Berlin" are closer to each other in embedding space than either is to "photosynthesis." "The dog bit the man" and "A man was bitten by a dog" map to nearly identical vectors, even though they share almost no tokens after tokenization.

flowchart LR
    text["raw text"] --> model["embedding model"]
    model --> vec["[0.12, -0.87, 0.33, ... 1536 dims]"]
    vec --> index["vector index"]
    query["query vector"] --> index
    index --> results["k nearest neighbors"]
    style model fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style index fill:#a855f7,stroke:#a855f7,color:#fff

Two immediate consequences. First, you can find semantically similar content even when the words don't overlap — which is the whole point of semantic search. Second, you have to be deliberate about what the model was trained to embed. A general-purpose English model will produce garbage representations of legal Latin phrases, domain-specific abbreviations, or low-resource languages it rarely encountered during training. MTEB aggregate scores tell you how good a model is on a benchmark suite; they don't tell you how it performs on your specific data distribution. Always evaluate on your actual data before committing.

Similarity metrics and the normalization gotcha

Once you have vectors, you need a distance function. Three dominate:

Cosine similarity measures the angle between two vectors, ignoring their magnitudes. Two vectors pointing in the same direction score 1.0 even if one is ten times longer.

Dot product measures both angle and magnitude — it's |a| × |b| × cos(θ). For two normalized vectors (magnitude = 1.0), this reduces to exactly the cosine formula. For normalized vectors, dot product and cosine produce identical rankings, and dot product is cheaper to compute because it skips the magnitude division.

L2 (Euclidean) distance measures straight-line distance in the vector space, which combines both angle and magnitude differences.

The production pitfall: OpenAI's text-embedding-3 models output normalized vectors, so you want dot product similarity — and if you configure your Pinecone index or pgvector column with cosine instead, you get correct rankings but waste CPU on redundant division. More dangerously, if you store raw embeddings without normalizing them (say, from a model that doesn't normalize by default), then switch to cosine, you might get semantically wrong results with no error raised and no obvious signal in the logs. The metric is silent about magnitude.

The rule: check your model's documentation to see if it outputs normalized vectors. If yes, use dot product. If no, decide whether magnitude carries signal for your use case — in recommendation systems it sometimes does. When in doubt, normalize and use dot product.

Choosing an embedding model

The 2026 MTEB leaderboard (as of mid-2026) has Qwen3-Embedding-8B at 70.6, topping all API-based models. But MTEB aggregate score is a blunt instrument; two decisions matter more for most teams:

API vs self-hosted. API models (OpenAI text-embedding-3-large at 3072 dims, Google Gemini Embedding at $0.008/1M tokens, Cohere Embed v4) give you zero ops overhead and are the right default under about 50M tokens/month. Beyond that threshold — or if you have data residency requirements — self-hosting an open model like text-embedding-qwen3-8b on a GPU instance becomes cheaper.

Dimensions and Matryoshka. More dimensions generally means better recall but more storage and slower queries. Matryoshka Representation Learning (MRL) is a training technique where the first N dimensions encode the most important signal, so you can legally truncate to a smaller size with predictable degradation: text-embedding-3-large at 256 dims loses roughly 2-3% precision versus its full 3072 dims, with 12x storage reduction. This enables a two-stage pattern: retrieve 200 candidates with 256-dim vectors, re-rank with full vectors.

For most RAG pipelines starting out in 2026: text-embedding-3-small (1536 dims, $0.02/1M tokens) is a fine default. Upgrade to text-embedding-3-large with 256-dim MRL retrieval if you're hitting recall problems on marginal queries, or switch to Gemini Embedding if your bill is getting large.

One warning that bites teams regularly: embedding models version. OpenAI deprecated ada-002 and its replacement text-3-small produces a different vector space. If you upgrade the model without re-embedding your corpus, your stored vectors and your query vectors are now in different spaces. The result is a coherent-looking system that returns subtly wrong results. Build model versioning into your ingestion pipeline from the start — it's much cheaper than debugging mystery recall degradation six months later.

ANN indexes: HNSW in depth

Exact nearest-neighbor search in 1536 dimensions over 10M vectors is O(N × D) per query — at N=10M and D=1536 that's 15 billion multiply-adds per lookup. Too slow. Approximate nearest neighbor (ANN) indexes trade a controlled amount of recall for orders-of-magnitude speedup.

HNSW (Hierarchical Navigable Small World) is the dominant algorithm in most vector databases. The intuition: build a layered graph where each vector is a node connected to its approximate neighbors. The top layer is sparse (few nodes, long-range connections), the bottom layer is dense (all nodes, short-range connections). A query starts at the top layer, greedily hops toward the nearest neighbor, then drops to the next layer and refines. Think of it like navigating with progressively more detailed maps.

{ "type": "hnsw", "mode": "build", "title": "How HNSW builds and searches" }

The HNSW construction parameters:

• M — max connections per node (typical: 16–64). Higher M improves recall but increases graph size.
• ef_construction — candidates explored during build (typical: 100–200). Higher improves graph quality but slows ingestion.
• ef_search — candidates explored per query. The main recall vs latency dial at query time.

The memory math is where HNSW gets expensive. Raw vectors at 10M × 1536 × 4 bytes = ~58 GB. The graph itself is cheap by comparison: neighbor-ID lists cost about M × 2 × 4 bytes per node on layer 0 plus small upper-layer link lists — 130–600 bytes at M=16–64, single-digit percent of a 6 KB vector. What actually inflates the bill is everything around it: allocator overhead, mutable-segment copies during ingestion, query-time buffers, and serving-runtime headroom. Plan for a total in-memory footprint of 1.5–2× the raw vector size — ~90–120 GB in RAM. That surprises teams who budgeted only for "the vectors," because a serving process needs far more than the bytes it stores.

The alternative at scale is IVF (Inverted File Index): cluster vectors into K cells with k-means, then at query time probe only the nearest few cells (typically 5-10% of cells) to find approximate neighbors. IVF has a lower memory footprint than HNSW for the same corpus size and is more amenable to disk-based storage. It's slower for random inserts (you need to retrain the centroids periodically) but better suited to billion-scale corpora that change slowly. If you're at 100M+ vectors and can afford periodic offline reindexing, IVF with quantization is worth evaluating.

Which vector database

This is the question everyone wants a simple answer to. The honest answer is: start with Postgres.

flowchart TD
    start["How many vectors?"] --> small["< 50M vectors"]
    start --> medium["50M – 100M vectors"]
    start --> large["> 100M vectors"]
    small --> pg["pgvector + pgvectorscale\non your existing Postgres"]
    medium --> eval["Evaluate: query QPS, latency SLA, hybrid search needs"]
    eval --> pg2["Still pgvectorscale if <100 QPS"]
    eval --> dedicated["Dedicated store if >100 QPS or <10ms p99"]
    large --> dedicated2["Purpose-built: Qdrant, Weaviate, or Milvus"]
    style pg fill:#0e7490,color:#fff
    style dedicated fill:#a855f7,color:#fff
    style dedicated2 fill:#a855f7,color:#fff

pgvector + pgvectorscale on Postgres. The pgvectorscale extension from Timescale benchmarks at 471 QPS at 99% recall on 50M vectors — faster than vanilla Qdrant at the same recall point. You get ACID transactions, existing Postgres tooling, and 60-80% cheaper TCO than a managed vector service. The ceiling is real: beyond 100M vectors, Postgres starts struggling with memory pressure and you'll want a store built for this workload. But most products never get there, and the operational simplicity of staying in one database is substantial.

Qdrant is the best choice if you need a free tier, aggressive cost control, or strong pre-built hybrid search. Their v1.9+ named-vector support enables first-class hybrid indexing, and their learning-based filtered ANN query planner (2025) is one of the better solutions to the pre-filter/post-filter problem. Qdrant Cloud starts at $25/month.

Weaviate has the most mature hybrid search implementation — BlockMax WAND combined with native BM25 gives cleaner keyword matching than most alternatives. If your use case is document search where keyword precision matters (legal, compliance, support KB), Weaviate's hybrid search is worth the additional configuration complexity.

Pinecone is the easiest to operate — 7ms p99, no infrastructure to manage, solid SDK ecosystem. The cost is the cost: ~$700+/month at 100M vectors as of mid-2026, prices subject to change. It makes sense when your team's time costs more than the infrastructure bill. It makes less sense when you're at 5M vectors and a $350/month managed service is 3x what a Postgres instance would cost.

Milvus 2.5+ is the choice for billion-scale workloads or organizations that want self-hosted control at scale. Native Sparse-BM25 hybrid search, sharding built in, and an active community (44K GitHub stars). Zilliz Cloud provides a hosted option if you want Milvus semantics without the Kubernetes overhead.

A word on the newcomers: LanceDB (zero-copy columnar storage, no server process) and Turbopuffer (50K namespace support, cost-focused) are solving specific problems well. They're worth knowing about but not the default choice for a new project in 2026.

flowchart LR
    pgvector["pgvector\n(Postgres extension)"] -->|"lowest ops cost"| pgfit["< 50M vectors"]
    qdrant["Qdrant\n(managed or self-hosted)"] -->|"best filtered ANN"| qfit["50M–500M vectors"]
    weaviate["Weaviate\n(managed or self-hosted)"] -->|"best hybrid search"| wfit["50M–500M vectors"]
    pinecone["Pinecone\n(fully managed)"] -->|"best ops simplicity"| pfit["any scale"]
    milvus["Milvus\n(self-hosted / Zilliz)"] -->|"best at billion scale"| mfit["> 100M vectors"]

Metadata filtering: where production systems break

Every real-world vector search comes with a filter. "Find similar documents — but only from this tenant. Only from this date range. Only with this document type." This sounds like a SQL WHERE clause bolted onto ANN search, and it almost is. Except that it silently destroys recall in ways that take weeks to diagnose.

Two strategies, both with real costs:

Pre-filter: Apply the metadata predicate first, restrict the ANN search to matching vectors only. The problem: HNSW's graph was built assuming all nodes are reachable. If 95% of your nodes are filtered out, the remaining 5% are sparsely connected — the greedy graph traversal can't reach many of them. On high-selectivity predicates (you're looking for documents from one specific user in a corpus of 10M), recall drops below 50% with no warning. The index happily returns results; they're just not the actual nearest neighbors.

Post-filter: Run ANN search without the filter, retrieve k × oversampling_factor candidates, then apply the filter. Reliable recall, but you need to over-fetch by 10-20x on selective predicates to ensure enough results survive the filter — which increases query latency and cost proportionally.

Qdrant v1.9+ implements a learned filtered-ANN query planner that selects between pre-filter and post-filter dynamically based on the estimated selectivity of the predicate. For high-selectivity queries it uses a brute-force search over the filtered subset instead of graph traversal. This is the current best-in-class solution to the problem. pgvector doesn't have this — it does post-filter, which means you need to raise the over-fetch dial for the index you actually built: SET hnsw.ef_search for an HNSW index (the one in the code example below), or SET ivfflat.probes for IVFFlat, high enough that enough results survive the filter. pgvector 0.8+ also added iterative index scans, which keep scanning until the filtered LIMIT is satisfied — built precisely for this starvation problem.

The practical rule: if your application has any metadata filter that could be highly selective (user-specific, time-constrained, rare category), test recall explicitly at p95 filter selectivity before you ship. Set up a test that runs a query where only 1% of your vectors match the filter and checks that recall@10 stays above 90%.

{ "type": "hybrid-search", "alpha": 0.7, "title": "Hybrid search: dense + BM25 fused via RRF" }

Hybrid search: when pure vector retrieval isn't enough

Vector search is excellent for semantic similarity. It's bad for rare proper nouns, product SKUs, version numbers, and any term the embedding model didn't see often enough to embed distinctively. If a user searches "CVE-2024-0001," the embedding for that string is essentially random — there's no semantic cluster to anchor to. Pure vector search returns whatever happens to be nearest, which is probably wrong.

Hybrid search runs a dense vector retrieval and a sparse keyword retrieval (BM25) in parallel, then fuses the ranked lists. Reciprocal Rank Fusion (RRF) is the standard algorithm:

score(doc) = Σ 1 / (k + rank_in_list_i)  for each ranked list i

where k is typically 60. The key insight: RRF works on ranks, not raw scores. This sidesteps the problem of incompatible score scales — you can't sum a cosine similarity of 0.83 and a BM25 score of 12.4 directly because they mean different things on different scales. Rank is rank.

Dynamic weighted RRF goes further: set the weight alpha per query based on whether the query looks like a keyword lookup (low alpha, lean sparse) or a semantic query (high alpha, lean dense). This approach shows +2-7.5 percentage point gains in recall@10 over a fixed weight, according to evaluations from Weaviate and Qdrant engineering teams.

The implementation failure mode: adding hybrid search by concatenating the two ranked lists and sorting by average score. Don't do this. Use RRF, or use your vector database's built-in hybrid search implementation (Weaviate's BlockMax WAND + RSF, Milvus's native Sparse-BM25, Qdrant's named-vector hybrid) rather than rolling your own fusion logic.

Cost and scaling math for 10M vectors

Back-of-the-envelope that every team should run before picking infrastructure:

Corpus:      10M vectors × 1536 dims × 4 bytes = ~58 GB raw storage
In-memory:   ~1.5-2x total (graph links are ~2-5%; the rest is
             allocator/segment/serving overhead) → ~90-120 GB RAM
Managed:     ~$300-400/month (Pinecone, illustrative mid-2026)
Self-hosted: 128 GB RAM instance on AWS ~$80-120/month compute
             + ~$15-30/month storage (EBS)
             Total: ~$100-150/month, but you own the ops

At 100M vectors (10x scale):
Managed:     $700-1,000+/month
Self-hosted: 1+ TB RAM instance or sharded → $400-600/month
Migration inflection: usually somewhere between $300-500/month managed bill

The other cost that surprises teams: query costs dominate in production at 10-50x the storage cost. A million embedding lookups costs more than storing the vectors indefinitely. Track your QPS before committing to a pricing tier.

Quantization is the lever for storage cost:

• INT8 scalar quantization: 4x compression (58 GB → 14 GB), roughly 0.5% recall loss with re-ranking. Worth doing at almost any scale.
• Binary quantization with float32 re-ranking: 32x compression, fetches binary candidates then re-ranks with full vectors. Weaviate and Qdrant support this natively. The 2025 standard pattern for memory-constrained deployments.

At 10M vectors with INT8 quantization, run the arithmetic in components: quantized vectors in RAM ~15 GB, graph links ~1–3 GB, allocator and serving overhead another ~5–10 GB — call it 25–30 GB resident. The float32 originals still have to exist for the re-ranking step, but the standard pattern is to keep them memory-mapped on disk (~58 GB of SSD), not in RAM. Total: comfortably inside a 64 GB machine with room for the OS and query spikes.

Putting it together in code

A minimal working example of the full pipeline — embed, index, search — using the OpenAI API and Qdrant:

from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
import uuid

openai = OpenAI()
qdrant = QdrantClient(url="http://localhost:6333")

# 1. Create collection with dot product (normalized vectors)
qdrant.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1536, distance=Distance.DOT),
)

# 2. Embed and index a batch of documents
def embed_batch(texts: list[str]) -> list[list[float]]:
    response = openai.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return [r.embedding for r in response.data]

docs = [
    {"id": "doc-1", "text": "Refund policy: items must be returned within 30 days.", "category": "policy"},
    {"id": "doc-2", "text": "Shipping takes 3-5 business days.", "category": "policy"},
    # ... more documents
]

embeddings = embed_batch([d["text"] for d in docs])
points = [
    PointStruct(
        id=str(uuid.uuid4()),
        vector=emb,
        payload={"doc_id": doc["id"], "category": doc["category"]},
    )
    for doc, emb in zip(docs, embeddings)
]
qdrant.upsert(collection_name="docs", points=points)

# 3. Query with metadata filter — use post-filter to preserve recall
query_embedding = embed_batch(["how do I return a product"])[0]
results = qdrant.query_points(
    collection_name="docs",
    query=query_embedding,
    query_filter={"must": [{"key": "category", "match": {"value": "policy"}}]},
    limit=5,
)

for r in results.points:
    print(r.payload["doc_id"], r.score)

Two things to notice: Distance.DOT not Distance.COSINE (because text-embedding-3-small outputs normalized vectors), and the metadata filter is passed to query_points — Qdrant's query planner decides between pre-filter and post-filter based on estimated selectivity.

For pgvector on Postgres, the equivalent query looks like:

-- After: CREATE EXTENSION vector;
--        ALTER TABLE docs ADD COLUMN embedding vector(1536);
--        CREATE INDEX ON docs USING hnsw (embedding vector_ip_ops);  -- ip = inner product = dot product

SELECT id, category, embedding <#> $1 AS score  -- <#> is negative inner product
FROM docs
WHERE category = 'policy'
ORDER BY embedding <#> $1
LIMIT 5;

The <#> operator is pgvector's inner product (negate for similarity ordering). For selective WHERE clauses, pgvector falls back to a sequential scan over the filtered subset — same recall, higher latency than a purpose-built filtered ANN.

What breaks in production

The failure modes to know before you ship:

The normalization trap hits when you forget to check whether your model normalizes output. Configure the wrong metric, get confidently wrong rankings.

The metadata filter recall cliff hits when you add a filter that's highly selective. You'll see it in product metrics (users complaining search doesn't work) weeks before anyone looks at the ANN layer. Test recall under selective predicates before launch.

HNSW memory surprise: teams calculate storage cost from raw vector size and forget everything else that lives in RAM alongside them — segment copies, allocator overhead, query buffers. Budget total memory at 1.5–2× the raw vector footprint — a 64 GB instance for 58 GB of raw vectors is not enough.

Embedding model upgrade invalidation is the silent killer. When you rotate to a newer embedding model without re-embedding, stored vectors and query vectors are now in different spaces. The system looks healthy; it just returns wrong results. Always include model_version in your vector payload metadata.

The BM25 score scale mismatch hits when you implement hybrid search by summing raw scores. The BM25 distribution has a completely different scale than cosine similarity. Use RRF.

The chunk quality bottleneck is the quieter one: teams spend weeks optimizing their ANN index and hybrid fusion while leaving 2,000-token chunks with 80% noise content in the corpus. Chunk quality and relevance dominate retrieval quality far more than index choice. Fix your ingestion before your index.

Where to go next

What Are Embeddings: Geometry, Meaning, and the Math — the full treatment of what high-dimensional vector spaces actually encode, with the limits of linearity (negation, temporal ordering) that crash-course readers should know.

Similarity Metrics Demystified: Cosine, Dot Product, and L2 — goes deeper on when magnitude carries signal and exactly how to audit your normalization pipeline.

ANN Indexes Under the Hood: HNSW, IVF, and When Each Wins — complexity analysis, the IVF training phase, and Product Quantization as a third option for extreme compression.

Vector Database Landscape 2026 — full comparison with multi-tenancy patterns, price benchmarks at 10M/100M/1B vectors, and the LanceDB/Turbopuffer newcomers.

Hybrid Search: Fusing Dense Vectors and Sparse BM25 Correctly — SPLADE, learned sparse models, and the implementation differences between Weaviate's BlockMax WAND and Milvus's Sparse-BM25.

Metadata Filtering, Scaling, and Production Cost Control — the ACORN algorithm, multi-tenancy patterns, and the 60-80M queries/month economic tipping point for self-hosting.

RAG in One Pass: Build, Measure, Improve — if you haven't built your first retrieval pipeline yet, this module shows how these pieces fit together end-to-end.