Similarity Metrics Demystified: Cosine, Dot Product, and L2

Your new RAG pipeline passes the demo. Recall looks fine on the 50-document test set. Then you go to production with 2 million documents and start getting support tickets: the search finds the wrong things when users ask short, specific questions. The embeddings are fine — you verified that with nearest-neighbor spot-checks on toy examples. The model is fine. The problem is that your vector database is configured to use dot product, your embedding model outputs vectors with magnitudes ranging from 0.7 to 2.3 depending on input length, and dot product on unnormalized vectors rewards both "semantically close" and "high norm". Longer documents have higher norms. Your search is partially a length ranker.

No error was raised. No warning appeared. The metric mismatch shipped silently.

This is the central practical lesson of similarity metrics in production: the three common options (cosine, dot product, L2) produce identical rankings only in specific conditions, and outside those conditions the differences are real, measurable, and quiet.

What the three metrics actually compute

Start with two vectors, q (query embedding) and d (document embedding), each of dimension n.

Dot product (q · d) is:

q · d = Σᵢ (qᵢ × dᵢ)

High when the vectors point in a similar direction and when they have large magnitudes. It cares about both angle and scale.

Cosine similarity normalizes the dot product by the product of the magnitudes:

cos(q, d) = (q · d) / (||q|| × ||d||)

This strips out magnitude — the result is always between -1 and 1, measuring only the angle between the two vectors. Two vectors pointing in exactly the same direction have cosine 1.0, regardless of whether one is 10× longer than the other.

L2 (Euclidean) distance is the straight-line distance between the vector tips:

L2(q, d) = sqrt(Σᵢ (qᵢ - dᵢ)²)

Lower is more similar (unlike cosine and dot product, where higher is better). L2 is sensitive to both the angle and the difference in magnitudes between the vectors.

flowchart TD
    subgraph "2D example"
        A["q = [1, 0]\nUnit vector, pointing right"]
        B["d₁ = [2, 0]\nSame direction, 2× magnitude"]
        C["d₂ = [0.7, 0.7]\nDifferent direction, similar magnitude"]
    end

    A --> M1["cos(q, d₁) = 1.0\nDot(q, d₁) = 2.0\nL2(q, d₁) = 1.0"]
    A --> M2["cos(q, d₂) = 0.707\nDot(q, d₂) = 0.7\nL2(q, d₂) = 0.755"]

In this 2D example, cosine says d₁ is a perfect match (angle = 0). Dot product also ranks d₁ higher, but the score is 2.0 instead of 1.0 — it rewards the larger magnitude. L2 would rank d₂ as more similar (0.755 < 1.0), even though d₁ points in exactly the same direction. The "right" answer depends entirely on what question you want to ask.

When cosine and dot product are identical

The equivalence is algebraic. If ||q|| = ||d|| = 1 (both vectors have unit length), then:

cos(q, d) = (q · d) / (1 × 1) = q · d

They are the same number. More importantly, they produce the same ranking across all documents, which is all retrieval cares about. You don't need cosine similarity to produce a value of 1.0; you need it to put the most semantically similar document at rank 1 and the least similar at rank N. If your vectors are pre-normalized, dot product does this correctly and skips the two divisions in the denominator.

This is why the production pattern is: normalize at index time, configure dot product at query time. The normalization step is a one-time cost at ingestion, and every query is slightly faster because you skipped a sqrt and two divisions.

Most major embedding APIs handle this for you. OpenAI's text-embedding-3 models return normalized vectors. Cohere Embed v4 also normalizes outputs. Voyage AI's v3 models document cosine as the intended metric but their implementation also normalizes. When in doubt, check: import numpy as np; print(np.linalg.norm(embedding)) — if it prints 1.0, you're already normalized.

import numpy as np
from openai import OpenAI

client = OpenAI()

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-large"
    )
    return response.data[0].embedding

query = get_embedding("semantic search tutorial")
doc   = get_embedding("how to build a vector search engine")

q = np.array(query)
d = np.array(doc)

# Verify OpenAI normalizes for you
print(f"||q|| = {np.linalg.norm(q):.6f}")  # → 1.000000
print(f"||d|| = {np.linalg.norm(d):.6f}")  # → 1.000000

# On normalized vectors, these produce identical results
cosine   = np.dot(q, d) / (np.linalg.norm(q) * np.linalg.norm(d))
dot      = np.dot(q, d)
print(f"cosine: {cosine:.6f}, dot: {dot:.6f}")  # → same number

The output confirms they are identical. If your model outputs something other than 1.000000, you have two options: normalize the vectors yourself before indexing, or use cosine similarity in your vector database configuration (which normalizes internally at query time, at a small per-query cost).

When L2 is the right choice

L2 is the correct metric when vector magnitude encodes a signal you want to use in ranking.

The clearest example is in recommendation systems trained with a specific objective: some two-tower models are trained so that an item that has been purchased or clicked many times gets a higher-norm embedding, while obscure items have lower norms. In this setup, the dot product score naturally up-ranks popular items because the norm boost is intentional — it encodes a form of confidence or prior. Normalizing away that magnitude throws away signal that took training to build in.

L2 also appears in image embedding models where the norm captures something about image complexity or feature richness. And L2 is the default metric in FAISS, which makes it the implicit choice for any system that drops FAISS in without configuration — a frequent source of metric mismatches.

For most text embedding models used in RAG, L2 is wrong. Text models are trained with a contrastive objective that says "make semantically similar texts point in similar directions" — the training does not assign meaning to the magnitude. A 512-token document produces a higher-norm embedding than a 3-token title not because it's semantically richer but simply because it has more content. L2 on these embeddings penalizes length differences for no semantic reason.

Worked example — why L2 misbehaves on text embeddings

Query: "refund policy" (unnormalized model output, ||q|| = 0.9)
Doc A: "Payments are processed within 5 business days."
       (short, tangential: cos = 0.80, norm = 0.9)
Doc B: "Our comprehensive refund and return policy covers all purchases made
        through the platform, including digital goods and subscription plans,
        for a period of 30 days from date of purchase."
       (long, on-point: cos = 0.85, norm = 1.5)

L2² = ||q||² + ||d||² − 2·||q||·||d||·cos
L2(q, A) = sqrt(0.81 + 0.81 − 2·0.9·0.9·0.80) = 0.57
L2(q, B) = sqrt(0.81 + 2.25 − 2·0.9·1.5·0.85) = 0.87

Cosine ranking: B > A  (correct — B actually answers the question)
L2 ranking:     A > B  (wrong — B is pushed away purely because its
                        length-inflated norm doesn't match the query's)

L2 punished the better document for a norm difference that carries no
semantic signal. Dot product errs in the opposite direction — it hands
high-norm documents a bonus (that's the length bias from the opening
story). Cosine is the only one of the three that scores angle alone.

The normalization trap in production

The production failure mode is not "I chose the wrong metric." It's "I chose cosine but my database computed dot product on unnormalized vectors." These three configurations are all different:

1. Store unnormalized, compute cosine → DB normalizes at query time (slower, but correct)
2. Store normalized, compute dot product → equivalent to cosine, faster
3. Store unnormalized, compute dot product → neither cosine nor L2, a blend of both

Configuration 3 is the bug. It happens because dot product is often the default for performance reasons and developers don't notice that their embedding model doesn't normalize. The symptom is recall that degrades on document sets with variable text length.

The fix is a two-step verification:

import numpy as np

def verify_normalization(embeddings: list[list[float]], sample: int = 100) -> None:
    """Check if vectors are unit-normalized. Run this before indexing."""
    sample_vecs = np.array(embeddings[:sample])
    norms = np.linalg.norm(sample_vecs, axis=1)
    
    print(f"Norm range: [{norms.min():.4f}, {norms.max():.4f}]")
    print(f"Mean norm:  {norms.mean():.4f}")
    print(f"Std dev:    {norms.std():.4f}")
    
    if norms.std() > 0.01:
        print("WARNING: vectors are not normalized — use cosine metric or normalize first")
    else:
        print("OK: vectors appear normalized — dot product equivalent to cosine")

Run this once during development. If std > 0.01, you either normalize before indexing or explicitly configure your vector database to use cosine similarity (which normalizes on read).

In Qdrant:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient(url="http://localhost:6333")

# If vectors are pre-normalized: Dot is faster
client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(size=1536, distance=Distance.DOT)
)

# If vectors are NOT pre-normalized: use Cosine (Qdrant normalizes internally)
client.create_collection(
    collection_name="documents_raw",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)

In pgvector:

-- cosine distance operator: <=>
SELECT id, content, embedding <=> '[0.1, 0.2, ...]'::vector AS distance
FROM documents
ORDER BY distance
LIMIT 10;

-- inner product operator: <#> (multiply by -1 since it returns negative)
-- Only correct if vectors are pre-normalized
SELECT id, content, (embedding <#> '[0.1, 0.2, ...]'::vector) * -1 AS similarity
FROM documents
ORDER BY similarity DESC
LIMIT 10;

To see where this configuration actually takes effect, watch the ANN-search stage in the pipeline below — that is the moment your configured distance function either matches how the model was trained or silently doesn't.

{ "type": "rag-pipeline", "title": "Where similarity metrics live in the retrieval pipeline", "query": "policy" }

Matryoshka embeddings: an additional wrinkle

Matryoshka Representation Learning (MRL) produces embeddings where the first k dimensions form a valid sub-embedding, letting you truncate from 3072 dimensions to 256 with only 2-3% precision loss and a 12× storage reduction. OpenAI's text-embedding-3-large and Cohere Embed v4 both support this.

The wrinkle: truncated MRL sub-vectors are not unit-normalized, even if the full vector was. When you take the first 256 dimensions of a 3072-dim unit vector, the sub-vector has magnitude less than 1 and varies across documents.

Using dot product on truncated MRL embeddings is therefore incorrect. Cosine similarity normalizes the sub-vectors at comparison time and works correctly. Using L2 compounds the problem because the norm of a 256-dim sub-vector varies with document length in the same way as with full unnormalized text embeddings.

flowchart LR
    M["Full embedding\n3072 dims\n||v|| = 1.0"] --> T1["Truncated 256 dims\n||v|| ≈ 0.22 (varies)"]
    M --> T2["Truncated 512 dims\n||v|| ≈ 0.41 (varies)"]
    M --> T3["Truncated 1024 dims\n||v|| ≈ 0.68 (varies)"]
    T1 --> N["Re-normalize at\ncomparison time\n→ use cosine"]
    T2 --> N
    T3 --> N
    style N fill:#15803d,color:#fff

The pattern is to re-normalize the truncated sub-vectors before indexing, or to configure cosine distance in the vector database. At 10M documents × 256 dims, re-normalizing at ingest costs about 10M × 256 = 2.56B FLOPs, which a single machine does in well under a second. The storage is 10M × 256 × 4 bytes ≈ 10 GB — down from roughly 123 GB for the full 3072-dim float32 vectors.

See Choosing an Embedding Model in 2026 for a full comparison of MRL models and the precision/cost tradeoffs at each truncation level.

Matching the metric to the model

Different embedding models were trained with different objectives. The metric should match the training:

Model	Recommended metric	Notes
OpenAI text-embedding-3-large	Cosine or normalized dot product	Outputs are unit-normalized
OpenAI text-embedding-3-small	Cosine or normalized dot product	Outputs are unit-normalized
OpenAI text-embedding-ada-002	Cosine	Deprecated but still in production; also normalizes
Cohere Embed v4	Cosine	MRL support at 256/512/1024/1536 dims; normalize sub-vectors
Voyage AI (v3 series)	Cosine	Explicitly documented
Qwen3-Embedding-8B	Cosine	MTEB leader as of 2026; documentation recommends cosine
BGE / E5 models (HuggingFace)	Cosine	Standard open-weight pattern; normalize before FAISS
CLIP image embeddings	Cosine or dot product (pre-normalized)	OpenAI's CLIP normalizes image and text embeddings
Two-tower recommendation models	Dot product (unnormalized)	Magnitude often encodes confidence; check model docs

The place where this bites teams most is FAISS. FAISS defaults to L2 distance. If you build a FAISS index with faiss.IndexFlatL2 on unnormalized text embeddings from any of the models above, you are computing the wrong metric. The fix:

import faiss
import numpy as np

# Wrong: FAISS L2 on unnormalized text embeddings
index_l2 = faiss.IndexFlatL2(1536)  # default, wrong for text

# Right option 1: normalize first, then use inner product (= cosine on unit vectors)
vectors = np.array(embeddings, dtype=np.float32)
faiss.normalize_L2(vectors)  # in-place normalization
index_ip = faiss.IndexFlatIP(1536)  # inner product on normalized = cosine
index_ip.add(vectors)

# Right option 2: use FAISS cosine similarity directly
# (FAISS doesn't have a native cosine metric — normalize + IP is the standard pattern)

The faiss.normalize_L2() function is in-place and efficient. At 1M vectors × 1536 dims, it takes roughly 2-4 seconds on a single CPU — a one-time cost at index build.

What breaks

The failure modes cluster into three patterns.

Silent recall degradation. This is the most common. You configure the wrong metric, but the benchmark set doesn't expose it because your test queries are well-represented in the dataset and the top-1 result is usually correct regardless of metric. You ship to production, the real query distribution hits edge cases, and precision@5 drops from 78% to 63% without any system error. The fix is to benchmark with a held-out evaluation set that includes short queries against documents of varying length.

Magnitude-based length bias. Unnormalized text embeddings from most transformer models correlate vector norm with text length — longer inputs fill more of the embedding space. Dot product on these embeddings rewards longer documents. This produces a search that biases toward verbose text even when a short, precise answer is a better match. In one common scenario with internal documentation search, FAQ entries (1-3 sentences) consistently rank below wiki pages (10+ paragraphs) for questions the FAQ answers precisely — because the wiki pages have 1.5-2× higher embedding norms.

FAISS metric mismatch after an upgrade. A team builds their FAISS index with IndexFlatL2, it's good enough at 100K documents, they grow to 5M documents, someone upgrades to an HNSW index, and during the migration accidentally changes the metric to IndexHNSWFlat (which also defaults to L2). The recall appears unchanged because L2 was already wrong before the migration. Nobody notices until a much later audit. The lesson: document your metric choice in code, not just in configuration files, and add an assertion at index load time.

Hybrid search score incompatibility. When combining cosine similarity scores with BM25 scores in hybrid search, developers sometimes try to add the two scores directly. This fails because BM25 returns values in the range 0-20+ depending on term frequency and document length, while cosine returns -1 to 1. The result is dominated by BM25. The standard fix is Reciprocal Rank Fusion (RRF), which works on ranks rather than raw scores and sidesteps the incompatible scale problem entirely.

The decision in practice

The actual decision tree is short:

1. Check your model's documentation. Does it say cosine? Then use cosine (or normalize + dot product for speed). Does it say inner product or dot product? Use that. Does it say L2? That's unusual for text but follow the model docs.

2. Verify the output norms. Run verify_normalization() on a sample of 100 vectors from your model. If std < 0.01, vectors are normalized and dot product is equivalent to cosine. If not, either normalize at ingestion or set the database metric to cosine.

3. Configure your database explicitly. Don't rely on defaults. In Qdrant, set Distance.COSINE or Distance.DOT consciously. In FAISS, use IndexFlatIP with pre-normalized vectors. In pgvector, use the <=> operator for cosine. In Pinecone, the metric is set at index creation time — match it to your model.

4. For Matryoshka truncation, always use cosine. The sub-vectors are not unit-normalized even if the full vectors were. Cosine handles this correctly; dot product does not.

5. For recommendation models with explicit magnitude semantics, use dot product on unnormalized vectors. This is the exception, not the rule for text. Check model docs carefully.

The deeper point is that similarity metric choice is not a hyperparameter you tune by experimenting with different options and picking the one with the highest dev-set score. The correct metric is determined by how the model was trained, and using a different one is a category error rather than a suboptimal choice. Get this right once per model, document it, and the question rarely comes up again.

The next piece of work — choosing which embedding model to use in the first place — is covered in Choosing an Embedding Model in 2026. Once you have your embeddings and metric sorted, ANN Indexes Under the Hood covers how HNSW and IVF trade recall for speed at different scales, and Vector Database Landscape 2026 compares where Qdrant, Weaviate, Milvus, and pgvector actually differ in production. The end-to-end picture of how all of this fits into a retrieval system is in the RAG in One Pass crash course module.