Metadata Filtering, Scaling, and Production Cost Control

You shipped your RAG demo last quarter, and the retrieval worked great. Then someone asked "can we scope results to the current user's documents?" You added a user_id filter, tested it, and it looked fine. Three weeks later you get a Slack message from support: users with fewer than 200 documents in the system are getting garbage results — plausible-looking chunks from their own corpus that have nothing to do with what they asked. Not because the filter is broken — it's filtering correctly. It's that the filter is so selective that the ANN traversal can no longer reach the right vectors, so recall collapses and the top-K fills up with whatever eligible matches the crippled graph walk stumbles into. No error, normal latency, wrong answers. The filter worked. The ANN index didn't.

This is the gap between vector search as a concept and vector search as an operational system. Metadata filtering, multi-tenancy isolation, quantization, and cost control are the layer that most tutorials skip — and where most production deployments have their first expensive surprise.

How metadata filtering breaks ANN indexes

To understand why this is hard, you need the mental model for how HNSW builds its graph. The graph is constructed over the full vector collection, connecting every point to its approximate nearest neighbors across multiple layers. The structure is global — a point near the top layer acts as a highway entry point to the dense bottom layer. See ANN Indexes Under the Hood for the full construction algorithm.

When you add a metadata filter — user_id = 42, date >= 2026-01-01, category IN ['finance', 'legal'] — you're asking the search engine to only return results from a subset of vectors. There are two ways to enforce this, and both have failure modes.

Pre-filtering applies the predicate before ANN traversal. Only eligible vectors participate in graph search. The problem: the HNSW graph was built without knowing your filter. Many of the highway links at the top layer point to ineligible vectors, and those ineligible vectors are the bridges to eligible ones deeper in the graph. With a filter that passes 50% of vectors, graph connectivity degrades gracefully. With a filter that passes 1% of vectors — a single tenant with 10,000 docs in a 1M-vector collection — you've severed so many graph edges that the traversal gets stuck in local minima. Recall silently falls to 40–60%. The query runs fast and returns K results, but they're the wrong ones.

Post-filtering runs the full unfiltered ANN search, retrieves the top-N candidates, then discards those that don't match the predicate. Recall is preserved — the graph traversal is uncorrupted. The problem: if the filter is selective enough, many of the top-N candidates get discarded, and you end up with fewer than K results. You need to over-fetch: ask for 5–10x the desired K, then prune. On a filter that passes 1% of vectors, you'd need to retrieve the top-5000 to reliably get 50 valid results, which is slow and wastes compute.

flowchart LR
    A["100% of vectors in index"] --> B["Filter: user_id=42\n→ 1% of vectors eligible"]
    B --> C1["Pre-filter path:\ngraph traversal skips 99%\n→ broken highway links\n→ recall ≈ 45%"]
    B --> C2["Post-filter path:\ntraverse full graph\nretrieve top-5000\ndiscard 4950\n→ recall ≈ 97%\nbut 100x more candidate work"]
    style C1 fill:#ff2e88,color:#fff
    style C2 fill:#15803d,color:#fff

Neither approach wins universally. The right choice depends on filter selectivity, and selectivity is query-dependent. A single-tenant filter in a 100-tenant system passes 1% of vectors. The same filter in a 5-tenant system passes 20%. Same code, wildly different behavior.

The modern fix: adaptive query planning

Qdrant v1.9+ ships a cardinality-estimating query planner: it consults the payload index to estimate how many vectors each predicate passes before committing to a traversal strategy. For filters expected to pass >5–10% of vectors, it uses filtered graph traversal (fast, graph connectivity acceptable — Qdrant's HNSW variant adds extra links per indexed payload field to keep the filtered graph connected). For highly selective filters — a few thousand eligible vectors out of millions — it skips the graph entirely and brute-force scores the eligible subset exactly. This sounds like a regression but is the correct answer: exhaustively scoring 10,000 float32 vectors takes single-digit milliseconds and delivers 100% recall, which no graph strategy can match at that cardinality. For borderline cases, it runs a fast filtered traversal and expands if the result set comes back sparse.

The query itself doesn't change — the planner takes over based on the estimated cardinality of the filter:

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")

hits = client.query_points(
    collection_name="docs",
    query=query_vector,  # 1536-dim float32
    query_filter=models.Filter(must=[
        models.FieldCondition(key="tenant_id", match=models.MatchValue(value="acme")),
        models.FieldCondition(key="created_at", range=models.DatetimeRange(gte="2026-01-01")),
    ]),
    limit=20,
)
# Permissive filter → filtered HNSW traversal.
# Tiny eligible subset → exact scan over just those vectors, 100% recall.
# Requires a payload index on tenant_id, or the planner is guessing.

The research direction is exemplified by ACORN (Patel et al., SIGMOD 2024), which attacks the problem at index-build time instead: construct a denser, predicate-agnostic HNSW graph, then apply the filter during traversal, so connectivity survives even selective predicates the index never saw coming. The practical gain from filter-aware search, whichever mechanism delivers it: on highly selective filters that previously dropped to 45% recall with static pre-filtering, production systems recover 15–25 percentage points of recall on the same hardware.

Weaviate uses a different approach: payload indexes (B-tree or inverted index on metadata fields) are maintained separately, and the query planner can intersect a payload bitmap with HNSW candidate sets before or after traversal based on cost estimates. Pinecone's internal planner is less documented but empirically shows similar adaptive behavior at scale.

If you're using a database that doesn't have adaptive filtering — some older deployments or self-hosted versions — there are workarounds. One is partition-by-filter-value: if user_id is a dominant filter, store each user's vectors in a separate HNSW segment. Then the "graph with broken links" problem disappears because each segment only contains eligible vectors. The cost: one segment per tenant, which balloons operational overhead at thousands of tenants. This brings us to multi-tenancy.

Multi-tenancy patterns and where they break

A multi-tenant vector search system has to answer: "which of my users' documents are most similar to this query?" while enforcing that User A never sees User B's data, even under failure conditions. There are three canonical patterns.

flowchart TD
    subgraph Pattern1["Pattern 1: collection per tenant"]
        T1C["Tenant A → Collection_A"]
        T2C["Tenant B → Collection_B"]
        T3C["Tenant C → Collection_C"]
    end
    subgraph Pattern2["Pattern 2: namespace isolation"]
        NS["Single collection"]
        NS --> NsA["namespace: tenant_a"]
        NS --> NsB["namespace: tenant_b"]
        NS --> NsC["namespace: tenant_c"]
    end
    subgraph Pattern3["Pattern 3: metadata field filter"]
        SC["Single collection\nall tenants"]
        SC --> MF["Filter: tenant_id = X\nat query time"]
    end
    style T1C fill:#0e7490,color:#fff
    style T2C fill:#0e7490,color:#fff
    style T3C fill:#0e7490,color:#fff
    style NS fill:#a855f7,color:#fff
    style SC fill:#00e5ff,color:#0a0a0f

One collection per tenant gives perfect isolation — each tenant's vectors form their own HNSW graph with no risk of cross-tenant contamination. You can tune index parameters per tenant (a large tenant with 10M docs might need different HNSW ef and m parameters than a small one with 500 docs). The problem is operational scaling: at 1,000 tenants you're managing 1,000 collections. At 10,000 you're in a world of pain. Qdrant imposes a soft limit of ~10k collections per instance before operational overhead becomes significant. Pinecone's documentation recommends against using indexes as tenant boundaries at scale. Schema migrations touch every collection separately.

Namespace isolation (Pinecone namespaces, Qdrant shard keys, Weaviate multi-tenancy mode since v1.20) shares the operational infrastructure — one collection, one schema — while routing queries to a tenant-specific segment. This scales to tens of thousands of tenants without per-tenant management overhead. The tradeoff: cross-tenant searches are technically possible if your application layer has a bug — you need query-level enforcement that every search includes the tenant filter. In Qdrant's shard-key model, the shard routing is enforced by the database, which is stronger than a filter. In Pinecone's namespace model, the namespace is a query parameter and can be omitted — application-layer enforcement is required.

Single collection with a metadata field is the simplest to implement and the cheapest operationally, but falls into the high-selectivity filtering problem described above. For large tenants (>5% of total vectors) it works well. For small tenants — common in SaaS with power-law customer distributions — a tiny tenant's metadata filter is highly selective and recall degrades silently.

The practical answer for most SaaS deployments: namespace isolation for tenants with sufficient data (>10,000 vectors), and a shared "small tenants" namespace with a metadata field filter for tenants below that threshold. The small tenants collectively form a large-enough group that their combined filter is not highly selective. This hybrid approach handles the power-law distribution without spinning up thousands of isolated namespaces.

Quantization: cutting memory without cutting recall

At 10M vectors with 1536-dimensional float32 embeddings — OpenAI text-embedding-3-large or Cohere Embed v4 output — you need 10M × 1536 × 4 bytes ≈ 61 GB of raw vector storage. HNSW adds another 50–100% for graph links. You're looking at 90–120 GB of RAM just for the index on a moderately sized collection. Managed services bill for that RAM directly.

Quantization compresses each vector component.

Method	Compression	Recall cost	Re-ranking required?
float32 (baseline)	1x	none	no
INT8 scalar	4x	1–2% recall loss	optional
float16	2x	<0.5% recall loss	no
Binary (1 bit/dim)	32x	10–20% standalone	yes
Binary + float32 re-rank	8–16x effective	<3% with rerank	built-in

INT8 quantization replaces each 32-bit float with an 8-bit integer using a per-vector or per-dimension scale factor. The recall loss is small because the relative ordering of dot products is largely preserved. 4× compression on raw vectors cuts ~61 GB to ~15 GB; with HNSW graph overhead, the full index drops from ~90–120 GB to roughly 30–40 GB — a material change in instance sizing.

Binary quantization is more aggressive: each component becomes a single bit (1 if the float32 value is positive, 0 if negative). 32x compression, but significant standalone recall loss because sign bits alone lose magnitude and fine-grained direction information. The recovery mechanism is re-ranking: use Hamming distance on binary vectors to retrieve 4× the desired K cheaply, then re-score those candidates with the original float32 vectors. This is fast because you've reduced the float32 scoring to a small candidate set.

The math for why re-ranking works: on 1536-dimensional embeddings from a well-trained model, the top-4K binary candidates contain the true top-K at 97–99% recall. Hamming distance on 1536-bit vectors is a bitwise XOR and popcount — about 40x faster than float32 dot products — so the initial retrieval is very cheap even at 4x over-fetch.

Binary + re-rank sizing at 10M vectors, 1536 dims:

  Binary index:   10M × 1536 / 8 bytes ≈ 1.9 GB (+ HNSW overhead ~3 GB) ≈ 5 GB total
  Re-rank cache:  if you keep full float32 for top candidates only, negligible
  Full float32 backup (for re-ranking):
    Option A: keep on disk, fetch on demand → ~61 GB disk, ~2ms added latency
    Option B: keep compressed float16 in RAM → ~30 GB, <1ms re-rank
    Option C: recompute from original embeddings → slow, usually wrong

  Practical choice: binary index in RAM (5 GB) + float16 backup in RAM (30 GB)
  Total: 35 GB vs 90–120 GB for unquantized HNSW
  → 3–4x memory reduction, <3% recall loss

  The headline "8–16x" figure assumes Option A: float32 originals on disk,
  only the binary index (~5 GB) plus working buffers in RAM. Keep a float16
  re-rank copy hot and it's 3–4x. Budget with the option you'll actually run.

Hugging Face's embedding quantization blog post benchmarks this pattern across multiple models and confirms the recall recovery holds for models above 1024 dimensions. For smaller embedding models (256–768 dims), binary quantization loses more information per bit and the re-ranking recovery is weaker — INT8 is a better choice below 1024 dims.

The vector database is only one line on the RAG bill. Every retrieved chunk becomes prompt tokens, and at production QPS the LLM token spend usually dwarfs the retrieval infrastructure — worth modeling side by side:

{ "type": "token-cost", "title": "The other bill: LLM token spend per RAG query" }

The cost model no one explains upfront

Almost every post about vector database pricing starts and ends with storage costs: $X per GB-month, $Y per million vectors. Storage is visible in the pricing calculator and easy to project. It's also almost never the dominant cost.

At sustained production traffic, query compute — not storage — is the line that grows. Here's a realistic anatomy for 10M vectors at 1,000 QPS:

Illustrative monthly cost estimate (2026 pricing, managed service):

Storage component:
  10M vectors × 1536 dims × float32 ≈ 61 GB raw
  With HNSW overhead: ~100 GB RAM required (1.5–2× raw)
  Pinecone p2-equivalent: ~$0.096/GB/hr × 100 GB × 720 hr ≈ $6,900/month
  (or ~$83K/year — probably why you're reading this article)

With INT8 quantization (4× reduction of raw vectors):
  ~33 GB RAM → ~$2,300/month in storage-equivalent

With binary quantization:
  ~5 GB binary + 30 GB float16 = 35 GB → ~$2,419/month

Query compute on top (1,000 QPS):
  Pinecone read units: ~$2–5/million queries at p2
  1,000 QPS × 60 × 60 × 24 × 30 = 2.6B queries/month → $5,200–13,000/month
  → Query cost alone exceeds storage cost by 2–8x

Total managed cost at 1,000 QPS: $7,700–19,000+/month

Self-hosted (Milvus/Qdrant on 3× r6g.2xlarge, AWS us-east-1):
  3 nodes × ~$1,200/month = $3,600/month fully loaded
  At 1,000 QPS this is ample headroom
  → 2–5x savings vs managed, at the cost of ~0.5 FTE operational overhead

Migration tipping point: around 60–80M queries/month or $500+/month managed spend

These are illustrative estimates — prices change and actual rates depend on your contract tier and region. The point is the structural pattern: query compute dominates, quantization moves the biggest needle on storage, and self-hosting wins on cost at sustained high QPS but requires someone to operate it.

What managed services don't bill for visibly

Three costs that routinely blow budgets:

Replication. High-availability deployments replicate data across availability zones. Pinecone's standard HA tier uses 2 replicas; Qdrant recommends 3-node clusters. That's 2–3x raw storage and RAM. A 90 GB primary index becomes 180–270 GB of actual provisioned RAM.

Metadata storage growth. As you add more filter fields per document — user_id, document_type, created_at, language, sensitivity_level, department — the payload index (B-tree or inverted index on each field) grows independently of the vector index. Collections that started at 5 metadata fields per document and grew to 20 fields over a year frequently have payload indexes matching or exceeding their vector index sizes.

Egress. If your application servers are in us-east-1 and your vector database is in us-west-2 — common when different teams picked their region independently — you're paying AWS egress on every retrieved chunk. At 10 chunks per query and ~1 KB per chunk, 1,000 QPS generates 864 GB/day of cross-region egress at $0.02/GB = $518/month in egress alone. Co-locate your vector database with your application tier.

What breaks in production

The high-selectivity silent recall drop

You deployed metadata filtering, tested it with a developer account that has 5,000 documents, and recall looked fine. In production, your enterprise customer with 500 documents is getting bad results. Their filter passes 500 out of 10M vectors (0.005% selectivity). Pre-filter HNSW recall at this selectivity: approximately 20–35%. No error is thrown. Latency looks normal. The results are just wrong.

Fix: check whether your database uses adaptive query planning. If it doesn't, implement a collection-per-tenant pattern for small-document tenants, or switch to a database that handles this correctly. Qdrant v1.9+ and Weaviate with payload indexes both handle high-selectivity filters correctly. Always measure recall on your actual filter distribution, not just on developer-scale test data.

The embedding version mismatch

You upgrade your embedding model — OpenAI deprecated ada-002 and you're moving to text-embedding-3-large. You re-embed new documents with the new model and index them. Old documents still have ada-002 vectors. Now your collection has two incompatible vector spaces, and queries with text-embedding-3-large are comparing against ada-002 vectors. Results for old documents are meaningless.

This is subtle because the distances don't error out — you get numbers, just wrong ones. The fix is a versioning field in metadata (embedding_model: "text-3-large") and either a full re-embedding migration or a parallel collection for each model version. Many teams discover this the hard way when they notice old documents disappearing from search results after an embedding upgrade.

The cold start performance cliff

HNSW indexes live in RAM. When a Qdrant or Milvus pod restarts, the index must be loaded from disk before queries are fast. On a 90 GB index, loading takes 2–5 minutes. During that window, queries either time out or are served from a slow disk-backed path with 10–100x worse latency. If you're running a single-node deployment, a pod restart becomes a multi-minute outage for search.

Fix: run at least 2 nodes, stagger restarts (Kubernetes rolling updates), and configure a readiness probe that requires the index to be fully loaded before accepting traffic. Milvus's distributed architecture handles this at the segment level; Qdrant's on-demand loading can stream index data while serving queries, with degraded initial latency.

The quantization recall surprise on domain-specific data

Binary quantization recall recovery numbers (97–99%) are measured on benchmark datasets like BEIR. Your domain data — scanned legal contracts, technical code, multilingual support tickets — may have a different distribution. Low-variance embeddings (where many vectors cluster in a narrow region) reduce the information content of sign bits. Always measure recall on a holdout from your actual data before quantizing production indexes. A 5% recall drop that's invisible on BEIR can be a 20% drop on your specific corpus.

Choosing your quantization and filtering strategy

This decision depends on your collection size, QPS, and workload characteristics. There's no universal answer, but the following table is a reasonable starting point:

Collection size	Recommended quantization	Filter strategy	Multi-tenancy
<1M vectors	float32, no quantization	Post-filter with moderate over-fetch	Single collection with metadata
1–10M vectors	INT8 or float16	Adaptive planner (if available), else post-filter	Namespace isolation
10–100M vectors	Binary + float32 re-rank	Adaptive planner required	Namespace or partition-by-tenant
>100M vectors	Binary + float32 re-rank + PQ for segments	Purpose-built engine (Milvus 2.5+) with adaptive	Dedicated collections for large tenants

For the database choice, the Vector Database Landscape 2026 article covers architecture and baseline comparisons. The filtering-specific short version: Qdrant v1.9+'s cardinality-estimating planner handles high-selectivity filters well; Weaviate's payload index architecture makes metadata filtering a first-class citizen; Milvus 2.5+ is the right choice above 100M vectors with complex filter patterns; pgvector with pgvectorscale is competitive through 50M vectors if you're already running PostgreSQL and want to avoid a separate operational footprint.

For hybrid search — combining metadata filtering with BM25 keyword matching — see Hybrid Search: Fusing Dense Vectors and Sparse BM25 Correctly. Metadata filters compose with hybrid search but add another layer of query planner complexity.

The decision in practice

Two questions determine most of the architecture:

What is your worst-case filter selectivity? If the most restrictive filter you'll issue passes >5% of your collection, adaptive post-filtering is fine and you can use any major vector database. If you'll regularly issue filters passing <1% of vectors (small tenants in a shared collection, narrow date ranges, specific user IDs), you need either adaptive planning (Qdrant v1.9+, Weaviate), partition-by-filter-value, or namespace isolation.

What is your sustained QPS? Under 100 QPS at 10M vectors, managed services are economically reasonable and the operational simplicity is worth the premium. At 1,000+ QPS sustained, you are paying for a lot of compute and the self-hosted economics become compelling quickly. The inflection point is roughly 60–80M queries/month or $500/month managed spend, but run the numbers for your specific workload using actual QPS and quantized index sizes, not storage alone.

The teams that get blindsided by vector database costs are almost always the ones who projected on storage and forgot query volume. Size the query compute first, anchor the storage estimate after quantization is accounted for, and co-locate the database with your application servers to avoid egress. Do those three things and the surprises become manageable.

For the broader production RAG picture — ingestion pipelines, chunking, evaluation — see the RAG from Scratch article and the crash course on embeddings and vector databases at scale.