GraphRAG and Structured Retrieval for Multi-Hop Reasoning

Eight hundred merger due-diligence documents. One question from the M&A team: "which of the counterparty's subsidiaries have prior litigation involving the same IP claims?" A standard RAG system retrieved the chunks most similar to the query. It found individual mentions of subsidiaries. It found individual mentions of IP litigation. It never connected them, because no single chunk said "Subsidiary A has prior IP litigation" — that fact required joining a sentence on page 12 of one document with a footnote on page 94 of another. The system returned a partial answer, stated with confidence, missing the critical connection the deal team needed.

This is the structural limit of flat retrieval. Cosine similarity finds chunks that look like the query. It cannot follow relationships between entities across document boundaries. The moment the answer requires chaining two or more facts — multi-hop reasoning — flat search fails in a way that no reranker or query transformation can fully compensate for.

Why flat retrieval breaks on multi-hop queries

Standard RAG, even with hybrid search and reranking (covered in Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking), retrieves chunks by how similar they are to the query in embedding space. The retrieval step is inherently a similarity operation, not a reasoning operation.

For a query like "what are the connections between Company A, Company B, and the patent filed in 2019?", the relevant evidence might be:

• Document 1, page 3: "Company A licensed the '2019 patent' from Innovate LLC."
• Document 7, page 11: "Company B acquired Innovate LLC in 2021."
• Document 12, page 2: "The '2019 patent' is currently in litigation."

None of these chunks, individually, answers the query. Any single chunk has a moderate cosine similarity to the query — mentioning one or two of the entities — but the chunks are not similar to each other in the way that would allow a ranked list to surface all three together.

The visualizer below steps through the standard flat pipeline. Watch the retrieval stage: every chunk is scored against the query independently, one similarity comparison at a time. There is no step where two chunks are joined on a shared entity — which is precisely the operation multi-hop queries require.

{ "type": "rag-pipeline", "query": "policy", "title": "Standard RAG pipeline (baseline for comparison)" }

Query transformation techniques from Query Transformation: Rewriting, HyDE, and Multi-Query Expansion help by generating multiple query variants and retrieving for each, then merging. But this is a band-aid on a structural problem: you are still doing multiple flat lookups and hoping the union covers all necessary evidence. When the required chain has three or more hops — and the key entities are named with slight variations across documents — the gap between "we retrieved some relevant chunks" and "we retrieved all necessary evidence in the right relational context" is wide.

How GraphRAG solves it: index-time reasoning

The insight behind GraphRAG is to move relational reasoning from query time to index time. Instead of asking the LLM to synthesize facts from flat chunks during generation, you extract the relationships during indexing and store them in a form that makes multi-hop queries answerable by graph traversal.

Microsoft's GraphRAG system, released in 2024, implements this in four phases:

flowchart LR
    D[Raw documents] --> EE["Entity extraction\n(LLM per chunk)"]
    EE --> EB[Entity + relation\ngraph]
    EB --> CD["Community detection\n(Leiden algorithm)"]
    CD --> CS["Community summarization\n(LLM per community)"]
    CS --> HSI[Hierarchical\nsummary index]

    style D fill:#0e7490,color:#fff
    style EB fill:#a855f7,color:#fff
    style HSI fill:#00e5ff,color:#0a0a0f

Entity extraction. An LLM processes each chunk and extracts entities (people, organizations, locations, concepts) and the relationships between them. This is the expensive step — it requires an LLM pass over every chunk in the corpus.

Graph construction. Extracted entities and relationships are merged into a global graph. Entities mentioned across multiple documents are deduplicated and linked. The graph now represents the entire corpus as a network of connected facts.

Community detection. The Leiden algorithm partitions the graph into communities — groups of densely interconnected entities. The algorithm runs at multiple resolutions, producing communities at different granularities (fine-grained local clusters and coarser-grained global themes).

Community summarization. An LLM generates a natural-language summary of each community at each resolution level. These summaries are the retrieval artifacts for global queries.

At query time, GraphRAG offers two retrieval modes. Local search traverses the graph around entities mentioned in the query, retrieving entity descriptions, related entities, and the chunks they appear in. Global search does a map-reduce: it sends the query to many community summaries in parallel, asks each for a partial answer, then reduces the partial answers into a final response. Global search is how GraphRAG answers "what are the main themes across this entire corpus?" — a query that flat retrieval cannot handle at all, because no single chunk or small set of chunks represents the whole.

LightRAG: lower-cost graph retrieval

Microsoft GraphRAG's index construction cost is its primary practical barrier. LightRAG (Guo et al., 2025) provides a more economical alternative with a dual-channel retrieval pattern.

Where GraphRAG builds full community hierarchies with multiple summarization passes, LightRAG uses a leaner extraction step and retrieves through two channels simultaneously: a fine-grained channel that retrieves entity-level evidence (specific facts about named entities), and a high-level channel that retrieves concept-level summaries. The two channels are fused at generation time.

The tradeoff is that LightRAG's global summarization is less thorough than GraphRAG's hierarchical community summaries. For queries like "what are the main themes across this entire corpus?", full GraphRAG wins. For queries like "how are Entity A and Entity B connected?", LightRAG produces comparable accuracy at roughly one-third the construction cost.

In practice, LightRAG is the better default when you need graph-structured retrieval but have a limited token budget. Use full GraphRAG when global corpus-wide synthesis is the primary requirement and cost can be justified by the use case.

Graph-Aware Late Chunking: bridging graph and vector retrieval

A 2026 paper extended the late chunking technique — covered in Chunking Strategies That Actually Matter — by injecting relational metadata from the knowledge graph into the chunk embeddings. The result: flat vector retrieval that is aware of graph-level relationships without requiring full graph traversal at query time.

The mechanism: during late chunking's embedding step, each chunk's embedding is conditioned on the entity relationships that chunk participates in, extracted from the knowledge graph. A chunk mentioning "Innovate LLC" now has its embedding shifted toward the semantic neighborhood of the entities Innovate LLC is connected to — "Company B (acquired)", "2019 patent (licensed)", "IP litigation (related)".

This is practically significant because it enables a single HNSW index query (see ANN Indexes Under the Hood) to surface chunks that are relationally relevant without requiring graph traversal. The approach reduces query-time complexity while preserving most of the multi-hop accuracy gains from graph-structured indexing. It is early-stage as of mid-2026, but it points toward a convergence of graph and vector retrieval rather than a forced choice between them.

When to actually use graph retrieval

The single biggest mistake engineers make with GraphRAG is deploying it as a general upgrade to flat RAG. It is not. The cost-benefit only makes sense for specific query patterns.

flowchart TD
    QP{What kind of queries\ndo you actually have?} --> Q1["Single-hop factual:\n'What is the refund policy?'\n'Define term X'"]
    QP --> Q2["Multi-hop relational:\n'Which entities share both\ninvestors and suppliers?'"]
    QP --> Q3["Global synthesis:\n'What are the main themes\nacross this corpus?'"]
    QP --> Q4["Entity-centric:\n'What did Person A\nsay about Topic B?'"]

    Q1 --> A1[Flat hybrid search\nwith reranking]
    Q2 --> A2[Local graph search\nLightRAG or GraphRAG]
    Q3 --> A3[Global graph search\nMicrosoft GraphRAG]
    Q4 --> A4[Hybrid: vector for recall\n+ graph for entity context]

    style A1 fill:#15803d,color:#fff
    style A2 fill:#a855f7,color:#fff
    style A3 fill:#00e5ff,color:#0a0a0f
    style A4 fill:#0e7490,color:#fff

Graph retrieval is genuinely valuable in these scenarios:

• Research corpora: academic literature where the answer requires synthesizing findings across dozens of papers.
• Legal discovery: identifying connections across large document sets — who knew what, when, and who they communicated with.
• Competitive intelligence: "what are the relationships between these companies' patents, subsidiaries, and partners?" across thousands of filings.
• Scientific literature mining: "what proteins interact with this compound, and what are the downstream effects?" — where the knowledge graph mirrors real-world ontologies.

For enterprise Q&A, internal knowledge bases, customer support, and most document-search applications, hybrid search with reranking is the correct answer. The queries are typically single-hop, the documents are relatively well-organized, and the cost difference between flat and graph retrieval is not justified by any measurable accuracy gain.

The numbers: cost and latency

Scenario: 200,000-token legal corpus (roughly 150 dense documents)

OPTION A: Flat hybrid RAG (BM25 + dense + reranker)
  Ingestion: embed 200k tokens at Voyage-3 pricing (~$0.02/M tokens) = ~$0.004
  Query: 2 retrievals + rerank + generation ≈ $0.03/query
  Index rebuild after update: $0.004 (re-embed changed chunks only)
  Query latency: 500-800ms (retrieval + rerank + generation)

OPTION B: Microsoft GraphRAG
  Ingestion: ~6x token multiplier on 200k tokens = 1.2M tokens consumed
  At GPT-4o mid-2026 pricing (illustrative): ~$10-25 one-time construction cost
  Query (global, map-reduce over communities): 3-5 LLM calls = $0.10-0.30/query
  Query (local graph traversal): 1-2 LLM calls = $0.05-0.10/query
  Index rebuild after update: near-full reconstruction = ~$10-25 again
  Query latency: 2-8 seconds (map-reduce parallelized, still multiple calls)

OPTION C: LightRAG
  Ingestion: ~2-3x token multiplier = 400-600k tokens consumed
  Construction cost: ~$4-10
  Query: similar to GraphRAG local search = $0.05-0.10/query
  Query latency: 1-3 seconds

Break-even for GraphRAG over flat RAG:
  Extra index cost: ~$20 (illustrative, one-time)
  Extra query cost: ~$0.07/query vs $0.03/query = $0.04 premium
  If you serve 500 queries before the corpus changes: $20 + $20 extra = $40 total premium
  Only justified if multi-hop accuracy improvement is worth $40 + higher latency per refresh cycle.

And the benefit side of the ledger, so you are not weighing $40 against a vibe: the GraphRAG paper's own evaluation — LLM-as-judge on roughly million-token news and podcast transcript corpora — reported global search beating naive vector RAG on comprehensiveness and diversity in roughly 70-80% of head-to-head comparisons for global sensemaking questions. The LightRAG paper reports similar directional wins over both naive RAG and full GraphRAG on multi-document QA, at a fraction of the construction cost. Two caveats. These are benchmarks built around the query types graph retrieval exists to serve, and the judged dimensions (comprehensiveness, diversity) are not the same thing as factual accuracy — neither number tells you how often your users ask global questions or whether your corpus extracts cleanly.

For most teams, the cost profile makes the decision obvious: build the graph index only if you have validated, through measurement, that flat retrieval fails on your actual queries.

What breaks

Graph retrieval introduces failure modes that flat RAG does not have, on top of all the ones it shares.

Entity extraction errors propagate silently. If the LLM misreads "Dr. Jane Smith" as two separate entities "Dr. Jane" and "Smith" during extraction, or confuses two companies with similar names, those errors become structural facts in the graph. Downstream queries will get wrong answers with full confidence — there is no "chunk not found" signal, because the graph always has something to return. Validate extraction quality on a sample before trusting the graph at scale.

Relationship hallucination. LLMs sometimes infer relationships that do not exist in the source text. "Company A frequently works with Company B" might be synthesized from two separate mentions that never actually implied a working relationship. This is especially common in the community summarization step, where the LLM is asked to synthesize patterns from dozens of entities at once.

Stale graphs. GraphRAG indexes are expensive to rebuild. Unlike flat RAG where you can re-embed and re-index changed chunks cheaply, graph construction requires re-running the full extraction and summarization pipeline over the affected documents and their graph neighborhoods. In practice, this means teams accept staleness or batch updates rather than real-time indexing. For dynamic corpora — news, tickets, customer communications — this is a serious limitation.

Global query over-smoothing. Community summaries are inherently lossy compressions of the underlying documents. The global map-reduce query mode can return accurate high-level answers while completely missing specific details that are in the source but below the summarization threshold. Users who expect the system to find specific sentences will be surprised when it instead returns thematic overviews.

Graph sparsity for narrow corpora. Graph retrieval needs enough entities and relationships to form meaningful communities. For small, narrow corpora — a few dozen short documents on a specific topic — the graph is sparse, community detection produces poor groupings, and the system performs worse than flat retrieval while costing more to build.

The RAG Evaluation, Failure Modes, and the Decision Between RAG, Long Context, and Fine-Tuning article covers how to measure these failures with RAGAS and when to trust automated metrics vs. human eval. For GraphRAG specifically, you need to add graph-specific evaluation: entity extraction precision and recall on a sample, relationship hallucination rate, and per-query-type accuracy by retrieval mode.

Implementation sketch

Here is a minimal graph-retrieval setup using LlamaIndex's PropertyGraphIndex — the API LlamaIndex has recommended for graph RAG since 2024, replacing the deprecated KnowledgeGraphIndex. Be clear-eyed about what this gives you: LLM triple extraction at index time plus graph traversal at query time. It does no community detection and builds no community summaries — it handles multi-hop entity queries, not GraphRAG-style global search.

from llama_index.core import SimpleDirectoryReader
from llama_index.core.indices.property_graph import (
    PropertyGraphIndex,
    SimpleLLMPathExtractor,
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Step 1: construct the property graph index
# This is the expensive step — runs LLM extraction over every chunk
documents = SimpleDirectoryReader("./docs").load_data()

llm = OpenAI(model="gpt-4o", temperature=0)

pg_index = PropertyGraphIndex.from_documents(
    documents,
    llm=llm,
    embed_model=OpenAIEmbedding(model="text-embedding-3-large"),
    kg_extractors=[
        SimpleLLMPathExtractor(llm=llm, max_paths_per_chunk=10),
    ],
    show_progress=True,
)

# Step 2: query — retrieves matching triples and the chunks they came from
query_engine = pg_index.as_query_engine(include_text=True)

response = query_engine.query(
    "Which subsidiaries share both suppliers and legal counsel with the parent company?"
)
print(response)

For community summaries and global search — the parts that make GraphRAG GraphRAG — you need the official graphrag pipeline, which handles entity extraction, Leiden community detection, and hierarchical summarization end to end:

# Using the official graphrag pipeline (pip install graphrag)
# Requires OPENAI_API_KEY; 'graphrag init --root .' scaffolds the config

# Build the index: extraction, communities, hierarchical summaries
graphrag index --root .

# Local search: entity-focused graph traversal
graphrag query --root . --method local --community-level 2 \
  --query "Which entities have relationships to the 2019 patent dispute?"

# Global search: map-reduce over community summaries
graphrag query --root . --method global --community-level 2 \
  --query "What are the primary risk themes across the entire document corpus?"

For programmatic use, the supported surface is the async graphrag.api.local_search / graphrag.api.global_search functions, which take the loaded config plus the output dataframes (entities, communities, community reports) — do not import the CLI's internals.

The community_level parameter selects which level of the Leiden hierarchy to query. Level 0 is the root: the broadest communities, giving more holistic but less specific answers from fewer summaries. Higher values select smaller, finer-grained sub-communities — more specific answers, more map-reduce calls. In practice, level 2 is a reasonable default for most corpora.

Routing between retrieval strategies

In a real system, you rarely want to route all queries through graph retrieval. The practical architecture is a query classifier that sends queries to the right retrieval strategy:

sequenceDiagram
    participant U as User query
    participant C as Query classifier
    participant V as Vector/hybrid index
    participant G as Knowledge graph
    participant L as LLM

    U->>C: "Which companies share board members and litigation history?"
    C->>C: classify: multi-hop relational → graph retrieval
    C->>G: local graph search (entity + neighborhood)
    G-->>C: subgraph: entities + relationships + source chunks
    C->>L: generate with graph context
    L-->>U: "Company A and Company B share..."

    Note over C,V: Single-hop query path
    U->>C: "What is the refund policy?"
    C->>C: classify: single-hop factual → vector retrieval
    C->>V: BM25 + dense + reranker
    V-->>C: top-3 relevant chunks
    C->>L: generate with chunk context
    L-->>U: "Refunds are processed within..."

The classifier itself can be a lightweight LLM call (50-100ms with a small model) or a fine-tuned classifier. The routing decision saves the graph traversal cost for the 80-90% of queries that don't need it.

This hybrid approach is also where the Agentic RAG pattern becomes relevant: if the classifier's confidence is low, or if the initial retrieval returns insufficient evidence, an agent can decide to try the other retrieval mode before giving up.

The decision in practice

GraphRAG is the right tool when three conditions are simultaneously true: (1) your queries genuinely require multi-hop reasoning or global corpus synthesis, (2) you have validated that flat hybrid retrieval fails on those queries by measuring it, and (3) your corpus is stable enough that the index construction cost amortizes over many queries before a rebuild is needed.

If condition (1) is not true, you are paying a 5-8x indexing cost for a capability you do not use. If condition (2) is not validated, you are guessing that you need GraphRAG based on intuition rather than evidence. If condition (3) is not true — for example, your knowledge base updates daily — graph retrieval's rebuild cost makes it operationally impractical.

The right path for most teams: start with flat RAG built correctly and upgraded to hybrid search with reranking. Measure your query failure rate and classify failures by type. If a non-trivial fraction of failures are structurally multi-hop — the answer requires evidence from multiple documents that no flat retrieval strategy surfaces together — then build a GraphRAG index, route multi-hop queries to it, and keep flat retrieval for everything else.

LightRAG is the pragmatic middle ground: lower construction cost than full GraphRAG, better multi-hop accuracy than flat search, and straightforward to deploy via LlamaIndex or its own SDK. Start there before committing to the full Microsoft GraphRAG pipeline.

And if your corpus is small enough to fit in a context window, or dynamic enough that graph rebuilds are not feasible, consider the trade-off discussed in Long Context vs RAG: The Million-Token Question — sometimes the operationally simplest answer is feeding the whole thing to the model, accepting the 20-24x cost premium on the subset of queries that warrant it.

Knowledge graphs in RAG are not a general upgrade. They are a specialized tool for a specific problem. The teams that benefit most from them deploy them knowingly, having confirmed through measurement that their queries require it.