Study finds GraphRAG’s citation precision and faithfulness vary widely across corpora and judges

A new arXiv preprint introduces a triple-robustness evaluation framework for retrieval-augmented generation (RAG) that holds the retrieval architecture fixed while varying three axes: the embedder, the corpus, and the judge model. The authors run 4,440 main-matrix experiments, 600 cross-corpus experiments, and 1,200 paired faithfulness judgments to isolate where and why GraphRAG’s performance diverges from vector RAG baselines.

Across all configurations, GraphRAG exhibits systemic over-citation, emitting between 11 and 15 citation IDs per answer while achieving citation precision between 0.12 and 0.23 and retrieval recall between 0.68 and 0.87. The authors label this pattern architecturally universal, meaning it persists regardless of the embedder or corpus used.

Faithfulness outcomes, however, are not universal. On typed-edge DO-178C requirements, GraphRAG’s faithfulness collapses from 74% to 40% as the number of reasoning hops increases. On Wikipedia paragraph chains via MuSiQue, the same pipeline shows the opposite trend, rising from 42% to 58% because over-cited paragraphs remain topically supportive even if not strictly necessary. The authors describe this as a corpus-conditional consequence of over-citation.

The study also finds that the relative performance of GraphRAG versus vanilla vector RAG is stratum-conditional but embedder-robust. For example, vanilla RAG wins on two-hop reasoning over DO-178C requirements, while GraphRAG wins on two-hop reasoning over MuSiQue paragraph chains, and these outcomes are consistent regardless of whether the local e5-small embedder or Azure text-embedding-3-small is used.

Evaluation stability is another concern. The authors report that single-judge LLM faithfulness is fragile to retrieval state: GPT-5.4’s self-kappa across embedders is 0.137, with verdict changes occurring on 41% of items. This suggests that judge-dependent metrics can vary widely even when the underlying retrieval and generation pipeline is unchanged.

To improve routing reliability, the authors train a learned router on dense embeddings alone and achieve a macro-F1 of 0.86 on hop classification. They argue that triple-robustness testing—varying embedders, corpora, and judges—should be the minimum bar for trustworthy claims about RAG architectures.