Ask an AI assistant built on retrieval-augmented generation (RAG) what your travel policy says about hotel rates in London, and it will usually get it right. The answer sits in one passage of one document. Ask it which customers are affected if one supplier misses a delivery, and it often struggles. That answer is spread across purchase orders, supplier records and customer contracts, and the link between them isn’t written down in any single passage.
That second kind of question is where a knowledge graph earns its place, and where pairing knowledge graphs and LLMs starts to pay off. A knowledge graph stores entities and the relationships between them explicitly, so a system can follow the connections instead of hoping similar-looking text turns up in a search. Not every AI use case needs one, though. Recent research shows graph-based retrieval wins on some question types and loses on others, so the decision should start with the questions you need to answer.
This article explains what knowledge graphs and LLMs each contribute, when vector search alone is enough, when you need a graph as well, and how to combine the two.
Key takeaways
- Knowledge graphs and LLMs cover different ground: a knowledge graph stores entities and relationships explicitly, while an LLM reads and writes language. Together they answer questions that depend on how things connect.
- Vector RAG is enough for questions answered by one passage, such as policy wording, how-to steps and single facts.
- Add a knowledge graph when questions span several entities or documents, need exact counts or hierarchies, involve ambiguous names, or ask about themes across a whole corpus.
- In GraphRAG-Bench (ICLR 2026), graph methods beat vanilla RAG on complex reasoning and summarization but not on simple fact retrieval.
- Most enterprise assistants end up using knowledge graphs and LLMs in a hybrid design: a graph and a vector index behind a router that sends each question to the right store.
What a knowledge graph adds to an LLM
A knowledge graph is a structured map of things and how they relate. Nodes are entities such as customers, products, suppliers, contracts or patients. Edges are typed relationships, such as “supplies,” “is covered by” or “reports to.” An ontology defines which entity types and relationships are allowed, so everyone means the same thing by “customer.”
An LLM has none of that by default. It knows language and general facts from training, and with vector RAG it can pull in passages that look similar to the question. Vector search finds text that resembles the question, but it has no model of how a supplier connects to a plant, a part and a customer. A knowledge graph gives the LLM three things it lacks:
- Explicit relationships it can follow across several hops, from supplier to part to plant to customer.
- One identity per entity, so “IBM,” “International Business Machines” and a customer ID resolve to the same node.
- Facts that can be checked. A graph query returns the exact records behind an answer, which makes the answer easier to explain and audit.
| Dimension | LLM alone | LLM with vector RAG | LLM with a knowledge graph |
|---|---|---|---|
| Where answers come from | Training data | Passages similar to the question | Entities and relationships in your graph, often plus passages |
| Strong at | General language tasks | Answers found in one passage | Multi-hop, aggregate and cross-document questions |
| Weak at | Company-specific facts | Connecting facts spread across sources | Open-ended questions about text the graph never captured |
| How answers are checked | Hard to check | Citation to a passage | Citation to nodes, edges and passages |
| Effort to build | None | Moderate: parsing, chunking, embeddings | Higher: ontology, extraction, entity resolution |
What an LLM adds to a knowledge graph
The help runs both ways. Knowledge graphs have been around for years, but two problems kept them expensive: building them by hand, and querying them, which needs a graph query language such as Cypher or SPARQL. LLMs ease both.
- Building the graph. An LLM can read contracts, reports and tickets and propose entities and relationships to add, which a review step then confirms. That extraction depends on reading documents accurately, which is the job of document intelligence tools such as our DocuIntel™ accelerator.
- Querying the graph. A user asks a question in plain English, and the LLM turns it into a graph query, runs it, and explains the result.
- Explaining the result. The LLM turns a set of nodes and edges into a readable answer, with each statement traceable to the records behind it.
When vector RAG alone is enough
For many enterprise questions, a well-built vector RAG system is the right answer and a graph would add cost without adding accuracy. That holds when:
- the answer sits in one passage, such as a policy clause, a product spec or a support article;
- the question is procedural, such as how to reset access or file an expense;
- the corpus is small or flat, with few cross-references between documents;
- speed to launch and low running cost matter more than handling the hardest questions.
The research backs this up. GraphRAG-Bench, a benchmark presented at ICLR 2026, starts from the observation that graph-based RAG frequently underperforms vanilla RAG on real-world tasks. On its Novel dataset, vanilla RAG with reranking scored 60.9% on fact retrieval, and no graph-based method beat it. A separate 2025 systematic comparison of RAG and GraphRAG by Han and colleagues reached a similar conclusion: each approach has distinct strengths depending on the task.
When you need a knowledge graph as well
The same benchmark shows where graphs pay off. On complex reasoning, the best graph method scored 53.4% against 42.9% for vanilla RAG. On contextual summarization, the gap was 64.4% against 51.3%.
In practice, five signals tell you a graph is worth adding:
- Multi-hop questions. The answer depends on following relationships across entities or systems, such as supplier to part to plant to customer.
- Aggregates and hierarchies. Questions ask “how many,” “which of these” or “everything under this business unit,” and the answer has to be exact.
- Ambiguous entities. The same name means different things, or one thing has many names across systems.
- Questions about a whole corpus. Users want recurring themes across thousands of incident reports or contracts, which no single passage contains.
- Agents that take actions. An agent changing a record needs to know what that record is connected to, and what the change will affect.
| Question type | Example | Best fit |
|---|---|---|
| Single fact in one document | What is our hotel rate cap in London? | Vector RAG |
| Procedure or how-to | How do I request access to the data warehouse? | Vector RAG |
| Multi-hop across entities | Which customers are affected if Supplier X misses its next delivery? | Knowledge graph with an LLM |
| Aggregate or hierarchy | How many open claims involve policies renewed this quarter? | Knowledge graph or SQL, with an LLM |
| Themes across a corpus | What are the recurring root causes in last year’s incident reports? | Graph-based summaries (GraphRAG) |
| Mixed questions | A general enterprise assistant | Hybrid, with a router |
How knowledge graphs and LLMs work together
Most enterprise assistants get a mix of all these question types, so the practical way to combine knowledge graphs and LLMs is a hybrid design. Han and colleagues found that combining RAG and GraphRAG, by routing each query or merging their results, gave consistent gains over either approach alone.
A workable build sequence looks like this:
- Start from the questions. Collect the real questions users ask and mark which ones need relationships, counts or cross-document reasoning. Those decide whether you need a graph at all.
- Design a small ontology. Define only the entity types and relationships those questions need. A focused ontology of a dozen entity types beats an enterprise-wide model that never ships.
- Load structured data first. CRM, ERP and master data already hold clean entities and relationships, so they make the most reliable first layer of the graph.
- Enrich from documents. Use an LLM to extract entities and relationships from contracts, reports and tickets, with human review for the relationship types that drive decisions.
- Resolve entities. Merge duplicates so each customer, supplier or product has one node. This is usually the hardest step, and it’s a data quality problem more than an AI problem.
- Link passages to nodes. Keep the vector index for text, and tag each passage with the entities it mentions, so the system can move between the graph and the source documents.
- Route, generate and cite. A router sends each question to the graph, the index or both. When the LLM writes a graph query, validate it before it runs, and cite both the records and the passages behind the answer.
Permissions apply to the graph too. A relationship can reveal as much as a document, so the graph needs the same access rules as the sources it was built from.
What it costs, and where projects stall
A knowledge graph is a bigger build than a vector index, and the costs show up in predictable places.
- Extraction cost. Using an LLM to pull entities and relationships out of every document costs far more than embedding the same documents. Microsoft’s LazyGraphRAG cuts this by deferring extraction until a question needs it, according to an August 2026 VentureBeat review of GraphRAG research.
- Ontology sprawl. Teams that try to model the whole enterprise before answering a single question rarely finish. Keep the ontology tied to the questions in scope.
- Entity resolution. Duplicate and conflicting records across systems produce a graph that connects the wrong things, and the LLM will explain those wrong connections fluently.
- Freshness. The graph has to change when the source systems change, or answers drift out of date.
- Evaluation. The same VentureBeat review notes an audit that found position bias in LLM judges could shift reported win rates by more than 30 points. Test with reference answers for each question type, and don’t rely only on a judge model comparing two outputs.
Most of these are data problems, and they decide whether the graph helps. Clean, connected and governed data is what makes a knowledge graph trustworthy, and it matters as much as the model choice.
How Compunnel Digital helps
Compunnel Digital builds knowledge graphs for AI through its Enterprise Knowledge and Document AI practice. The work covers document ingestion and classification with DocuIntel™, intelligent extraction of entities, clauses, metrics and relationships, enterprise search and retrieval with SearchCore™, and a knowledge graph and semantic layer that maps relationships between documents, entities and concepts for context-aware AI responses. Compunnel reports 80% to 90% faster search and review where SearchCore™ is deployed.
In our AI Document Intelligence engagement, a five-stage AI pipeline replaced manual data extraction from complex scientific PDFs and produced structured, simulation-ready data. The same knowledge layer can feed role-based AI Workforce and Copilots, which are built on SearchCore™ and integrate enterprise knowledge into the tools people already use.
If your assistants handle single-passage questions well but stall on questions that cross systems, talk to our team about where a knowledge graph would help and where vector search is enough.
Frequently asked questions
What is GraphRAG?
GraphRAG is retrieval-augmented generation that uses a knowledge graph as one of its retrieval sources. Instead of only finding similar passages, the system follows entities and relationships in the graph, or uses summaries of graph communities, and gives those results to the LLM to write the answer.
Do knowledge graphs stop LLM hallucinations?
They reduce them for questions the graph covers, because the LLM works from explicit records and each statement can be traced to the nodes and edges behind it. They don’t eliminate errors. A graph built from bad data, or an LLM that writes a wrong graph query, still produces wrong answers, so validation and evaluation are still needed.
Do we need a graph database?
Not always. If your questions are mostly aggregates over well-structured tables, SQL with an LLM that writes queries may be enough. A graph database pays off when questions need to follow several relationships, when relationships change often, or when the same entity appears across many systems.
Can an LLM build the knowledge graph automatically?
An LLM can draft one by extracting entities and relationships from documents, but it needs a defined ontology, entity resolution and human review for the relationships that drive decisions. Structured systems such as CRM and ERP usually make a better starting layer than documents.
What is the difference between a knowledge graph and a vector database?
A vector database stores text as embeddings and finds passages that are similar in meaning to a question. A knowledge graph stores entities and the explicit relationships between them. Vector search answers “what text looks like this?” and a graph answers “what is connected to this, and how?”
Conclusion
Knowledge graphs and LLMs do different jobs. The LLM understands the question and writes the answer. The graph supplies the connections that aren’t written down in any single passage. If your questions live in one document, vector RAG is usually enough. If they span entities, systems or a whole corpus, add a graph, start from the questions, and invest in entity resolution before anything else.
See how our DocuIntel™ and SearchCore™ accelerators turn enterprise documents into a connected, searchable knowledge layer.
Sources
- Xiang et al., When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation (GraphRAG-Bench), ICLR 2026; results from Table 2, Novel dataset
- Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights, 2025, revised March 2026
- VentureBeat, Stop graphing everything: When GraphRAG actually beats vector RAG, August 2026
- Compunnel Digital, Enterprise Knowledge and Document AI and AI Document Intelligence case study
