Why Knowledge Graphs Are Becoming AI's Missing Layer
Copyright
Why Knowledge Graphs Are Becoming AI's Missing Layer
Most AI platforms today are built around similarity search-based retrieval: break documents into chunks, embed them as vectors, and pull back whatever looks semantically similar to a query. This works remarkably well for a lot of problems, but has a blind spot that impacts quality of AI outcomes as our AI applications become more sophisticated - it has no concept of how things are connected.
As AI platforms evolve from systems that surface information to systems that are expected to reason over the information, that blind spot turns into a bottleneck. Agents need memory and situational awareness, not just a bag of retrieved text chunks. Retrieval-augmented generation (RAG) needs higher-quality context, not simply more context. And increasingly, the questions worth asking aren't “what sounds similar to this?” but “how is this connected to that, and why does it matter?”
This is the gap that knowledge graphs are designed to close. The research summarized here looks at what graph-based approaches can and can't do, how they actually work under the hood, and what it looks like to put one into practice.
Where Similarity Search Excels - and Where It Runs Out of Road
Vector-based retrieval is genuinely good at a specific set of problems:
- Retrieving images or documents based on content similarity
- Powering personalized recommendations from user-interaction patterns
- Flagging anomalies in fraud or system-error detection
- Clustering and classifying data for curation
- Finding chunks of text that share similar words or tokens for RAG context
The limitation isn't that vector search is weak - it's that similarity and relevance are different things. A chunk can be semantically close to a query and still be contextually irrelevant, or structurally important without being at all semantically similar. More specifically, similarity search:
- Finds similar content but doesn't understand relationships between entities
- Has no native way to perform multi-hop reasoning (following a chain of connections across several steps)
- Doesn't explicitly represent how entities connect across documents
- Has no built-in mechanism for detecting communities or clusters of related knowledge
The right way to frame this isn't “graphs versus vectors” - it is additive. Vector search finds the neighborhood; a graph maps the roads connecting the houses in it.
What a Graph Layer Adds
A graph layer contributes capabilities beyond those achievable through similarity scoring structurally alone:
- It reveals hidden relationships: Two records can be connected through a shared entity even if they never reference each other directly - for example, two separate support tickets that turn out to describe the same underlying component failure; or an employee’s travel expense transactions and company expense reimbursement policy.
- It supports multi-hop reasoning: A graph can follow a chain of relationships to answer a compound question - tracing from a component to the product it is part of, to the orders that included it, to the customers who placed those orders - in a single traversal.
- It detects communities and clusters automatically: Graph algorithms can surface groups of related documents, teams, or systems without anyone manually tagging the relationships.
- It connects knowledge across boundaries: Documents, people, assets, decisions, and timelines can live in one queryable layer, even when they are scattered across different data systems.
Bridging the Structured/Unstructured Divide
One of the more persistent problems in enterprise data is that structured and unstructured data rarely meet. Structured data - database tables, schemas, metadata - is queryable using SQL and lives in warehouses and catalogs. Unstructured data - documents, PDFs, images, audio, videos - is searchable using embeddings or full-text search and lives in object stores and file systems. A product entity sitting in a database table and the same product mentioned in a support document are, today, two completely isolated records. Metadata enrichment helps describe each side better, but it doesn't make them traversable together.
A graph bridges this gap in a conceptually simple way: entities extracted from unstructured content - through named-entity recognition and relation extraction - become nodes. Structured records such as database rows also become nodes. Wherever a shared identifier, name, or resolved entity exists between the two, an edge connects them. The result is a single layer, where a database record and a document mention of the same real-world entity are no longer isolated - they're the same node, reachable from either direction.
Getting the Terminology Right
Three terms get used almost interchangeably in casual conversation, but they describe distinct things:
- A graph is a data model - a way of representing information as nodes and the relationships between them.
- A graph database is an engine: software that stores, queries, and manages graph-structured data.
- A knowledge graph is a graph populated with real-world entities and their relationships, typically enriched with an ontology and some capacity for reasoning.
The building blocks underlying all three are consistent:
- Node (or vertex): An entity - a person, document, product, or concept.
- Edge (or relationship): A connection between two nodes, such as “created by,” “depends on,” or “approved by.”
- Label: A category applied to a node or edge, like “Person” or “Document.”
- Property: A key-value attribute attached to a node or edge, such as {status: “active”}.
- Triple: The foundational unit in one modeling tradition - a subject-predicate-object statement, such as “Document A → approved by → Person B.”
Two Philosophies for Modeling a Graph
Not all graph technologies solve the same problem, and the split largely comes down to two modeling philosophies.
Property graphs represent data as nodes and edges that can each carry multiple key-value properties. They're flexible, intuitive, and well suited to application-specific graphs - the kind of structure you'd build to power an agent's working memory or a recommendation engine. Their query style tends to be pattern-matching: describe the shape of the connections you're looking for, and the engine finds matches.
Triple-based (semantic) graphs represent data strictly as subject-predicate-object statements. They're standards-based and ontology-driven, and they support formal reasoning and inference - the ability to derive new facts from existing ones based on defined rules. Their query style is built around matching triple patterns rather than arbitrary graph shapes.
The distinction isn't about which is better - it's about which question you're asking. Property graphs answer “how are things connected in this specific application?” Triple-based graphs answer “what does this thing mean across systems and standards?” A patient-visited-clinic-prescribed-medication relationship, with dosage and date as properties, is a property-graph problem. “This medication is a type of anti-inflammatory, and anti-inflammatories are contraindicated for this condition” is a semantic-reasoning problem.
One clarification is worth making explicit, because it's a common source of confusion: a widely used API specification for requesting structured data from client applications is often mistaken for a graph database query language. It isn't a graph database query language - it has no native concept of graph traversal, pattern matching, or path-finding. It solves a different problem (a flexible API contract between a client and a server), not graph querying.
Four Architectural Patterns, Four Different Jobs
Graph technology isn't a single architecture - it shows up in at least four distinct patterns, and they aren't competing with each other so much as serving different operational needs:
- Native engines, purpose-built with the graph as the primary storage model. These deliver the best traversal performance and the most mature tooling, at the cost of more complex horizontal scaling.
- In-memory engines, which hold the graph primarily in RAM for the fastest possible multi-hop traversal and real-time updates - ideal for real-time agent context, limited by available memory.
- Multi-model engines, where a single system supports graph alongside document, key-value, and search/vector models. These reduce operational complexity for teams that don't want to run a separate database, at some cost to graph-specific performance.
- Zero-ETL / virtualized query engines, which overlay graph query capabilities directly on existing data stores without copying data anywhere. These offer the fastest time-to-value and no data movement, with performance that depends on the underlying storage or database.
Choosing among these is a workload decision, not a trend decision: native engines suit system-of-record graph workloads, in-memory suits real-time agent context, multi-model suits teams consolidating infrastructure, and virtualized engines suit rapid prototyping and exploration over data that already exists somewhere else.
How Graph Storage Actually Works
The performance case for native graph storage comes down to one idea: index-free adjacency.
In a relational database, relationships are computed at query time through join operations over indexes. Finding a node's immediate neighbors means a join; finding neighbors of neighbors means two joins; a five-hop path becomes computationally impractical as the dataset grows, because the cost compounds with every additional hop.
In a native graph engine, each node record directly contains a pointer to its neighboring nodes. Traversing from one node to an adjacent one is a constant-time operation, regardless of how large the overall dataset is. A graph with a billion nodes traverses a local neighborhood at the same speed as a graph with a thousand nodes, because the engine never has to search the whole dataset to find a connection - it just follows a pointer that's already there.
What Benchmarking Actually Showed
Performance testing against a standard, industry-recognized financial-services graph benchmark - run at two scales, one with roughly 100,000 entities and 588,000 relationships, and a second about ten times larger - found that a virtualized, zero-ETL query approach performed competitively with a native graph engine running its own dedicated storage. At the larger scale, the virtualized approach was marginally faster in some configurations; in others, it was essentially on par.
The practical takeaway is that organizations don't have to treat “migrate everything into a dedicated graph engine” as a prerequisite for getting value from graph queries. A well-designed graph virtualization layer over data that already exists can deliver comparable latency without a data-movement project first - which matters enormously for how quickly a graph initiative can get off the ground.
Seeing It in Practice: Connecting Fraud Signals
To ground these ideas, consider a proof-of-concept built around a common enterprise problem: fraud signals that are split between structured transaction data and unstructured documents such as contracts, and that are slow and manual to connect today. According to the Association of Certified Fraud Examiners' 2026 Report to the Nations, organizations lose an estimated 5% of annual revenue to occupational fraud, and the median scheme runs about twelve months before detection - a gap measured in the time it takes evidence sitting in different systems to actually meet.
The demonstrated approach works like this: documents and transaction records are ingested continuously rather than in scheduled batches. An ontology-guided extraction process identifies the entities that matter for this domain - vendors, people, accounts, addresses, and contracts - along with the relationships between them, such as “paid to,” “has account,” “signed,” and “located at.” Multiple embedding passes index the content for semantic search, and the extracted entities and relationships are merged into a live graph that updates automatically as new source documents arrive.
The moment this becomes valuable is worth walking through directly. An investigator asks, “Is this vendor connected to anyone internal?” At that point, nothing looks unusual - no flags. A new vendor contract is then uploaded to a document repository. In the background, the pipeline runs automatically: text is extracted, split into chunks, entities are pulled out, embeddings are generated, and the graph is updated - no one re-runs anything by hand. The investigator asks the exact same question again, and this time the answer is different, with cited sources: the graph reveals that the account referenced in the new contract already links to an internal employee and to another vendor entirely. A ring that was invisible a moment earlier is now explicit, because the connection existed in the data all along - it just hadn't been made queryable.
Patterns like a shared bank account or address between an employee and a vendor, or a single person signing on behalf of multiple vendors, are naturally graph patterns. They're difficult to spot in an isolated spreadsheet or a single document, but they're exactly the kind of multi-hop connection a graph traversal surfaces directly.
This also illustrates why hybrid retrieval matters in practice. Three distinct query modes are each good at different things: structured queries over transaction data are best for aggregates, totals, and rankings; search over document text is best for finding passages by meaning or exact name; and graph traversal is best for multi-hop connections with cited sources. None of them alone sees the whole picture - a question like “is this vendor connected to anyone internal” needs all three combined, with a reasoning layer picking the right mode for each sub-question and citing the document that backs up the answer.
Practical Takeaways
A few conclusions are worth carrying into any real evaluation of graph technology:
- Treat graphs as complementary to similarity search, not a replacement for it: The strongest designs combine both rather than picking one.
- Graph construction is the hard part, and it's where the real investment belongs: Entity extraction, identity resolution (recognizing that “John Smith” and “J. Smith” are the same person), and human review of low-confidence extractions all require real iteration. Underinvesting here produces a graph full of noise and duplicate entities.
- Schema and ontology decisions made early shape everything downstream: Deciding which entity and relationship types matter is a design decision that should come before pipeline-building, not after.
- Avoid locking into a single engine or a single modeling style: Different teams and use cases legitimately want different tools - some need fast application-specific traversal, others need formal reasoning and semantic governance.
- Start with a narrow, high-value flow: A pilot doesn't need to cover an entire enterprise graph to prove the pattern - a focused scenario at a modest scale is enough to validate the approach before scaling further.
Where This Leaves Us
As AI platforms shift from retrieving information to reasoning over it, the connections between data points start to matter as much as the data points themselves. A knowledge graph is simply the infrastructure that makes those connections explicit, queryable, and explainable instead of leaving them buried in disconnected systems or embedded implicitly in vector space.
If you're evaluating where this might apply in your own organization, a useful starting question is this: "where does information about the same entity - a customer, a product, a vendor, a case - currently live in two or more systems that don't talk to each other?" That's usually the clearest signal of where a connected knowledge layer would pay off first.
As AI deployments evolve to agents that need to query, reason, and act, graphs are becoming a key part of context engineering. When designed correctly, the use of graphs not only accelerates the agent's ReAct loop, but also leads to less reasoning and accurate actions. Our recent AIDP advancements with semantic layer and knowledge graphs enable this by providing agents with graph-based knowledge and retrieval that is best matched to underlying data, agent queries and enterprise use cases.