“Isn’t RAG enough? Do we really need an Ontology on top of it?” We get this question a lot. This article tries to answer it honestly. It isn’t here to trash RAG and sell you an Ontology — it’s here to draw the line between what each one can actually do and where it stops working.
What RAG Does Well
RAG (Retrieval-Augmented Generation) is strong on questions where the answer already sits inside a document — policies, manuals, contracts, meeting notes: anything you can “find and summarize.” It’s also cheap to build. Load your documents into a vector database, bolt on a retrieval pipeline, and you can have a demo running in days.
Because of that, RAG genuinely works well for internal wiki search, assisting customer support responses, or helping new hires search onboarding docs. For questions like “what does this clause mean” or “what did we decide in last year’s meeting,” RAG alone is often enough.
Where RAG Falls Short
The trouble starts the moment a question stops being about “finding” and starts being about “connecting and computing.” Take “which customers in the top 20% of this quarter’s revenue also had rising support inquiries over the last three months” — that’s not a question you can answer by searching a single document. You need to join and aggregate a revenue table, a customer table, and a support-history table under exact, correct rules.
RAG keeps running into the same two problems on questions like this.
- Guessing at meaning: it infers what column names and values mean from context alone — there’s no defined dictionary behind it.
- Re-discovering relationships every time: because it re-derives the join path between tables from scratch each run, the same question can get answered via a different path — and a different result — each time you ask it.
Answers that shift subtly from run to run rarely show up during a demo. They surface the moment real practitioners start using it every day.
The Line Between Them
| Criterion | RAG | Ontology-based |
|---|---|---|
| Best-fit questions | Answer already lives in a document | Requires joining and computing across multiple tables |
| How meaning is defined | LLM infers it from context | Defined up front, in business terms |
| Consistency of answers | Can vary by execution path | Relatively stable — follows the same relationship structure every time |
| Executability | Provides an answer only; execution is a separate system | Can connect straight through to a defined Action |
| Upfront build cost | Low | Higher (needs design and maintenance) |
| Right-fit org size | Single department, simple document search | Organizations that continuously deal with many systems |
Where the Need for an Ontology Actually Kicks In
In practice, you can spot the point where an Ontology becomes necessary through three signals.
First, when the same question starts returning different answers. If a RAG-based chatbot gives a different rationale today than it gave yesterday, that’s a sign it’s re-discovering relationships from scratch every time.
Second, when the number of question-specific queries keeps growing. If you find yourself constantly adding exceptions — “this type of question uses this query, that type uses that one” — you’re hardcoding relationships case by case. Defining those relationships once, in an Ontology, resolves that scaling problem structurally.
Third, when the answer needs to lead to an action — approval, processing. If you need to go beyond “find the at-risk customers” to “automatically trigger a care action for the customers you found,” that’s the domain of Actions, not search, and it was never something RAG alone was designed to cover.
On the Other Hand: When an Ontology Is Overkill
An Ontology isn’t always the right call. If all you need is to pull a handful of reports on a regular schedule, or your data already lives cleanly inside a single system, the cost of designing and maintaining an Ontology can outweigh what it buys you. In those cases, RAG or your existing BI tools are enough on their own. Making that judgment clearly is the whole point of this article — an Ontology isn’t “always the better choice.” It’s “the choice that becomes necessary past a certain point.”
Testing the Signals with One Question
Ask “Why did revenue drop this quarter?” to a RAG chatbot and to an ontology-based agent, and the difference is clear.
- RAG chatbot: finds passages about “revenue decline” in manuals or past reports and summarizes the cause written there (for example, “temporary dip after the March promotion ended”). It works when the answer is documented, but if this quarter’s cause is a new pattern nobody wrote down, RAG cannot produce it.
- Ontology-based agent: walks the relationships from the revenue object to regions, products and customer segments, and computes “which products in which regions fell” from live data. The answer does not need to exist in any document.
That is exactly where the three signals show up: relationship-driven exploration, real-time calculation, and answers that come from data rather than documents.
Using RAG and an Ontology Together
In practice, using both is more common than choosing one. “Does the policy have an exception for this?” goes to RAG; “Analyze this customer’s last six months of orders” goes to the ontology. One agent can route each question to the right path.
Quick Decision Table
| Question type | Best fit |
|---|---|
| “What does the policy or manual say?” | RAG |
| “How are these two data sets actually connected?” | Ontology |
| “Aggregate as of right now” | Ontology |
| “Was there a similar case last year?” (document-based) | RAG |
Six questions to decide whether a RAG chatbot needs an ontology
- Are you seeing more cases where the same question gets different answers on different days?
- Are requests shifting from document search to joins and calculation across tables?
- Are the meanings of columns and values documented, so the AI does not have to guess?
- Are requirements emerging to turn answers into actions (Action)?
- Is reproducibility and consistency now more important than a low initial cost?
- Do you operate separately the areas where RAG is enough and the areas that need an ontology?
Quick answers
Q. Should we drop the RAG chatbot and switch to an ontology?
It is a division of roles, not a replacement. RAG handles document-search questions; a structure with defined meaning is more stable for questions that join and compute across tables. Many teams use both.
Q. What if the ontology build cost is a burden?
Rather than building it all at once, start small with concepts tied to the most frequent questions and expand.
Q. How should numbers in a database be served?
Not by embedding rows like documents, but through a path that queries and computes. How to Serve Internal Database Data to an LLM covers how to choose among tool calls, Text-to-SQL, and a semantic layer.
Further Reading
To see this judgment call play out against a real failure case, read Why Enterprise AI Stalls at the Chatbot Stage. For what an Ontology actually looks like once it’s implemented, see Palantir Foundry Architecture Overview: From Connect to AI in Five Layers. And if you want to start from “what is an Ontology, exactly,” read What Is Palantir Ontology? A Practitioner’s Explanation.