Ever connected ChatGPT to your internal documents and databases, only to find it fell well short of what you expected once you actually used it? This article breaks down why that disappointment happens, and what has to change to move past it, from a practitioner’s point of view.
Between 2023 and 2024, a huge number of companies followed almost the same script. They uploaded or connected company documents to ChatGPT or an internal LLM, kicked off a pilot expecting “now we can ask it anything about our company’s data,” and the early demo looked pretty convincing. Then the moment it went into real use, strange answers, vague answers, and plausible-but-wrong answers started showing up — and the project often stalled out at “an occasionally useful search box.”
Why the Demo Succeeds and the Real Thing Fails
There’s a reason this pattern repeats. The questions used in a demo are usually questions the builder anticipated in advance. They pick a handful of documents and tables and demo the questions those sources answer well. Real practitioners, meanwhile, ask questions outside that range. Something like “which customers who churned this month were also in the top 20% of revenue last quarter” isn’t something you can answer by searching one or two documents — it requires connecting information scattered across multiple tables under exact rules.
The first approach most companies reach for here is RAG (Retrieval-Augmented Generation): index your documents or tables into something searchable, and when a question comes in, pull the pieces that look relevant and feed them to the LLM alongside the question. It’s a solid fit for document-based Q&A — especially strong on questions where the answer is written down verbatim, like company policy or a manual.
The Trouble Starts the Moment You Move to Tables
The trouble starts once you cross over into structured data — tables. Internal database columns are usually abbreviated: TB_CUSTOMER, CUST_ID, ORD_STAT_CD. A person seeing these for the first time can only guess what they mean, and the LLM does the exact same guessing. It might get lucky in a demo, but a real practitioner notices immediately when the result is wrong.
The bigger problem is the relationships between tables. The single concept of “customer” might be called CUSTOMER in the CRM, CLIENT in the ERP, and LEAD in the marketing system — and it’s common for the key format linking the three to differ across systems too. RAG has to “rediscover” these relationships every single time a question comes in. A question that got the right answer yesterday might get retrieved through a different path today and produce a different answer. It only takes a few instances of this inconsistency to lose a practitioner’s trust for good.
Why “a Bigger Model” Doesn’t Fix It
At this point, the common reaction is “maybe the model just isn’t good enough.” So teams swap in a bigger model, tune the prompt more carefully, stuff in a pile of examples. It improves things somewhat. But the root cause isn’t the model’s reasoning ability — it’s that no information anywhere defines what these columns and tables actually mean, or how they connect to each other. No matter how much bigger the model gets, it can’t invent information that doesn’t exist. It fills that gap with a plausible-sounding guess, and that guess is exactly what practitioners flag as a wrong answer.
Many teams end up working around this by building a dedicated query for every question type. They collect common question patterns into sub-queries and add orchestration logic to manage them. The first few work fine, but as the number of question types grows, so does the pile of queries and exception rules to maintain — until it reaches a point where nobody can keep up with it.
So What’s a Different Approach?
Flip the structure of the problem around, and the direction becomes clear. Instead of making the system “guess” at meaning every time a question comes in, define the meaning before you ever start querying the data. Translate technical names like TB_CUSTOMER and CUST_ID into business terms — “Customer,” “Customer ID” — ahead of time, and pre-link the relationships between tables — “a customer places an order,” “an order leads to a shipment” — in a form people actually understand. There are several ways to build this kind of semantic structure — ontologies, knowledge graphs, data catalogs — and this blog mostly approaches it from the Ontology angle.
Once that semantic structure exists ahead of time, AI stops guessing at column names and instead follows already-defined concepts and relationships to find its answer. Consistency goes up because the same question follows the same path every time, and if that answer is connected to an Action (approve, execute, etc.), it can go beyond just answering in text and actually flow through to real work getting done.
This Doesn’t Mean Throw Out RAG
One thing worth being clear about: none of this means RAG is wrong. For Q&A where the answer already sits verbatim inside a document — policy, manuals, contracts — RAG is still fast and efficient, with a low upfront cost. The trouble only starts when teams try to make RAG alone solve everything in territory that requires handling company data accurately, repeatedly, and all the way through to execution. The two approaches aren’t competitors — they’re tools you reach for depending on the nature of the question.
Same Question, With a Semantic Structure in Place
Take the question again: “Among customers who churned this month, who was in the top 20% by revenue last quarter?” If an ontology-style semantic structure is already defined, it resolves like this:
- The Customer Object Type already defines churn status and churn date as properties. Whether the column is called
TB_CUSTOMER.CHURN_FLGorCRM_LEAD.IS_CHURNED, the ontology unifies it asCustomer.churned. - The Link Type between Customer and Order already encapsulates how they join (which columns, whether type conversion is needed), so the AI does not guess join keys each time.
- “Top 20% by revenue last quarter” aggregates along the Order relationship and computes a percentile, reusing aggregation logic already defined on the ontology.
Ask the question two or three times and you get the same answer by the same path. RAG invents a plausible search path each time; an ontology provides a fixed path of verified relationships and calculations.
RAG Alone vs. Ontology-Based, for Structured Queries
| Situation | RAG alone | Ontology-based |
|---|---|---|
| Column names and terms differ per system | Exposed as-is, causing confusion | Unified as Object property names |
| Multiple tables must be joined | Inferred anew each time (risk of wrong join keys) | Join method already defined on the Link Type |
| Same question repeated | Answers can vary | Same path, same answer |
| Aggregation or calculation needed | Approximated through text generation | Computed exactly from real data |
| Simple document-based questions | Effective | May be overengineering |
Six steps to validate a demo against real questions
- Beyond demo questions, have you collected 30–50 questions the business actually asks?
- Have you classified them into answerable from one table versus needing joins and calculation across tables?
- For tables with abbreviated column names, is there a meaning glossary?
- Do you have a table of terms and join-key formats that differ across systems?
- Did you ask the same question more than once to check answer consistency?
- Do questions that find answers in documents and questions that compute numbers go through different paths?
Quick answers
Q. The demo worked, so why does it fail in practice?
Demo questions are usually simple lookups answerable from one table. Questions that join several tables and add calculation are hard to answer consistently with simple retrieval.
Q. Does RAG fix this?
It works well for questions answered in documents but not for aggregation and joins over tables. Structured data needs a path that queries and computes. See How to Serve Internal Database Data to an LLM for the distinction.
Q. What should we do first?
Build an evaluation set of real questions. If you do not know what is wrong, you cannot confirm that any improvement works.
What to Look at Next
In short: if ChatGPT connected to your company data felt useless, it’s rarely because of the model’s limits — it’s usually because the data had no semantic structure behind it. To dig deeper into this structural cause, Why Enterprise AI Stalls at the Chatbot Stage compares RAG and Ontology-based approaches side by side. And if “what exactly is an Ontology” is still an open question, start with What Is Palantir Ontology? A Practitioner’s Explanation.