Ever connected ChatGPT to your internal documents and databases, only to find it fell well short of what you expected once you actually used it? This article breaks down why that disappointment happens, and what has to change to move past it, from a practitioner’s point of view.
Between 2023 and 2024, a huge number of companies followed almost the same script. They uploaded or connected company documents to ChatGPT or an internal LLM, kicked off a pilot expecting “now we can ask it anything about our company’s data,” and the early demo looked pretty convincing. Then the moment it went into real use, strange answers, vague answers, and plausible-but-wrong answers started showing up — and the project often stalled out at “an occasionally useful search box.”
Why the Demo Succeeds and the Real Thing Fails
There’s a reason this pattern repeats. The questions used in a demo are usually questions the builder anticipated in advance. They pick a handful of documents and tables and demo the questions those sources answer well. Real practitioners, meanwhile, ask questions outside that range. Something like “which customers who churned this month were also in the top 20% of revenue last quarter” isn’t something you can answer by searching one or two documents — it requires connecting information scattered across multiple tables under exact rules.
The first approach most companies reach for here is RAG (Retrieval-Augmented Generation): index your documents or tables into something searchable, and when a question comes in, pull the pieces that look relevant and feed them to the LLM alongside the question. It’s a solid fit for document-based Q&A — especially strong on questions where the answer is written down verbatim, like company policy or a manual.
The Trouble Starts the Moment You Move to Tables
The trouble starts once you cross over into structured data — tables. Internal database columns are usually abbreviated: TB_CUSTOMER, CUST_ID, ORD_STAT_CD. A person seeing these for the first time can only guess what they mean, and the LLM does the exact same guessing. It might get lucky in a demo, but a real practitioner notices immediately when the result is wrong.
The bigger problem is the relationships between tables. The single concept of “customer” might be called CUSTOMER in the CRM, CLIENT in the ERP, and LEAD in the marketing system — and it’s common for the key format linking the three to differ across systems too. RAG has to “rediscover” these relationships every single time a question comes in. A question that got the right answer yesterday might get retrieved through a different path today and produce a different answer. It only takes a few instances of this inconsistency to lose a practitioner’s trust for good.
Why “a Bigger Model” Doesn’t Fix It
At this point, the common reaction is “maybe the model just isn’t good enough.” So teams swap in a bigger model, tune the prompt more carefully, stuff in a pile of examples. It improves things somewhat. But the root cause isn’t the model’s reasoning ability — it’s that no information anywhere defines what these columns and tables actually mean, or how they connect to each other. No matter how much bigger the model gets, it can’t invent information that doesn’t exist. It fills that gap with a plausible-sounding guess, and that guess is exactly what practitioners flag as a wrong answer.
Many teams end up working around this by building a dedicated query for every question type. They collect common question patterns into sub-queries and add orchestration logic to manage them. The first few work fine, but as the number of question types grows, so does the pile of queries and exception rules to maintain — until it reaches a point where nobody can keep up with it.
So What’s a Different Approach?
Flip the structure of the problem around, and the direction becomes clear. Instead of making the system “guess” at meaning every time a question comes in, define the meaning before you ever start querying the data. Translate technical names like TB_CUSTOMER and CUST_ID into business terms — “Customer,” “Customer ID” — ahead of time, and pre-link the relationships between tables — “a customer places an order,” “an order leads to a shipment” — in a form people actually understand. There are several ways to build this kind of semantic structure — ontologies, knowledge graphs, data catalogs — and this blog mostly approaches it from the Ontology angle.
Once that semantic structure exists ahead of time, AI stops guessing at column names and instead follows already-defined concepts and relationships to find its answer. Consistency goes up because the same question follows the same path every time, and if that answer is connected to an Action (approve, execute, etc.), it can go beyond just answering in text and actually flow through to real work getting done.
This Doesn’t Mean Throw Out RAG
One thing worth being clear about: none of this means RAG is wrong. For Q&A where the answer already sits verbatim inside a document — policy, manuals, contracts — RAG is still fast and efficient, with a low upfront cost. The trouble only starts when teams try to make RAG alone solve everything in territory that requires handling company data accurately, repeatedly, and all the way through to execution. The two approaches aren’t competitors — they’re tools you reach for depending on the nature of the question.
What to Look at Next
In short: if ChatGPT connected to your company data felt useless, it’s rarely because of the model’s limits — it’s usually because the data had no semantic structure behind it. To dig deeper into this structural cause, Why Enterprise AI Stalls at the Chatbot Stage compares RAG and Ontology-based approaches side by side. And if “what exactly is an Ontology” is still an open question, start with What Is Palantir Ontology? A Practitioner’s Explanation.