Prompting, RAG, or Fine-Tuning? How to Choose Based on the Failure You See

Summary — The three are not a ranking from weak to strong. They are tools that fix different kinds of failure. Diagnose whether the model does not know something (knowledge), does not follow something (instructions), or has not internalized something (behavior and format), and the choice almost makes itself. This post follows a hypothetical team building an internal AI assistant through three roadblocks, showing what they looked at and what they chose at each step. (The scenario is a generalized, fictional example.)

0. The conclusion first: a one-page summary

The default order is prompting → RAG → fine-tuning, because cost, time, and difficulty of reversing all grow in that order. The story below shows why that order makes sense.

1. The story begins — “an assistant that answers questions about internal policy”

A hypothetical team wants to build an AI assistant that answers employees’ questions about internal policies and procedures. They have three goals:

1. Accurate answers grounded in the policies
2. A consistent format (conclusion → evidence → responsible department)
3. Fast updates when policies change

The first job is not picking a model or collecting data. It is building 30–50 evaluation questions to record “what went wrong.” Every later decision is judged by the failure types in that set. Without it, you cannot even tell which of the three methods helped.

2. Act 1 — How far can prompting go?

The team starts with the cheapest method: a system prompt with a role, an answer format, a rule to say “I don’t know” when unsure, and two or three few-shot examples.

What prompting fixed – The uneven format (now fixed as conclusion → evidence → responsible department) – Wordy answers and inconsistent tone – Overconfident statements without evidence (now says “needs confirmation” when unsure)

What prompting did not fix – “What is the leave carry-over policy?” → a plausible answer with the wrong number

This is the key distinction. Format and attitude problems are solved by prompts, but facts the model never learned cannot be created by a prompt. No matter how carefully you write instructions, the internal policy is not inside the model.

Decision criterion ①: If you paste the correct information directly into the prompt, does the model answer correctly? – Yes → the model is capable and you need a way to supply the information → RAG – Still wrong → an instruction, reasoning, or format problem → improve the prompt or consider fine-tuning

This “paste it in” experiment takes five minutes and almost settles whether to go to RAG.

3. Act 2 — RAG: find what the model does not know and show it

Because pasting worked, the team moves to RAG: split and index the policy documents, and when a question arrives, retrieve the relevant pieces and include them in the prompt. (For how to build it, see RAG pipeline: chunking, hybrid search, reranking and RAG with Langflow.)

What RAG fixed – Answers grounded in internal policy – Updating a document is enough when a policy changes (no retraining) – Citations are possible → easier to verify and audit – Access control by document

Signals to choose RAG – The answer lives in knowledge that changes often or is private – Answers need citations – The knowledge is too large to fit in a prompt

But a new kind of failure appears. Running the evaluation questions, failures split into two.

Decision criterion ②: When RAG is wrong, look at retrieval and generation separately. Many teams skip this and conclude “RAG is not accurate.”

• Retrieval is the problem → adjust chunk size and overlap, add keyword + vector hybrid search, then reranking. Fine-tuning comes only after all of these.
• Generation is the problem → how the context is presented and how the model is told to answer — that is a prompt problem again.

4. Act 3 — Is fine-tuning really needed?

Suppose two problems remain after repeated improvements.

(A) Failure to retrieve internal terminology: internal abbreviations and nicknames cause a general-purpose embedding model to miss relevant documents. → Here, fine-tuning targets the retrieval model (embedding model, reranker), not the answering model. How to build a dataset from synthetic queries and hard negatives is covered in RAG fine-tuning: synthetic queries and hard negatives.

(B) A behavior or judgment problem: even with 20 rules in the prompt, the answer format and refusal criteria drift, and the long prompt raises cost and latency. → This is the territory of SFT (supervised fine-tuning). It does not inject knowledge; it makes behavior stick.

Signals to choose fine-tuning – The same type of failure keeps recurring after thorough prompt and RAG improvements – You can build hundreds to thousands of input/output examples of good quality – The requirement is behavior: style, format, classification criteria, tool-calling patterns – Call volume is high enough that cutting long-prompt cost and latency pays off – You want the same quality from a smaller model (distillation)

Signals NOT to fine-tune – The goal is “make it memorize the latest knowledge or internal documents” → RAG is better for knowledge. Fine-tuning injects knowledge unreliably, requires retraining to update, and cannot cite sources. – There is no evaluation set → you cannot tell whether it improved – Data is scarce or low quality → performance can actually get worse (risk of forgetting existing abilities) – Policies change often → retraining every time

Decision criterion ③: If what is missing is facts, use RAG; if it is behavior, use fine-tuning. Facts change, but behavior is relatively stable, so keep what changes outside (RAG) and put only what does not change inside (the model).

5. Looking again from an operations view — cost, latency, upkeep

Choices are not made on accuracy alone. At equal accuracy, the option that is cheaper, faster, and easier to fix wins.

Prompting – Cost is incurred per call as input tokens. The longer the rules and examples, the more repeated cost every call carries. – Changes take effect immediately and are easy to reverse. But as rules multiply they conflict, and it becomes hard to trace which rule helped.

RAG – Each call adds retrieval latency and context tokens. You must operate retrieval infrastructure (index, embedding refresh, document ingestion). – Most of the cost is not the model but data quality management. Stale, duplicated, or differently permissioned documents make answers unstable. Without a document owner and refresh schedule, RAG gets worse over time.

Fine-tuning – The cost of building the data exceeds the training cost. Good example answers must be created with human review. – When the base model changes, you retrain and re-evaluate. It is not a one-time job; the decision creates something to maintain. – In return, per-call cost can fall. Internalizing a long prompt into the model cuts input tokens, and a smaller model may do the same job. That is why it pays off for very high-volume services.

Criterion: fine-tuning is chosen more often because “a repeated behavior must be cheap and stable” than because “we need more accuracy.”

6. Practice cases — from symptom to choice

Try judging these yourself. Each case applies criteria ①–③ above.

The trick for working through cases is to first write “what is wrong” in one sentence. Not “accuracy is low” but “does not know the amended clause,” “format drifts,” or “the right document is not returned” — and the tool picks itself.

7. In practice, the answer is a combination

Real services rarely use only one. Common combinations:

1. Prompting only — transformation, summarization, classification that needs no knowledge. Try first.
2. Prompting + RAG — the default for internal-knowledge Q&A. Most services are fine here.
3. Prompting + RAG + retrieval-model fine-tuning — domains with heavy specialized terminology where retrieval keeps missing.
4. Prompting + RAG + SFT — knowledge via RAG, answer style and judgment criteria fixed by fine-tuning. High-volume, high-quality-bar services.
5. Agent + multiple sources — when you need DBs and APIs, not just documents. We cover that structure in the next post.

The principle when combining is keep what changes outside, and what does not change inside. Facts that change — policies, prices, product information — live in RAG (outside); stable behavior — answer format, judgment criteria, how to use specialized terms — lives in the prompt or the model (inside).

8. Decision flow (order of questions)

1. Do you have an evaluation set? If not, build it first (30–50 questions is enough to start).
2. If you paste the correct information into the prompt, does it answer correctly?
3. No → recheck prompt structure and model choice. If that still fails, consider fine-tuning
4. Yes → go to 3
5. Is the needed information outside the model (internal, latest, private)? → Yes: RAG (for unstructured data such as documents and images; numbers and aggregates in databases are covered in How to Serve Internal Database Data to an LLM)
6. Have you separated RAG failures into retrieval and generation?
7. Retrieval problem → chunking, hybrid search, reranking → if that fails, fine-tune the embedding model / reranker
8. Generation problem → improve the prompt
9. After improving prompt and RAG, do format, behavior, or judgment failures keep recurring? And do you have hundreds of high-quality examples? → SFT fine-tuning
10. A combination is the realistic answer. Most services are prompting + RAG, and some add a retrieval model or SFT fine-tuning.

9. Side-by-side comparison

10. Three common misconceptions

1. “Fine-tuning makes it memorize our documents” — It is an inefficient and imprecise way to inject knowledge. For document knowledge, RAG is the default.
2. “With RAG there are no hallucinations” — If retrieval is wrong or the model ignores the context, it is still wrong. Evaluate whether answers are grounded in the documents.
3. “Long-context models make RAG unnecessary” — If documents are few and stable, including them all can be the simplest option (an extension of the paste-it-in test). But as document volume, cost, latency, access control, and citation needs grow, RAG becomes necessary again.

11. Checklist before you start

• [ ] You have 30–50 evaluation questions with answer criteria
• [ ] You tried the experiment of pasting the correct information into the prompt
• [ ] You classified failures as “knowledge / retrieval / instruction / behavior”
• [ ] You organized the prompt first (role, format, examples, uncertainty rules)
• [ ] You measure RAG failures separately for retrieval and generation
• [ ] Before fine-tuning, you decided data quality and volume, evaluation method, and retraining plan
• [ ] You compare before and after on the same evaluation set

12. Wrap-up

• The starting point of the choice is not the method but diagnosing the failure symptom.
• If facts are missing, RAG; if behavior, fine-tuning; if instructions, prompting.
• Order: cheap and easy to reverse first — prompting → RAG → (retrieval model / SFT) fine-tuning.
• Fine-tuning is not a last resort but a tool you choose once evidence has built up. The evaluation set provides that evidence.

Next: how to serve internal database data to an LLM — when the question is about numbers and aggregates, not documents. After that: how to build an evaluation set — selecting questions, defining answer criteria, and measuring retrieval and generation separately.

The criteria here are based on generally published practical guidance and the author’s synthesis. Models, tools, and cost structures change quickly, so verify against the latest official documentation and your own experiments before adopting.