Should you fine-tune an AI model on your company documents, or use RAG?
By Validure team · · 14 min read
You want an assistant that knows your policies, price lists and product sheets. Fine-tuning sounds like the way to teach it, but the research and the providers' own guidance point the other way for facts. What each method actually changes, a worked cost and upkeep comparison, a decision table, and the 50-question test to run before choosing.

Say a 60-person distributor has about 400 documents that staff keep asking each other about: the leave policy, the credit terms for each dealer category, product sheets, the price list that changes every month. Someone proposes an internal assistant. Then the argument starts. One person says "we should train the model on our documents so it knows them". Another says "no, we should use RAG". Neither can say exactly what the difference would be in six months.
Here's the short answer. If the problem is that the model doesn't know your facts, use retrieval (RAG). Fine-tuning is the wrong tool for teaching a model facts, and the research on this is unusually clear. Fine-tuning earns its place for a different problem: getting a model to behave in a consistent way, in a fixed format or tone, cheaply and at volume.
The rest of this post shows why, with numbers you can redo for your own case, and a test that settles it on your own documents rather than on anyone's opinion, including ours.
What each one actually changes
Both methods start from a model someone else trained. They differ in where your information ends up.
Fine-tuning changes the model itself. You give it hundreds or thousands of examples (a question and the answer you want, or an input and the output you want), and the training nudges the model's internal weights so it produces answers like those. Your information ends up spread across the weights. You can't point to where a fact lives, and you can't remove one fact without training again.
Retrieval leaves the model alone. When someone asks a question, a search step finds the few passages from your documents that look relevant and pastes them into the prompt, with an instruction like "answer from these passages". The facts stay in your documents, in a search index you control. Change a document, rebuild the index for that file, and the next answer uses the new version.
OpenAI's own guide on improving accuracy puts it in terms of memory. Retrieval is like having the textbook open on the desk during an exam. Fine-tuning is like having attended the classes: it shapes how the model works, and that guide treats it as a fix for behaviour problems, where the model already has the knowledge it needs.
| Fine-tuning | Retrieval (RAG) | |
|---|---|---|
| Where your information lives | Inside the model's weights | In your documents and a search index |
| Updating one fact | Prepare new examples, train again, test again | Replace the file, re-index it |
| Can it show where an answer came from | No | Yes, it can cite the passage |
| Different staff see different documents | Hard: whatever is in the weights is available to every user | Filter the search by the user's permissions |
| Removing one person's data | Train again without it | Delete the document and its index entries |
| What it is good at | Format, tone, a narrow repeated task | Facts, especially facts that change |
What the research says about teaching facts by fine-tuning
This isn't just a rule of thumb. Two peer-reviewed studies from 2024 tested it directly.
Researchers at Microsoft (Ovadia and others, published at EMNLP 2024) compared fine-tuning and retrieval for injecting knowledge into several models across different subjects, including facts the models had never seen. Retrieval consistently did better, both for facts the models had partly seen in training and for entirely new ones. Fine-tuning helped a little, and helped more when the training data repeated the same fact in many different phrasings. Combining fine-tuning with retrieval did not beat retrieval alone in their tests.
A team at Google (Gekhman and others, also EMNLP 2024) went further. They varied how much of the fine-tuning data contained facts the model didn't already know. Those new facts were learned much more slowly than familiar ones, and as the model did learn them, its tendency to give wrong answers about things it previously knew went up, roughly in proportion. Their reading is that models get their factual knowledge mostly in the original training, and fine-tuning mainly teaches them to use it.
Put those together and you get the practical point. Fine-tune a model on your leave policy and it may learn to sound like your HR team while getting the number of days wrong, and you've also made it slightly less reliable on everything else.
Two honest caveats. These studies test question answering on public-style facts, not your exact documents. And the field moves quickly, so a later method may narrow the gap. That's why this post ends with a test you run yourself.
The upkeep is where fine-tuning really loses
Suppose you ignore the research and fine-tune anyway. The bigger cost isn't the first training run. It's every month after.
Take the example distributor again. Of its 400 documents, about 20 change in a typical month: the price list, a revised credit policy, a few new product sheets. The numbers below are illustrations; your rate of change is the figure to plug in.
With fine-tuning, the research above suggests each fact needs several phrasings to stick. Say you write 5 question-and-answer pairs per changed document, in 3 phrasings each. That's 20 x 5 x 3 = 300 training examples a month. If someone spends one minute checking each, that's 300 minutes, or 5 hours. Add a test run of 50 questions at about 2 minutes each to mark, another 100 minutes, and you're near 6 hours 40 minutes a month, plus the training run itself.
With retrieval, you replace the 20 files and re-index them, which takes minutes, and run the same 50-question test: about 100 minutes, so a little under 2 hours.
Then there's staleness. Say the price list changes on 3 November and you retrain on the 1st of each month. From 3 November to 30 November, 28 days, the fine-tuned model quotes last month's prices with full confidence. A retrieval setup that re-indexes every night is wrong for a day at most, and if you re-index when the file is saved, not even that.
What it costs per month to run
Running costs are usually small for an internal assistant, and they tilt slightly against retrieval because every question carries the pasted passages. It's worth seeing how small.
The prices below are example rates to make the arithmetic visible, not any provider's list price. Check your provider's price page.
Example: 1,500 questions a month. Each question sends about 4,000 tokens of retrieved passages plus 300 tokens of question and instructions, and gets a 300-token answer.
| Item | Working | Tokens | Example rate | Cost |
|---|---|---|---|---|
| Input | 1,500 x 4,300 | 6,450,000 | ₹25 per million | ₹161.25 |
| Output | 1,500 x 300 | 450,000 | ₹100 per million | ₹45.00 |
| Total | ₹206.25 |
At that volume the model bill is a rounding error next to the 2 hours of someone's time each month. So "retrieval costs more per question" is true and mostly irrelevant for a company this size.
Where fine-tuning does pay: shorter prompts at high volume
Now a different job. Say the same company classifies incoming dealer emails into eight categories (order, complaint, payment query and so on) and extracts a few fields, about 30,000 emails a month. To get consistent output, the prompt carries 2,500 tokens of instructions and worked examples, plus about 500 tokens for the email itself.
A fine-tuned model can learn the instructions and examples from training data, so each call only needs the email. On some providers a fine-tuned model is billed at a higher rate per token than the base model, so the example below doubles the rate.
The formula for the monthly saving on input tokens is:
saving = calls x (base tokens x base rate - tuned tokens x tuned rate) / 1,000,000
Worked through with example rates of ₹25 per million for the base model and ₹50 for the fine-tuned one:
| Base model with long prompt | Fine-tuned model | |
|---|---|---|
| Input tokens per call | 3,000 | 500 |
| Tokens per month (30,000 calls) | 90,000,000 | 15,000,000 |
| Example rate per million | ₹25 | ₹50 |
| Monthly input cost | ₹2,250 | ₹750 |
Check with the formula: 30,000 x (3,000 x 25 - 500 x 50) / 1,000,000 = 30,000 x 50,000 / 1,000,000 = ₹1,500 a month, the same as 2,250 minus 750. Output tokens are the same either way, so they drop out.
₹1,500 a month doesn't pay for anyone's time building and maintaining a training set. At 300,000 emails a month it's ₹15,000, which starts to. The more common reason to fine-tune here is consistency: a model that returns exactly your eight categories, in your exact field format, every time, after prompting has stopped improving. Note that nothing in this job asks the model to know a changing fact. That's the pattern.
Fine-tuning is also more practical than it used to be. The LoRA method (Hu and others, 2021) trains a small add-on instead of the whole model; on GPT-3 175B the authors reported 10,000 times fewer trainable parameters and a third of the GPU memory compared with full fine-tuning. That makes it cheaper to try, but it doesn't change what fine-tuning is good at.
A decision table
| Your problem | Use | Why |
|---|---|---|
| The model doesn't know our policies, prices or products | Retrieval | Facts stay in documents you control and can update |
| Our facts change monthly or faster | Retrieval | Re-index a file instead of training again |
| Staff need to see which document an answer came from | Retrieval | It can cite the passage it used |
| Some documents are for managers only | Retrieval with permission filters | A fine-tuned model can't hide part of what it learned from some users |
| Answers are right but the format or tone wanders | Better prompts first, then fine-tuning | This is behaviour, which is what fine-tuning changes |
| A narrow, repeated task at high volume with a long prompt | Fine-tuning, often of a smaller model | Shorter prompts per call, as worked above |
| The model misreads our trade terms and abbreviations | A glossary in the prompt first | Cheap to try; fine-tune only if it still fails |
| We have under about 50 good examples | Neither fine-tuning nor a decision yet | OpenAI's guide suggests starting with 50 or more quality examples and an evaluation set |
Most internal "knows our documents" assistants land in the first four rows. That's why our default advice is retrieval.
Run the 50-question test before you spend anything
Arguments about methods end quickly once you have a test set. Write 50 questions that staff actually ask, each with the correct answer and the document it comes from. Have someone who knows the business write them, not the person building the system. OpenAI's guide makes the same point: an evaluation set of questions with known answers is the baseline everything else gets measured against.

Include a few awkward ones on purpose:
| Check | What to look for | What a hit means |
|---|---|---|
| Right answer, right source | The answer is correct and names the document it came from | Working as intended |
| Something that changed last month | The old figure comes back | The index wasn't refreshed, or the facts live in a model that needs retraining |
| A question your documents don't answer | It answers anyway, confidently | It needs an instruction and a test for "not in our documents" |
| A question a junior shouldn't get answered | It answers from a manager-only file | Permissions aren't applied at the search step |
| The same question asked three ways | Three different answers | Retrieval finds different passages for each phrasing, or the prompt is loose |
Score it simply: correct or not, marked by a person. Say the first version gets 41 of 50 right, 82 per cent. You change the chunking and it gets 45, 90 per cent. With 50 questions each one is worth 2 points, so don't celebrate a change of one or two questions; that could be noise. A change of four or more, held across two runs, is worth taking seriously.
Re-run the same 50 after every change: new documents, a new model version, a new prompt. If you later try fine-tuning for format, the same set tells you whether it helped or quietly broke the facts, which is the risk the Google study describes.

Privacy and erasure tip the balance further
India's Digital Personal Data Protection Act, 2023 gives people a right to ask for correction and erasure of their personal data (section 12), subject to retention the law requires. Most of the Act's operational duties are being brought into force in phases under the notification of November 2025, with the main ones due around May 2027, so check which apply to you and when.
If an employee letter or a customer record went into a fine-tuning set, honouring an erasure request means training again without it, and proving it's gone is hard. With retrieval, you delete the file and its index entries. That's one more reason to keep personal data out of training sets altogether. This is information, not legal advice; take your own case to a lawyer.
When you need neither
Under about 30 short documents, you may not need retrieval either. Many models now accept long prompts, so you can paste the relevant policy straight in, or keep a well-organised shared folder with a search box. If the same 15 questions come up every week, a written FAQ page beats any model on cost and accuracy. Build the assistant when people are actually spending hours hunting through documents, not because the technology exists.
Where we come in
We build software and use these models ourselves, so we have an interest. The test above costs you nothing and you can run it without us. Where we help is the plumbing around it: wiring retrieval into an existing system with permission filters, keeping the index in step with the documents, and setting up the evaluation run, which falls under our AI and intelligent solutions and data and analytics work. For a first try over a few dozen documents, you probably don't need anyone.
Sources and checks
Models, prices and methods change quickly. These are the primary sources the claims above rest on; check current versions before you build.
- Ovadia and others, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, EMNLP 2024 (also arXiv 2312.05934).
- Gekhman and others, Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, EMNLP 2024.
- Hu and others, LoRA: Low-Rank Adaptation of Large Language Models, 2021.
- Lewis and others, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
- OpenAI, Optimizing LLM accuracy, for the memory framing, the 50-example starting point and the evaluation-set advice.
- Digital Personal Data Protection Act, 2023 and the DPDP Rules, 2025, from the Ministry of Electronics and Information Technology. Commencement dates checked 18 November 2026.
All prices and volumes in the worked examples are illustrations, not quotes.
Frequently asked questions
Can I fine-tune a model so it knows our company policies?
You can try, but it's the weaker option. Studies published in 2024 found retrieval beat fine-tuning for adding facts, and that fine-tuning on new facts made models more likely to get other things wrong. Use retrieval for facts.
What is fine-tuning actually good for?
Behaviour: a fixed output format, a consistent tone, or a narrow task like classifying emails done at high volume, where a fine-tuned model lets you drop long instructions from every prompt.
Is RAG more expensive to run than a fine-tuned model?
Per question, slightly, because it sends retrieved passages each time. In the example above, 1,500 questions a month came to about ₹206 at example rates. The staff time to keep a fine-tuned model current usually costs far more.
Can I combine fine-tuning and RAG?
Yes, for different jobs: retrieval for the facts, fine-tuning for the format. Don't expect fine-tuning to improve factual accuracy on top of retrieval; in the Microsoft study the combination didn't beat retrieval alone.
How many examples do I need to fine-tune?
OpenAI's guide suggests starting with 50 or more good-quality examples, testing, then adding more if needed. Quality matters more than count.
How do I know which approach works for our documents?
Write 50 real staff questions with known answers and sources, and score each setup against them. Re-run the same set after every change.
