RAG or fine-tuning? A plain-language guide for business decisions

Qasim Ali · Senior Software Engineer (Java, AI), CS2 Technologies
Published · 6 min read

Sooner or later, every company building with generative AI hits the same wall. A general-purpose model writes fluent text, but it doesn’t know your products, your policies or last week’s price change. Someone on the team then suggests one of two fixes: “we should use RAG” or “we should fine-tune a model.”

Both are real techniques, and they solve different problems. Picking the wrong one usually means spending money on the harder path and still not getting the answers you wanted.

This guide explains both in plain language, shows where each one fits, and covers the questions about cost, upkeep and privacy that a manager should ask before approving either.

1. Retrieval-augmented generation, in one paragraph

Retrieval-augmented generation (RAG) leaves the model as it is and hands it the right material at the moment of each question. Your documents are split into passages and indexed so they can be searched by meaning, not only by keyword. When someone asks a question, the system finds the most relevant passages, passes them to the model along with the question, and asks it to answer from that material.

Think of it as an open-book exam. The model brings the language skills, and your documents bring the facts. Because the answer is built from specific passages, the system can show which document each part came from.

Our AI medical scribe, Notermed, works this way when it answers questions from a clinic’s own guidelines, forms and procedures.

For managers: RAG gives a model access to your information without changing the model itself.

2. Fine-tuning, in one paragraph

Fine-tuning changes the model. You take an existing model and train it further on a set of examples, usually pairs of “here’s the input” and “here’s the output we want.” After training, the model leans toward producing outputs that look like your examples, without being shown them each time.

If RAG is an open-book exam, fine-tuning is coaching. It shapes how the model writes and behaves. It’s a poor way to teach it facts that change, because whatever it learned is fixed at the moment training ended.

For managers: Fine-tuning changes how a model behaves. It isn’t a reliable way to keep it up to date on facts.

3. When retrieval is the better fit

Choose retrieval when the answers depend on facts that live in your documents and change over time: product catalogues, policies, contracts, support articles, internal procedures. Update a document, re-index it, and the next answer reflects the change. No retraining involved.

Retrieval is also the natural choice when people need to check the answer. In regulated work, or anywhere a wrong answer has consequences, being able to say “this came from section 4 of the 2026 policy” matters more than fluent prose.

And it handles permissions well. The search step can be limited to the documents a given user, team or customer is allowed to see. When we built document search into Notermed, each clinic’s search had to stay inside that clinic’s own documents, and the restriction had to apply inside the AI search itself, not only in the database.

For managers: If your question starts with “what does our documentation say about…”, you’re looking at a retrieval problem.

4. When fine-tuning earns its place

Fine-tuning makes sense when the problem is consistency of form rather than missing knowledge. A few examples: the output must follow a strict structure every time, the tone has to match a house style that’s hard to describe in instructions, or a narrow classification task needs to run at high volume on a smaller, cheaper model.

It can also make sense when you’ve already tried careful instructions with good examples and the model still drifts. That last condition is important. Modern models follow detailed instructions well, and a lot of what used to need fine-tuning can now be handled with a clear prompt and a few worked examples.

The two approaches also combine. A system can retrieve current facts and pass them to a model that was fine-tuned for format. That’s a later step for most companies, not a starting point.

For managers: Consider fine-tuning when you need the model to behave differently, and only after good prompts have been given a fair try.

5. What each one costs to build and to keep

Retrieval asks for upfront work on your documents: cleaning them, splitting them into sensible passages, choosing what gets indexed and testing whether the right passages come back. Running costs grow with the number of questions, since each answer sends extra text to the model. Upkeep means keeping the index in sync with your documents.

Fine-tuning asks for a set of high-quality examples, which someone has to write or curate, plus the training run and the hosting of a custom model. The cost people underestimate is upkeep. When your style guide changes or the provider retires the base model, the examples need revisiting and the training often has to be redone.

Retrieval has a version of this too. The searchable index depends on the model that created it, so changing that model means re-indexing your documents. We did exactly that when Notermed moved to Amazon Bedrock, in a fresh store with the old one kept for rollback. It’s manageable, but it should be in the plan.

For managers: Ask for the cost of year two, not only the build. That’s where the two approaches differ most.

6. What happens to your data

With retrieval, your documents stay in a store you control. Only the passages relevant to a question are sent to the model, under whatever agreement you have with the provider. If a document must be removed, you delete it from the store and it stops appearing in answers.

With fine-tuning, your examples become part of the model’s weights. Pulling one record back out afterwards isn’t practical, and anyone who can use the model may get output shaped by data they were never meant to see. If your training examples include personal or confidential information, that needs a careful review before training starts, and possibly a different approach.

Either way, the usual questions apply: which provider, which region, what the provider may do with your data and what agreement you can sign. In Canada that includes PIPEDA and any sector rules you’re under.

For managers: If you might need to delete or restrict a piece of information later, keep it out of training data and serve it through retrieval.

7. Why most business projects start with retrieval and good prompts

Most first business requests are about answering from the company’s own information, and that’s the problem retrieval was designed for. It’s quicker to start, easier to update, easier to audit and easier to undo. Clear instructions, a few worked examples in the prompt and a requirement to answer only from the retrieved material cover a large share of what companies ask for.

Start there, test it against a set of real questions with known good answers, and measure where it falls short. If the gaps are missing or outdated facts, improve the documents and the search. If the facts are right but the output keeps coming back in the wrong shape or voice, that’s the point to look at fine-tuning, with real evidence to justify it.

For managers: Ask your team to show you where retrieval and good prompts fall short on real questions before you approve a fine-tuning budget.

The short version

RAG gives a model your facts at the moment it answers. Fine-tuning changes how the model behaves. For changing information, citations and per-user access, retrieval fits better and is easier to maintain and to keep private. Fine-tuning earns its place for consistent format, tone or narrow high-volume tasks once good prompts have been tried. Most business projects should start with retrieval.

Qasim Ali

Senior Software Engineer (Java, AI), CS2 Technologies

Qasim has 18 years of backend experience, including ten years on the backend platform of Vodafone Kabel Deutschland, and builds AI features that run in production for CS2’s own products and for clients. If your team is debating RAG or fine-tuning, he’s happy to look at the use case with you.