Fine-Tuning vs RAG vs Prompt Engineering: How to Choose the Right Approach

"Should we fine-tune a model?" is one of the most common questions we hear from teams starting with generative AI. The honest answer is usually "not yet". Fine-tuning is powerful, but it solves a narrower problem than most people expect. This guide lays out what prompt engineering, retrieval-augmented generation (RAG) and fine-tuning each actually change, and how to decide.

#Key takeaways

• Prompt engineering changes the instructions. Start here, always.
• RAG changes the knowledge available at query time. Use it for facts, private data and anything that changes.
• Fine-tuning changes the model's behavior: format, style, domain conventions, or making a smaller model do a narrow task well.
• Fine-tuning is a poor way to teach facts. It's expensive to update and doesn't provide citations.
• Most mature systems combine them: good prompts + RAG, with fine-tuning added only where evals show a clear gap.

#The one-line mental model

Prompting tells the model what to do. RAG tells it what to know right now. Fine-tuning changes who it is.

#Prompt engineering

What it is: Writing instructions, context and examples that steer a general model toward your task. (Deep dive: prompt engineering for developers.)

Strengths

• Fastest iteration loop: minutes, not days.
• No training data or infrastructure needed.
• Portable across models, with re-testing.

Limits

• Everything must fit in the context window, on every request.
• Long prompts with many examples add cost and latency.
• Some behaviors are hard to specify in words, such as a very particular house style.

Use it for: every project, as the baseline you measure everything else against.

#Retrieval-augmented generation (RAG)

What it is: At query time, search your data for relevant passages and include them in the prompt. (Deep dive: RAG explained.)

Strengths

• Works with private, proprietary and constantly changing data. Update the index, not the model.
• Answers can cite sources, which builds trust and makes auditing possible.
• Access control can be enforced per user at retrieval time.

Limits

• Quality depends heavily on retrieval: chunking, search and reranking.
• More moving parts: ingestion pipelines, indexes, freshness.
• Adds some latency per request.

Use it for: support bots, internal knowledge search, document Q&A, product assistants, anything where correctness depends on specific facts.

#Fine-tuning

What it is: Continuing to train a model on your examples so its weights change. Most teams today use parameter-efficient fine-tuning (such as LoRA adapters) or a provider's hosted fine-tuning API rather than full retraining.

Strengths

• Teaches consistent format and style that's hard to describe in a prompt.
• Lets a smaller, cheaper, faster model match a larger one on a narrow task.
• Shortens prompts, because instructions and examples get "baked in", reducing per-request cost at high volume.
• Can improve performance on domain-specific language patterns such as legal clauses or medical coding conventions.

Limits

• Needs high-quality training data, typically hundreds to thousands of carefully curated examples.
• Knowledge baked into weights goes stale and can't be cited or permission-filtered.
• Retraining is needed when the base model or task changes.
• Risk of degrading general capabilities if done carelessly.

Use it for: high-volume narrow tasks (classification, extraction, routing), strict output styles, distilling a big model's behavior into a small one, and latency-critical paths.

#Side-by-side comparison

#A decision framework

Work through these questions in order:

1. Have you pushed prompting seriously? Clear instructions, few-shot examples, structured outputs, and an eval set that measures the gap. If not, do that first. Many "we need fine-tuning" problems disappear here.
2. Is the failure about missing or outdated knowledge? The model doesn't know your pricing, your API or last week's policy change → RAG.
3. Is the failure about behavior? Wrong format, inconsistent tone, ignoring conventions, despite good prompts and examples → fine-tuning is a candidate.
4. Is the problem cost or latency at scale? A large model works but is too slow or expensive for millions of calls → fine-tune a smaller model using the large model's good outputs as training data (distillation).
5. Do you have the data? No curated examples → you can't fine-tune well yet. Start logging production inputs and corrected outputs now.

#Real-world scenarios

Customer support assistant for a SaaS product. Knowledge changes weekly and answers must reference help-center articles. → RAG + strong prompt. Fine-tuning adds little.

Extracting 30 fields from insurance claim PDFs, 2 million per month. A large model does it well but costs too much. → Prompt + large model to generate high-quality labels, then fine-tune a smaller model, validated against a held-out set. See also intelligent document processing.

Marketing copy in a very specific brand voice. Few-shot examples get close but drift. → Fine-tune on approved copy, and keep product facts in the prompt or retrieved context.

Internal assistant that answers HR policy questions per country. Access control and freshness matter. → RAG with metadata filters by country and employee permissions.

#Combining approaches

These aren't mutually exclusive. A common mature architecture:

textFine-tuned small model (format + domain conventions)
      + RAG (fresh facts with citations)
      + Tight prompt (task instructions, guardrails)
      + Evals in CI (catch regressions)

Another emerging pattern is fine-tuning a model to use retrieval better: to cite properly, abstain when the context is insufficient, and ignore irrelevant chunks.

#Cost reality check

Don't compare approaches on training cost alone. Include:

• Engineering time to build and maintain data pipelines or training sets.
• Per-request cost at your expected volume. At high volume, fine-tuned small models can be dramatically cheaper.
• Update cost when facts, policies or base models change.
• Evaluation cost. You need evals for all three approaches; that part is non-negotiable.

For a deeper look at the spend side, read LLM cost optimization.

#FAQ

#Can fine-tuning teach a model new facts?

Somewhat, but unreliably. Models can memorize facts seen during fine-tuning, but recall is inconsistent, hard to update and impossible to cite. For factual knowledge, RAG is almost always the better tool.

#How much data do I need to fine-tune?

For narrow tasks with parameter-efficient methods, a few hundred high-quality examples can produce meaningful gains. A thousand or more is common for robust results. Quality and diversity matter more than raw count.

#Is fine-tuning expensive?

Parameter-efficient fine-tuning of small and mid-sized open models is relatively affordable, and hosted fine-tuning APIs price by training tokens. The bigger costs are usually data preparation, evaluation and ongoing maintenance.

#Should I fine-tune an open-source model or use a provider's API?

Open models give you control, on-prem deployment and no per-token vendor fees, but you own hosting and operations. Hosted fine-tuning is simpler to start with. Decide based on data-privacy requirements, volume and your team's ML-ops capacity.