Quick answer: Start with representative evaluations and clearer prompting. Use RAG when the model lacks private, changing, or sourceable information. Consider fine-tuning when you have strong examples and need more consistent behavior, style, format, or task performance. Combine methods only when tests show both a knowledge problem and a behavior problem.
Checked July 17, 2026. The decision framework is vendor-neutral. Product support, prices, limits, and fine-tuning options change, so check current first-party documentation before implementation.
Key takeaways
- Define the failure and evaluation before choosing a method.
- Prompting changes the instructions and examples sent at run time.
- RAG retrieves external context; it does not guarantee factual answers or correct citations.
- Fine-tuning changes model weights using examples; it does not provide automatically current knowledge.
- These methods are tools for different problems, not steps on a sophistication ladder.
Diagnose the problem first
| Observed problem | First method to test | Why |
|---|---|---|
| The instruction is unclear or the output format varies | Prompting with clearer rules and examples | The model may already have the needed capability and context |
| The answer needs private, recent, or source-specific facts | RAG | Retrieve approved information at answer time |
| The task is repeated and behavior stays inconsistent despite strong prompts | Fine-tuning | Learn a repeated response pattern from representative examples |
| The system lacks current knowledge and also needs specialized behavior | RAG plus fine-tuning, if evaluations justify both | One supplies context; the other shapes behavior |
Prompting: change the instructions first
Prompting includes the task, constraints, context, examples, output format, and evaluation instructions sent to a model. It is usually the fastest and least complex place to start because no model weights or retrieval system need to change.
- Good fit: clearer tone, structure, classification rules, extraction fields, and a small number of examples.
- Strength: quick to change and test.
- Limit: a prompt cannot supply knowledge it does not include or make every response consistent.
- Evidence: compare prompt versions on a held-out representative set, not a few favorite examples.
RAG: supply approved knowledge at answer time
Retrieval-augmented generation searches an external collection and places relevant passages into the model’s context. It is useful when information changes, is private, or should be traceable to a source. The document index can be updated separately from the model.
- Good fit: policies, product documents, support knowledge, research, or internal files that change.
- Strength: separates maintained knowledge from model weights and can expose supporting passages.
- Limit: retrieval can miss, rank, chunk, or permission the wrong information; generation can still misread it.
- Evidence: test retrieval recall, answer quality, citation accuracy, access control, and behavior when no good passage exists.
Fine-tuning: change repeated behavior
Fine-tuning trains model weights on representative examples. It can improve repeated task behavior, style, format, terminology, or performance when prompts alone are not consistent enough. It requires careful data, evaluation, maintenance, and provider support.
- Good fit: many examples of a stable task with a clear standard.
- Strength: can make specialized behavior more consistent and reduce long repeated instructions.
- Limit: weak, biased, private, or inconsistent examples can teach the wrong behavior; knowledge is not automatically current.
- Evidence: separate training and held-out evaluation sets, compare with the best prompt baseline, and test failure cases.
A practical decision path
- Write the task and failure in one sentence.
- Build a representative evaluation set and rubric.
- Try a clearer prompt and examples.
- If required facts are missing or change often, test RAG with approved sources.
- If behavior remains inconsistent across many examples, compare fine-tuning with the best prompt/RAG baseline.
- Choose the simplest method that meets the written quality, privacy, security, cost, latency, and maintenance gates.
- Monitor and rerun the evaluation after meaningful changes.
Small-team examples
- Prompting: turn meeting notes into one consistent template using explicit headings and two approved examples.
- RAG: answer staff questions from the current handbook while showing the source section.
- Fine-tuning: classify a high-volume, stable request type after prompts and examples still produce inconsistent labels and the team has a quality dataset.
- Combined: retrieve the current policy and use a fine-tuned model to produce a specialized structured output—only if separate tests show each part is needed.
Limitations
Prompting can be brittle. RAG depends on source quality, chunking, ranking, permissions, and context use. Fine-tuning depends on representative data and can create new failure modes. All three can produce inaccurate or harmful output. Protect sensitive data and use the human-review matrix for consequential work.
Related guides
Read What Is RAG?, Training vs Inference, What Is an LLM?, the long-document workflow, and AI Tools.