Prompting, RAG or fine-tuning: start with prompting, add retrieval when the model must answer from your own documents, and fine-tune only when a consistent style or format cannot be reached any other way. Choose by what information the model needs, how often it changes, and how much testing and maintenance you can sustain.
Key takeaways
- Prompting changes the instructions, RAG changes the information the model sees, and fine-tuning changes the model's behaviour.
- Start with prompting and a small set of hand-checked test cases before building anything more complex.
- Use retrieval when answers depend on documents that change or must be traceable to a source.
- Consider fine-tuning only when you have high-quality examples and the other approaches still miss on format or consistency.
- Factor in ongoing maintenance, regulatory duties and data handling before you commit to an approach.
What does each approach change in the model?
Each approach changes a different part of the system. Prompting changes the instructions sent with each request, RAG changes the evidence the model receives at answer time, and fine-tuning changes the model's weights using your example input and output pairs.
Prompting means writing the role, the rules, a few examples and the output format into every request. The underlying model is not modified, so a change takes effect immediately and can be rolled back in minutes.
Retrieval-augmented generation, usually shortened to RAG, searches a store of your documents for passages relevant to a question and places them in the prompt. The model's built-in knowledge stays the same, but the material it reasons over is current and can be cited.
Fine-tuning trains an existing model further on examples of the behaviour you want. It shapes tone, structure and classification more reliably than long prompts do, but it is a poor way to add facts, because what the model learns from training is hard to inspect, update or trace.
When is prompting enough on its own?
Prompting is enough when the task relies on general language ability and all the facts it needs fit comfortably inside the request. Summarising a supplied text, drafting replies in a fixed tone and extracting fields from a known document layout are typical cases.
Before you rely on prompting, check the following:
- The required facts are short, stable and can be pasted into each call.
- The output format can be described in plain instructions and checked automatically.
- A small set of hand-checked test cases passes consistently across typical and awkward inputs.
- The team can maintain the prompt, with a named owner and a change log.
Prompting fails in predictable ways. Prompts grow long and contradictory as edge cases accumulate, every call carries the extra text in cost and latency, and small wording changes can shift results in ways nobody tested. When a prompt needs a page of exceptions, that is a signal to look at retrieval or a different workflow.
When does retrieval (RAG) make sense?
Use RAG when answers must come from documents that change, when the source material is too large to fit in a prompt, or when users need to see where a statement came from. Internal policy libraries, product manuals and contract archives are common examples.
A RAG system has three moving parts. Documents are split into chunks and indexed, a retrieval step selects the chunks most relevant to each question, and the model is instructed to answer only from those chunks and to say when they are insufficient. Each part can fail independently, so each needs its own tests.
The typical failure modes are specific. The wrong chunk is retrieved and the answer sounds confident but is off, the index goes stale after documents change, or access rules are ignored so that users see content they should not. Connecting the document store to the systems that own it is usually the hardest engineering task, and it is where system integration work tends to matter more than model choice.
Is fine-tuning worth the effort?
Fine-tuning is worth the effort when you have many high-quality examples of the desired output and prompting plus retrieval still miss on format, consistency or domain-specific classification. It is rarely the first step.
The cost sits mostly outside the training run. Someone must curate the examples, agree what a correct output looks like, build an evaluation set that the model has never seen, and repeat that work whenever the base model changes or the requirements shift. A fine-tuned model still makes mistakes, and it can still state outdated information with confidence.
Fine-tuning fails when the dataset is small or noisy, when the examples disagree with each other, or when the team uses it to store facts that will go out of date. In those cases the result looks trained but behaves unpredictably, and the time is better spent on clearer prompts or a retrieval layer.
How do you choose step by step?
Work through the decision in order, and stop at the first step that solves the problem. Each step adds cost, so a simpler option that meets the test set is the right answer.
- Define the task and the cost of a wrong answer. A wrong draft email and a wrong compliance statement call for different levels of checking.
- Test prompting against a small set of hand-checked cases that includes awkward inputs.
- If failures come from missing or changing information, add retrieval and test again.
- If failures come from behaviour, format or consistency that examples can teach, prepare a fine-tuning dataset and an evaluation set.
- Plan maintenance before launch: prompt reviews, index refreshes, retraining and monitoring.
Where the scope is unclear, an AI consulting review of the task and its data can establish which step is actually needed before any build begins.
| Criterion | Prompting | RAG | Fine-tuning |
|---|---|---|---|
| Changes the underlying model | No | No | Yes |
| Handles frequently changing facts | Poorly | Well | Poorly |
| Makes answers traceable to sources | Limited | Strong | Limited |
| Setup effort | Low | Medium | High |
| Ongoing maintenance | Prompt reviews | Index and retrieval upkeep | Retraining and re-evaluation |
Hypothetical examples of the choice in practice
Here are two hypothetical scenarios: the first shows a case where prompting is enough, and the second shows where retrieval becomes the right answer. Neither describes a real organisation.
In the first scenario, an operations team wants a drafting assistant that replies to routine supplier emails in a fixed, polite tone. A prompt with the tone rules, two model replies and a list of forbidden phrases handles most cases. Retrieval would only be added if replies had to quote the current price list.
In the second scenario, an IT lead wants an assistant that answers staff questions about an internal policy library that is revised every few weeks. Fine-tuning would freeze today's policies into the model and go out of date with the next revision. Retrieval over the current policy documents, with the source section shown beside each answer, fits the change pattern and the need for traceability.
Does regulation change the choice?
Yes, when the system handles personal data or falls within scope of the EU Artificial Intelligence Act. The approach you choose affects what data is copied into indexes or training sets, what documentation you keep, and how you can explain outputs. The Act sets duties that depend on the risk category of the system, so read the official text of Regulation (EU) 2024/1689 for the obligations that apply to your use case.
Personal data raises the same questions in a retrieval index or a training set as it does anywhere else. Each copy of a document that contains personal data is a processing activity, and the General Data Protection Regulation (EU) 2016/679 governs how that data may be processed. Deleting a source document should also remove it from the index, which is a design requirement to set early.
This article is general information and not legal or tax advice. Rui Codex provides technical implementation support, not legal advice, audits or certification, so classify your system and confirm your obligations with qualified counsel.
Frequently Asked Questions
Can prompting replace RAG for a business task?
Prompting can replace RAG only when the information needed is short, stable and fits inside each request. If answers depend on large or changing document collections, prompting alone cannot keep the model supplied with the right facts, and the prompt becomes expensive, slow and error-prone.
Does fine-tuning teach a model new facts?
Fine-tuning is a poor way to add reliable new facts, and it mainly shapes style, format and task behaviour. Knowledge learned through training is hard to inspect or update, so changing information is better handled with retrieval. Fine-tuning remains useful for consistent classification, structured output and tone.
What data do I need before choosing between these approaches?
You need a representative set of real questions, the documents or examples those questions depend on, and a way to judge whether answers are correct. Without a test set you cannot tell which approach improved the result. Review personal data handling before you index or train on any material.
How long does each approach take to build?
Prompting usually needs the least setup, RAG adds indexing and retrieval engineering, and fine-tuning adds dataset curation and evaluation. Timelines depend on data quality, access rules and integration work, so scope them with a technical review before committing.
Conclusion
Prompting, RAG and fine-tuning solve different problems, so the right choice starts with the failure you are trying to fix. Begin with prompting and a test set, add retrieval when answers depend on documents that change, and consider fine-tuning only for consistent behaviour that the other two cannot reach. Each option carries maintenance work, from prompt reviews to index refreshes and retraining, and that work belongs in the decision from the start. To discuss your task, request a project consultation at https://ruicodex.com/Contact or email info@ruicodex.com to discuss your project.
Sources cited in this article
- Regulation (EU) 2024/1689 (eur-lex.europa.eu)
- General Data Protection Regulation (EU) 2016/679 (eur-lex.europa.eu)