You want your LLM to know about your data. Three approaches: stuff it in the prompt (prompt engineering), retrieve relevant context at query time (RAG), or train the model on your data (fine-tuning). Here’s when to use each.

Quick comparison

	Prompt engineering	RAG	Fine-tuning
Setup time	Minutes	Days	Weeks
Cost	$0	$50-500 (infra)	$500-10,000
Data freshness	Manual updates	Real-time	Requires retraining
Knowledge size	Limited by context window	Unlimited	Baked into weights
Quality	Good for simple tasks	Best for knowledge-heavy	Best for style/behavior
Maintenance	Low	Medium	High

Prompt engineering — start here

Put your knowledge directly in the system prompt or few-shot examples.

system_prompt = """You are a customer support agent for Acme Corp.

Product info:
- Basic plan: $10/mo, 5 users, 10GB storage
- Pro plan: $25/mo, 25 users, 100GB storage
- Enterprise: Custom pricing, unlimited users

Refund policy: Full refund within 30 days. After 30 days, prorated refund."""

Use when:

• Your knowledge fits in the context window (<10K tokens)
• Information changes rarely
• You need it working in 5 minutes

Don’t use when:

• Knowledge exceeds context window
• Information changes frequently
• You need to search across large document sets

RAG — for knowledge-heavy apps

Retrieval-Augmented Generation retrieves relevant documents from a vector database and includes them in the prompt at query time.

# 1. User asks a question
query = "What's the refund policy for annual plans?"

# 2. Retrieve relevant docs
docs = vector_db.search(query, top_k=3)

# 3. Include in prompt
prompt = f"""Answer based on these documents:
{docs}

Question: {query}"""

response = call_llm(prompt)

Use when:

• Large knowledge base (100+ documents)
• Information updates frequently
• Users ask diverse questions across many topics
• You need citations/sources

Don’t use when:

• Knowledge is small and static (use prompt engineering)
• You need the model to change its behavior/style (use fine-tuning)
• Retrieval quality is critical and your data is unstructured

See our embeddings guide and vector database comparison for implementation.

Fine-tuning — for behavior change

Fine-tuning trains the model on your data, changing its weights. The model learns your style, terminology, and patterns.

Use when:

• You need consistent output format/style
• Domain-specific terminology that the base model gets wrong
• You want to distill a large model’s behavior into a smaller, cheaper model
• You have 1,000+ high-quality training examples

Don’t use when:

• You just need the model to know facts (use RAG)
• Your data changes frequently (retraining is expensive)
• You have fewer than 500 training examples
• You’re using a proprietary model (can’t fine-tune Claude/GPT easily)

Decision framework

Need the model to KNOW things?
  → Small knowledge base?     → Prompt engineering
  → Large knowledge base?     → RAG

Need the model to BEHAVE differently?
  → Consistent format/style?  → Fine-tuning
  → Follow specific rules?    → Prompt engineering

Need both?
  → Fine-tune for behavior + RAG for knowledge

Cost comparison

Approach	Setup cost	Monthly cost	Maintenance
Prompt engineering	$0	API costs only	Update prompts manually
RAG	$100-500 (vector DB setup)	$20-200 (hosting + embeddings)	Update documents
Fine-tuning	$500-10,000 (training)	API costs (cheaper per token)	Retrain quarterly

The practical path

1. Start with prompt engineering — get a working prototype in hours
2. Add RAG when your knowledge exceeds the context window
3. Fine-tune only when prompt engineering and RAG can’t achieve the quality you need

Most production AI apps never need fine-tuning. Prompt engineering + RAG covers 90% of use cases at a fraction of the cost.

Related: What is RAG? · What is a Vector Database? · Embeddings Explained · Vector Databases Compared · How to Reduce LLM API Costs · Context Engineering Explained