Your local RAG pipeline works on a handful of PDFs. You ask it questions, it retrieves context, the LLM responds with grounded answers. Great. Now try loading 10,000 documents and letting 100 users hit it simultaneously. Everything falls apart — slow ingestion, irrelevant retrievals, ballooning costs, and responses that drift into hallucination territory.

This guide covers the architectural decisions that separate a RAG demo from a RAG system. We’ll walk through chunking strategies, embedding pipelines, vector database selection, retrieval optimization, caching layers, and monitoring — the full production stack.

Chunking strategies: the foundation everything else depends on

Chunking is the most underestimated decision in a RAG pipeline. Get it wrong and no amount of retrieval tuning will save you. The chunk is the atomic unit your system retrieves, so its quality directly determines answer quality.

Fixed-size chunking

Split text every N tokens (typically 256–512) with some overlap (50–100 tokens). It’s dead simple, predictable, and fast. The problem: it slices through sentences, paragraphs, and logical boundaries without care. A chunk might start mid-sentence and end mid-thought.

When to use it: homogeneous documents with consistent structure (log files, structured reports), or as a baseline to benchmark against.

Recursive character splitting

This is what LangChain popularized. You define a hierarchy of separators — double newlines, single newlines, sentences, then characters — and the splitter tries the largest separator first, falling back to smaller ones only when chunks exceed your target size. It respects document structure better than fixed-size while remaining deterministic.

When to use it: general-purpose documents where you want a reasonable default without heavy preprocessing.

Semantic chunking

Instead of splitting on characters, you split on meaning. The approach: generate embeddings for each sentence, then measure cosine similarity between consecutive sentences. When similarity drops below a threshold, you insert a chunk boundary. The result is chunks that represent coherent topics.

When to use it: documents with varied structure — meeting transcripts, long-form articles, mixed-format knowledge bases. The tradeoff is compute cost (you’re embedding every sentence during ingestion) and non-determinism (threshold tuning matters).

Practical recommendation

Start with recursive splitting at 512 tokens with 50-token overlap. Measure retrieval quality. If you’re seeing chunks that miss context or split key information, move to semantic chunking for those document types. Most production systems use different strategies for different source types — code gets split differently than legal contracts.

The embedding pipeline: batch processing and incremental updates

A naive pipeline embeds documents one at a time on every query. At scale, you need two separate paths.

Batch ingestion

When loading a new corpus, process embeddings in batches. Most embedding APIs and local models support batch inference. With a model like nomic-embed-text or OpenAI’s text-embedding-3-small, batch sizes of 64–256 are typical. Parallelize across workers if you’re self-hosting.

Key design decisions:

• Deduplication — hash each chunk’s content before embedding. Skip chunks you’ve already processed.
• Metadata attachment — store source document ID, chunk index, creation timestamp, and document type alongside each vector. You’ll need this for filtering and cache invalidation.
• Checkpointing — for large ingestion jobs, write progress to a durable store so you can resume after failures.

Incremental updates

Documents change. The pattern that works: maintain a content hash per source document. On update, re-chunk and re-embed only changed documents, then delete old vectors and insert new ones. This is where metadata pays off — you can delete all vectors where source_doc_id = X without scanning the entire index.

Avoid the temptation to append-only. Stale vectors from deleted or updated documents cause retrieval drift — your system returns outdated information with high confidence.

Vector database selection

Your choice of vector database depends on scale, operational complexity, and existing infrastructure. Here’s how the main options compare in 2026.

Chroma

Embedded, runs in-process, zero infrastructure. Perfect for prototypes and single-user applications. Starts struggling above ~500K vectors and doesn’t support concurrent writes well. If you’re building a desktop app or a developer tool, Chroma is a solid choice.

pgvector

Postgres extension. If your team already runs Postgres, this is the lowest-friction option. Recent versions support HNSW indexes with good recall. The advantage is transactional consistency — your vectors live alongside your relational data, and you get ACID guarantees. The disadvantage is that Postgres wasn’t designed for vector search, so performance at tens of millions of vectors requires careful tuning (index parameters, maintenance memory, connection pooling).

Pinecone

Fully managed, scales horizontally, handles billions of vectors. You pay per vector stored and per query. Best for teams that don’t want to operate infrastructure and need predictable latency at scale. The lock-in is real — there’s no self-hosted option.

Weaviate

Open-source, self-hostable, supports hybrid search natively (BM25 + vector in a single query). Strong multi-tenancy support. More operational overhead than Pinecone but more flexibility. Good choice if hybrid search is central to your architecture.

Decision framework

Under 1M vectors with an existing Postgres stack? Use pgvector. Need managed infrastructure at scale? Pinecone. Want hybrid search and self-hosting? Weaviate. Prototyping? Chroma. The database is not the bottleneck in most RAG systems — chunking and retrieval strategy matter more.

Retrieval optimization: getting the right chunks

Dense vector search alone isn’t enough. The gap between “returns something related” and “returns exactly what the user needs” is where production RAG systems earn their keep.

Hybrid search: BM25 + dense retrieval

Dense embeddings capture semantic similarity but miss exact keyword matches. BM25 (sparse retrieval) excels at exact terms — product names, error codes, acronyms. Combine both with Reciprocal Rank Fusion (RRF): run both searches, merge results by combining their rank positions.

def reciprocal_rank_fusion(dense_results, sparse_results, k=60):
    scores = {}
    for rank, doc in enumerate(dense_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    for rank, doc in enumerate(sparse_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Weaviate and Elasticsearch support this natively. With pgvector, you’ll need to implement it in application code using ts_rank for BM25 alongside vector similarity.

Re-ranking

Retrieve a broad set (top 20–50), then re-rank with a cross-encoder model like bge-reranker-v2 or Cohere Rerank. Cross-encoders are too slow for first-stage retrieval but dramatically improve precision when applied to a small candidate set. This two-stage pattern — fast recall followed by precise re-ranking — is standard in production systems.

Metadata filtering

Don’t search the entire index when you know the user’s context. If a user is asking about Q1 2026 financials, filter to date >= 2026-01-01 AND date <= 2026-03-31 before running vector search. Pre-filtering reduces the search space and improves relevance. This is why attaching rich metadata during ingestion matters.

Caching: the layer most teams skip

RAG systems have two expensive operations: embedding queries and searching vectors. Both are cacheable.

Query cache

Hash the user’s query and cache the final retrieved chunks (or even the full LLM response). Identical or near-identical questions from different users hit the cache instead of running the full pipeline. Use a TTL that matches your document update frequency — if your corpus updates daily, cache for 12–24 hours.

Embedding cache

If you’re using an external embedding API, cache query embeddings keyed by the normalized query string. This saves API calls and latency. For self-hosted models, the compute savings are smaller but still worthwhile at high concurrency.

Semantic cache

A step beyond exact-match caching: embed the incoming query, search a cache index of previous queries, and if a cached query is above a similarity threshold (e.g., 0.95), return the cached result. This catches paraphrased questions. Libraries like GPTCache implement this pattern.

Monitoring: measuring what matters

You can’t improve retrieval quality without measuring it. These are the metrics that matter for production RAG, and they tie directly into how you evaluate AI systems more broadly.

Retrieval metrics

• Hit rate — for a set of test queries with known relevant documents, how often does the relevant document appear in the top-K results?
• Mean Reciprocal Rank (MRR) — where in the ranked list does the first relevant result appear?
• Context relevance — use an LLM-as-judge to score whether retrieved chunks actually help answer the query (frameworks like RAGAS automate this).

End-to-end metrics

• Faithfulness — does the generated answer stay grounded in the retrieved context, or does it hallucinate?
• Answer relevance — does the response actually address the user’s question?
• Latency percentiles — p50, p95, p99 for the full pipeline (embedding → retrieval → generation).

Operational monitoring

Track embedding throughput, vector database query latency, cache hit rates, and error rates. Set alerts on retrieval latency spikes — they usually indicate index degradation or resource contention.

Build a small evaluation dataset (50–100 query-answer pairs with labeled relevant chunks) and run it on every pipeline change. This is your regression test suite for retrieval quality.

Putting it together

A production RAG architecture looks like this: documents flow through a chunking pipeline (strategy chosen per document type), into a batch embedding service, into a vector database with rich metadata. At query time, the user’s question hits a semantic cache first, then gets embedded, runs hybrid search with metadata filters, passes through a re-ranker, and the top chunks feed into the LLM with a carefully constructed prompt.

Each layer is independently tunable and measurable. That’s the difference between a prototype and a system — not more code, but more intentional architecture.

Start with the simplest version that works, measure retrieval quality, and add complexity only where the metrics tell you to. The chunking strategy and re-ranking step will give you the biggest quality gains. Caching and hybrid search will give you the biggest performance gains. Everything else is optimization at the margins.