Production checklist

A pragmatic checklist for taking RAG from prototype to production. Not every item applies to every project — use it as a prompt to make deliberate decisions rather than accidental ones.

Data & indexing

• Parsing handles all your real formats, including tables and scanned PDFs.
• Text is cleaned and normalised; boilerplate removed.
• Chunking strategy chosen and tuned against real questions.
• Metadata schema covers source, title, timestamp, tenant, and permissions.
• Incremental add/update/delete works without a full rebuild.
• Deletion removes vectors and caches (privacy/compliance).

Retrieval & generation

• Same embedding model for indexing and queries; index is versioned.
• Hybrid (dense + sparse) search where exact terms matter.
• Reranking over a wide candidate set.
• Query rewriting for multi-turn chat.
• Prompt instructs grounding and gives a "say I don't know" escape hatch.
• Low-confidence/empty retrieval handled honestly.
• Citations returned and verifiable.

Quality

• Golden evaluation set exists and is version-controlled.
• Retrieval metrics (recall@k) and generation metrics (faithfulness, relevancy) measured.
• Evaluation runs in CI on every change.
• User feedback captured and fed back into the golden set.

Operations

• Latency budget met (streaming, parallelism, right-sized models).
• Caching in place where it pays off.
• Cost per query measured and monitored.
• Timeouts, retries, and fallbacks for external services.
• Full per-request tracing (query, chunks, scores, prompt, answer, cost).
• Dashboards and alerts for latency, cost, and retrieval-quality drift.

Security

• Access control enforced in the retrieval filter, not the prompt.
• Multi-tenant isolation tested deliberately.
• Indirect prompt injection mitigated; model capabilities constrained.
• PII handling and data-residency requirements met.
• Output screening before display.

Written by Bipin Singh

Senior Full-Stack Engineer · AI & AWS. I build production RAG and LLM systems for enterprises.

Work with me