Getting RAG into production without regrets

Retrieval-augmented generation (RAG) has become the default way to give a large language model access to your own knowledge. The demo is deceptively easy: embed some documents, drop them in a vector store, stuff the top matches into a prompt, and you have something that looks like magic.

The problem is that “looks like magic” and “runs reliably for a year” are separated by a lot of unglamorous engineering. Here is the checklist I work through before I’m comfortable calling a RAG system production-ready.

Treat retrieval as a first-class system

Most RAG failures are retrieval failures, not model failures. If the right chunk never makes it into the context window, no amount of prompt tuning will save you. That means:

• Chunking strategy is a design decision, not a default. Test a few and measure.
• Metadata filtering matters as much as vector similarity — scope by tenant, recency and document type.
• Re-ranking the top-k results before they hit the prompt often beats a bigger model.

Build the evaluation harness first

You cannot improve what you cannot measure. Before shipping, I want a golden set of questions with expected answers, and an automated way to score responses on relevance, faithfulness and completeness. This turns “the model feels worse today” into a number you can act on.

If a change to your prompt, chunking or model can’t be evaluated against a fixed set, you’re tuning by vibes.

Guardrails and cost controls are infrastructure

A production LLM feature needs the same operational thinking as any other service:

• Guardrails on inputs and outputs — prompt-injection defence, PII handling, refusal behaviour.
• Token budgets and caching, because costs scale with usage in ways that surprise finance teams.
• Tracing every call so you can debug a bad answer three weeks later.

The takeaway

RAG is not an AI problem so much as a systems problem with an AI component. Apply the same discipline you’d apply to any distributed system — observability, evaluation, cost awareness — and it becomes just another reliable service. Skip that discipline and you’ve built a very expensive way to generate confident nonsense.

If you’re taking an AI feature from prototype to production, that’s exactly the kind of work DeployCraft does.