RAG in Production: What the Demos Don’t Tell You

The demo is always magical. You point a retrieval-augmented generation system at a pile of documents, ask a hard question, and get a fluent, sourced, correct answer. The room nods. Someone says “we should ship this.” And then production happens — and production has questions the demo never asked.

I have built RAG systems that run in production, including one layered over a manufacturing data pipeline processing billions of IoT rows — where the system doesn’t just retrieve documents, it diagnoses faulty equipment and suggests fixes. The gap between that and a demo is everything I am about to describe.

Chunking Is Where Demos Go to Die

Every RAG demo works because someone carefully prepared the documents. In production, nobody prepares the documents. They arrive as PDFs with broken formatting, Confluence pages with five years of edits, ticket histories, manuals scanned sideways. Your chunking strategy meets this reality on day one.

Chunking — how you split documents into retrievable pieces — is the highest-leverage decision in the system, and it’s the one most teams spend the least time on. Chunk too large and the retriever returns bloated, imprecise context; the model drowns in irrelevant text and the answer quality drops. Chunk too small and you shred meaning across fragments; the retriever finds pieces but the model can’t assemble the full picture. Overlap between chunks helps, but it’s a bandage, not a strategy.

What actually works: chunking driven by document structure, not fixed sizes. Respect natural boundaries — sections, procedures, Q&A pairs. Different document types get different strategies; the approach that works for product manuals will mangle ticket histories. And critically: evaluate chunking choices against real questions, not vibes. Build a small set of representative questions with known-good answers, and measure retrieval quality as you tune. The teams that treat chunking as an empirical problem ship better systems than the teams that treat it as a configuration choice.

One more thing demos hide: metadata. In production, retrieval quality often depends less on the embedding model and more on the metadata attached to each chunk — document type, date, product version, access level. A system that can filter “only show me troubleshooting guides for firmware 4.x” will outperform a system with a fancier model and no metadata, every time.

Your Data Is Stale Before You Ship

The demo uses a static document set. Production data moves. New documents arrive, old ones get updated, some get deleted — and every change creates a window where your system confidently answers from outdated information.

This is a pipeline problem, not a model problem. You need ingestion that keeps up: change detection on source systems, incremental re-indexing (not nightly full rebuilds once the corpus is large), and deletion handling — because a document that was removed for being wrong must actually stop being retrieved. I have seen production systems answer questions from documents that were deleted months earlier, because nobody built the deletion path.

Freshness requirements vary by use case, and you should define them explicitly. A system answering questions about product documentation might tolerate daily updates. A system layered over operational data — like the manufacturing pipeline I built, where equipment telemetry streams continuously — needs freshness measured in minutes, not days. Define the freshness SLA for your use case, then design the pipeline to meet it, and monitor it like any other SLA. Stale answers delivered confidently are worse than no answers at all.

Versioning matters too. When a document updates, do you replace the chunks or version them? For regulated or safety-critical answers, you may need to show which version of a document an answer came from. Build this in early; retrofitting provenance onto a running system is painful.

Latency and Cost at Real Query Volume

The demo answers one question at a time, with nobody watching the clock or the meter. Production answers thousands, concurrently, with users who leave if it takes too long and finance teams who notice the bill.

Latency in RAG is a chain: embedding the query, retrieving candidates, re-ranking, assembling context, generating the answer. Each step adds milliseconds, and the generation step dominates. Practical levers: retrieve fewer, better chunks (quality retrieval beats quantity); cache aggressively — common questions, common documents, even embedding computations; consider smaller, faster models for the generation step when the retrieved context is strong (a good retriever with a small model often beats a mediocre retriever with a large one); and stream responses so users see progress instead of staring at a spinner.

Cost follows the same chain. Embedding is cheap per document but not free at scale — re-embedding an entire corpus because you changed chunking strategy is a real line item. Generation is the expensive step, priced per token, and long retrieved contexts make every query costly. The discipline is the same as any production system: measure cost per query, set budgets, and treat context-window bloat the way you’d treat any other resource leak.

The architecture insight: design for the query volume you’ll have in a year, not the demo volume you have today. The retrieval infrastructure that handles a hundred queries a day gracefully may fall over at ten thousand. Load-test the full chain early — it’s much cheaper than re-architecting under pressure.

Guardrails Aren’t Optional

A demo that hallucinates is embarrassing. A production system that hallucinates is a liability. Guardrails are the difference, and they need to be engineered, not hoped for.

Grounding and citations. The system should show its work — every substantive claim traceable to retrieved sources. This isn’t just good UX; it’s how users learn to trust the system, and how you debug it when answers go wrong. If an answer can’t cite its sources, that should be visible, not hidden.

PII and sensitive data. Your corpus contains things the model should not repeat — personal data, credentials in old tickets, confidential business information. Redaction at ingestion, access-aware retrieval (users only retrieve documents they’re permitted to see), and output filtering are all necessary. Access control deserves emphasis: a RAG system that retrieves across permission boundaries is a data leak with a chat interface. Filter by the user’s permissions at retrieval time, not after generation.

Evaluation as a guardrail. Build an evaluation harness and run it continuously — not just during development. A small set of golden questions with expected answers, run against every change to prompts, models, chunking, or data:

for question, expected_sources in eval_set:
    answer, sources = rag_pipeline.ask(question)
    score_retrieval(sources, expected_sources)  # right documents?
    score_faithfulness(answer, sources)          # grounded in them?
    flag_pii(answer)                            # nothing leaked?

When eval scores drop, you know before users tell you. The teams that run evals in CI ship with confidence; the teams that don’t ship with hope.

Human escalation paths. Some questions shouldn’t be answered by the system — low-confidence retrievals, safety-critical topics, anything where a wrong answer is expensive. Design the handoff: when confidence is low, say so and route to a human. “I don’t know” is a feature, not a failure.

Treat retrieved content as untrusted input. This one surprises teams: your documents can attack your system. A document containing instructions (“ignore previous instructions and…”) gets retrieved, placed in context, and the model may follow it — this is indirect prompt injection, and it works against production systems today. Defenses include separating retrieved content from instructions structurally, scanning retrieved chunks for instruction-like language, and never letting retrieved text drive privileged actions (sending email, changing data, executing commands) without human confirmation. If your RAG system can do things rather than just answer, this moves from important to critical.

When RAG Is the Wrong Answer

After all this, an honest admission: RAG is not always the right architecture, and knowing when to say so is part of doing this work well.

RAG excels when the knowledge is large, changes over time, and needs to be cited — documentation, policies, operational knowledge. It struggles when the task requires reasoning over structured relationships rather than retrieving passages. “Which components fail together, and why?” is a question about relationships — and that’s where knowledge graphs earn their place.

In the manufacturing system I mentioned, RAG alone wasn’t the answer. The pipeline detects faulty IoT devices and components from billions of telemetry rows — but diagnosis benefits enormously from a knowledge graph of the equipment: which components connect to which, which failures cascade, what the maintenance history shows. The graph captures structure; RAG captures narrative. Together, they answer questions neither handles alone: the graph finds what’s related, RAG explains what it means. If your problem has rich entity relationships — equipment, suppliers, systems, dependencies — consider the graph as a complement, not a competitor, to retrieval.

And sometimes the answer is fine-tuning, or a smaller specialized model, or — heresy — a well-built search interface without a language model at all. The right architecture follows the problem. Anyone selling RAG as the universal answer is selling, not engineering.

The Bottom Line

Production RAG is a systems problem wearing a model’s clothes. The demo shows you the model; production tests your chunking, your data pipelines, your latency budget, your guardrails, your evals, and your judgment about when RAG is even the right tool. Get those right and you have something genuinely valuable. Skip them and you have a demo that bills by the token.

If you’re building RAG for real users — or staring at a pilot that isn’t surviving contact with production — reach out through the contact page. I’ve built these systems where they have to work, and I’m glad to help you think through yours.

Related: the Generative AI Foundations learning path, and case studies including an IoT + RAG manufacturing system.