You can build a RAG prototype over a weekend: load some PDFs into a vector database, connect an LLM and get a chat that answers "from the documents". It looks convincing in a demo. The problems start in the first week of real use: the assistant quotes an outdated contract, shows a manager a document only the CFO should see, confidently invents a policy clause, and the API bill grows faster than the user count.
I build RAG systems for businesses where an answer has consequences: customer support, internal knowledge bases, legal and technical documentation, product catalogs. This article explains what separates production from a prototype, and what the work looks like when a client needs a system rather than a demo.
Prototype versus production
| Question | Prototype | Production |
|---|---|---|
| Data | A few PDFs uploaded by hand | Thousands of changing documents from Google Drive, Confluence, a CRM, databases |
| Access | Everyone sees everything | Search respects roles, departments and per-document permissions |
| Search | Vector similarity only | Hybrid search, metadata filters, reranking |
| Answer | Text with no sources | An answer with citations and links, or an honest "I don't know" |
| Quality | "Seems to work" | A test question set, metrics and regression checks before every release |
| Cost | Not tracked | Limits, caching, the right model per step, a budget per request |
Almost every item in the right column is a separate engineering task. They take most of the time, not the LLM call.
Data: ingestion that does not break
Most of a RAG system's quality is decided by what goes into the index. The first part of the work is the ingestion pipeline. It pulls documents from the client's sources through APIs or exports, understands structure (headings, tables, lists), removes duplicates and noise, and splits text into chunks by meaning rather than by a fixed character count.
Each chunk gets metadata: source, date, version, language, department, document type and who may access it. Without this there are no filters, no permissions and no answer to "where did this come from".
Freshness is its own topic. Documents change, get deleted and get new versions. The pipeline has to be incremental: update only what changed, remove from the index what was removed at the source, and keep version history where it matters, for example for contracts.
Permissions inside search
The most dangerous mistake in enterprise RAG is a leak through search. If permissions are filtered after the LLM has already received the retrieved chunks, confidential data is in the model's context and can end up in the answer. So permissions are enforced in the index query itself: a user cannot retrieve a chunk of a document they have no access to.
In practice this means syncing roles and groups from the client's systems (Google Workspace, Microsoft 365, a CRM or their own user database), metadata filters in the vector store, and logging which documents each request saw. For some clients this is an audit requirement, not a nice-to-have.
Search that finds the right thing
Pure vector search is weak on SKUs, contract numbers, product names and exact terms. In production I almost always use hybrid search: full-text (BM25) plus vectors, followed by reranking the best candidates with a separate model. For complex questions I add query rewriting and decomposition, and where relationships between entities matter, graph approaches I described in https://zavada.dev/blog/rag-2-0-and-graph-rag.
The answer is built only from retrieved chunks and always cites them. If nothing relevant is found, the system does not invent an answer. It says so and, when needed, hands the question to a person.
Evaluation and observability
Without measurement every change to a RAG system is a lottery. A new embedding model, a different chunk size or a new prompt can improve some answers and break others. Together with the client I collect a set of real questions with reference answers and sources, and every change runs against it before release.
| Metric | What it shows |
|---|---|
| Retrieval recall | Whether the right document made it into the top retrieved chunks |
| Citation accuracy | Whether the source actually supports the answer |
| "I don't know" rate | Whether the system refuses when an answer exists, or invents one when it does not |
| Latency | Time to first token and total answer time |
| Cost per request | The price of one answer including search, reranking and generation |
| User feedback | Which answers were marked bad and why |
In production every request is traced: what was searched, what was found, what the model received, what it answered and what it cost. I use Langfuse or similar tools for this, more in https://zavada.dev/blog/langfuse-what-it-is-and-how-to-use-it.
Security and cost
Documents are input too, and they can contain malicious instructions. Retrieved text never gets more rights than the user, and any action the system can take (create a ticket, update a record) is checked separately on the server. Personal data is masked where the model does not need it.
On cost: cheap models for classification and query rewriting, a strong model only for the final answer, caching for repeated queries and prompts, per-user limits. For sensitive data a local model on the client's own infrastructure is an option.
What a project looks like
1. Data and use-case audit: which sources, who the users are, the most common questions, and what counts as a correct answer.
2. A pilot on one use case and one data source, with a test question set and metrics.
3. Expansion: more sources, permissions, integration into the product, CRM or messenger, monitoring.
4. Ongoing support: data updates, new use cases, quality and cost control.
The cost of such a project depends on the number and quality of sources, access and security requirements and integrations, so I quote a price after the brief, together with a written scope and milestones.
Conclusion
Production RAG is not "chat with a PDF". It is a data system: an ingestion pipeline, access control, good retrieval, verifiable answers, measurement and cost control. That part decides whether a business trusts the assistant or switches it off after a month.
If you need AI search or an assistant over your own documents that works with real data and real permissions, see how I approach AI projects: https://zavada.dev/ai-solutions. Describe the task in the brief form and I will come back with a plan and an estimate: https://zavada.dev/contacts?service=ai-solutions
Need a developer for the long run?
I can start now and work with you for months or years, not hand off after two weeks. Send me a message and we will talk through your project, requirements and next steps.