RAG in Production: AI Document Search a Business Can Trust

2026-11-14

You can build a RAG prototype over a weekend: load some PDFs into a vector database, connect an LLM and get a chat that answers "from the documents". It looks convincing in a demo. The problems start in the first week of real use: the assistant quotes an outdated contract, shows a manager a document only the CFO should see, confidently invents a policy clause, and the API bill grows faster than the user count.

I build RAG systems for businesses where an answer has consequences: customer support, internal knowledge bases, legal and technical documentation, product catalogs. This article explains what separates production from a prototype, and what the work looks like when a client needs a system rather than a demo.

Prototype versus production

Prototype and production RAG
QuestionPrototypeProduction
DataA few PDFs uploaded by handThousands of changing documents from Google Drive, Confluence, a CRM, databases
AccessEveryone sees everythingSearch respects roles, departments and per-document permissions
SearchVector similarity onlyHybrid search, metadata filters, reranking
AnswerText with no sourcesAn answer with citations and links, or an honest "I don't know"
Quality"Seems to work"A test question set, metrics and regression checks before every release
CostNot trackedLimits, caching, the right model per step, a budget per request

Almost every item in the right column is a separate engineering task. They take most of the time, not the LLM call.

Data: ingestion that does not break

Most of a RAG system's quality is decided by what goes into the index. The first part of the work is the ingestion pipeline. It pulls documents from the client's sources through APIs or exports, understands structure (headings, tables, lists), removes duplicates and noise, and splits text into chunks by meaning rather than by a fixed character count.

Each chunk gets metadata: source, date, version, language, department, document type and who may access it. Without this there are no filters, no permissions and no answer to "where did this come from".

Freshness is its own topic. Documents change, get deleted and get new versions. The pipeline has to be incremental: update only what changed, remove from the index what was removed at the source, and keep version history where it matters, for example for contracts.

Permissions inside search

The most dangerous mistake in enterprise RAG is a leak through search. If permissions are filtered after the LLM has already received the retrieved chunks, confidential data is in the model's context and can end up in the answer. So permissions are enforced in the index query itself: a user cannot retrieve a chunk of a document they have no access to.

In practice this means syncing roles and groups from the client's systems (Google Workspace, Microsoft 365, a CRM or their own user database), metadata filters in the vector store, and logging which documents each request saw. For some clients this is an audit requirement, not a nice-to-have.

Search that finds the right thing

Pure vector search is weak on SKUs, contract numbers, product names and exact terms. In production I almost always use hybrid search: full-text (BM25) plus vectors, followed by reranking the best candidates with a separate model. For complex questions I add query rewriting and decomposition, and where relationships between entities matter, graph approaches I described in https://zavada.dev/blog/rag-2-0-and-graph-rag.

The answer is built only from retrieved chunks and always cites them. If nothing relevant is found, the system does not invent an answer. It says so and, when needed, hands the question to a person.

Evaluation and observability

Without measurement every change to a RAG system is a lottery. A new embedding model, a different chunk size or a new prompt can improve some answers and break others. Together with the client I collect a set of real questions with reference answers and sources, and every change runs against it before release.

What I measure in a RAG system
MetricWhat it shows
Retrieval recallWhether the right document made it into the top retrieved chunks
Citation accuracyWhether the source actually supports the answer
"I don't know" rateWhether the system refuses when an answer exists, or invents one when it does not
LatencyTime to first token and total answer time
Cost per requestThe price of one answer including search, reranking and generation
User feedbackWhich answers were marked bad and why

In production every request is traced: what was searched, what was found, what the model received, what it answered and what it cost. I use Langfuse or similar tools for this, more in https://zavada.dev/blog/langfuse-what-it-is-and-how-to-use-it.

Security and cost

Documents are input too, and they can contain malicious instructions. Retrieved text never gets more rights than the user, and any action the system can take (create a ticket, update a record) is checked separately on the server. Personal data is masked where the model does not need it.

On cost: cheap models for classification and query rewriting, a strong model only for the final answer, caching for repeated queries and prompts, per-user limits. For sensitive data a local model on the client's own infrastructure is an option.

What a project looks like

1. Data and use-case audit: which sources, who the users are, the most common questions, and what counts as a correct answer.

2. A pilot on one use case and one data source, with a test question set and metrics.

3. Expansion: more sources, permissions, integration into the product, CRM or messenger, monitoring.

4. Ongoing support: data updates, new use cases, quality and cost control.

The cost of such a project depends on the number and quality of sources, access and security requirements and integrations, so I quote a price after the brief, together with a written scope and milestones.

Conclusion

Production RAG is not "chat with a PDF". It is a data system: an ingestion pipeline, access control, good retrieval, verifiable answers, measurement and cost control. That part decides whether a business trusts the assistant or switches it off after a month.

If you need AI search or an assistant over your own documents that works with real data and real permissions, see how I approach AI projects: https://zavada.dev/ai-solutions. Describe the task in the brief form and I will come back with a plan and an estimate: https://zavada.dev/contacts?service=ai-solutions

Need a developer for the long run?

I can start now and work with you for months or years, not hand off after two weeks. Send me a message and we will talk through your project, requirements and next steps.

Ask about experience or services