Short answer: a general chatbot such as ChatGPT answers from what it learned in training and from whatever you paste into it. A self-hosted retrieval-augmented generation (RAG) system answers only from your own document library, shows the page each claim came from, and runs on servers you control. For contracts, statements and filings, where a wrong or unsourced answer has consequences, that difference decides which one you can use.
What is RAG, in plain terms?
Retrieval-augmented generation splits answering a question into two steps:
- Retrieve. Search your document library for the passages most relevant to the question.
- Generate. Give those passages to a language model and ask it to answer using only them.
The model still writes the answer, but it is working from your documents rather than from memory. That is what makes it possible to cite sources, and to say "the documents do not cover this" instead of guessing.
Where does a general chatbot fall short on company documents?
- No sources. A chatbot's answer reads confidently whether or not it is right. Without a link back to the page, someone has to check every answer by hand.
- Guessing. When the answer is not in what it was given, a general model tends to fill the gap with something plausible. In a contract review, a plausible but wrong renewal date is worse than no answer.
- Scale. Pasting documents into a chat works for one file. It does not work for a library of thousands of contracts, statements and spreadsheets that changes every week.
- Data leaving your network. Every document and question sent to a public service leaves your control. Many organisations cannot allow that for client contracts or financial records.
- No access control. A chat window does not know that the finance team may see some collections and the sales team may not.
What does a self-hosted RAG system add?
| General chatbot | Self-hosted RAG | |
|---|---|---|
| Where answers come from | Training data and pasted text | Only your document library |
| Sources | Usually none | Every claim linked to its document and page |
| When the answer is missing | Often guesses | Says the documents do not hold it |
| Where documents go | The provider's cloud | Your own servers |
| Who sees what | Everyone who can open the chat | Role-based access per collection |
| Audit | Limited | Every query and change logged |
What makes RAG answers trustworthy?
Retrieval alone is not enough. Good systems add checks around it:
- Grading passages before they reach the model, so weak matches do not steer the answer.
- Checking each claim against its source before answering, not just generating and hoping.
- Keeping document structure during ingestion, so tables survive and a figure can be cited to its row.
- Reading scans with OCR, so older paper records are searchable too.
This is how Provena, our agentic RAG product, works: an agent refines the question, retrieves and grades passages, and checks every claim against its sources before it answers. Each claim cites the document, page and region it came from, and answers from spreadsheets cite the sheet and row.
Do we have to send our documents to the cloud?
No. A self-hosted RAG system can run entirely on your own servers, with self-hosted language models served by tools such as vLLM or Ollama. Documents and questions then never leave your network. Cloud models remain an option for teams that prefer them, but they are not a requirement.
When is a general chatbot enough?
For drafting emails, brainstorming or summarising a single public document, a general chatbot is quick and good enough. The case for self-hosted RAG starts when three things are true at once: the documents are yours and confidential, there are too many to paste, and people will act on the answers.
Key takeaways
- A general chatbot answers from memory; RAG answers from your documents.
- Citations to the page turn "trust me" into "check it yourself in one click".
- Self-hosting keeps confidential documents and questions inside your network.
- Access control and audit logs matter as soon as more than one team uses it.
If you want to see this on your own documents, request a demo of Provena.