A RAG chatbot is one that looks something up before it answers. RAG stands for retrieval-augmented generation, and the idea behind the acronym is simple: instead of asking a language model what it remembers, you first fetch the relevant passages from a body of content you control, hand them to the model, and ask it to answer from those.

Almost every support chatbot that claims to answer from your documents is a RAG chatbot underneath. The term is useful to a buyer for one reason only — it tells you the vendor’s answers depend on a search step, and search steps can be evaluated. Everything else about the acronym is engineering vocabulary you are entitled to ignore.

What the three letters do

Retrieval is a search over your content. Your documents are split into passages of roughly 900 characters, which is about a paragraph or two, and each question is matched against them. Serochat takes the top six passages and allows at most two from any single document, so one long PDF cannot dominate an answer.

Augmentation is the unglamorous middle step: those passages are pasted into the instructions sent to the model, along with the question and any relevant conversation history. The model is told to answer from the supplied text and to name the document it used.

Generation is the model writing the reply. This is the only part that resembles the chatbots people are used to, and it is the part with the least control over whether the answer is right. What makes a RAG chatbot trustworthy is what surrounds it.

What can go wrong with a RAG chatbot

Vendors rarely publish this list, so here is ours.

  • Retrieval misses. A question phrased in words your documents never use may return the wrong passages, or none. This is the most common failure and the least discussed.
  • Chunk boundaries. If the answer straddles two passages and only one is retrieved, the reply can be confidently half right.
  • Contradictory sources. Two documents with different refund windows produce inconsistent answers, each correctly cited.
  • Drift. Given the right passages, a model can still add a detail that was not in them, because a plausible sentence is what it was built to produce.
  • Staleness. Retrieval reads whatever was last indexed. A bot syncing daily is a day behind your edits until you press sync.

Any vendor who cannot describe how they handle drift is selling you the first three letters and hoping about the last one. Serochat checks the model’s named source against the passages actually retrieved and discards the answer if it does not match, then compares the wording of the reply against those passages and replaces it with a handoff if it has wandered too far.

RAG, grounding and fine-tuning

These three terms get used interchangeably in marketing, and they mean different things.

RAG is the machinery: retrieve, then generate. It describes how an answer is assembled.

Grounding is the promise: this answer came from your content and nothing else, here is which document, and if the content runs out I will say so. Our post on what a grounded AI chatbot is covers how to test that promise from the outside. RAG is how grounding is usually implemented; it is not the same as achieving it, because a RAG system with no verification still lets drift reach the customer.

Fine-tuning is a different technique entirely: adjusting a model’s weights on your examples. It teaches style well and facts badly, cannot cite anything, and has to be redone when your content changes. For customer support it is the wrong tool, and a vendor using “trained on your data” to mean fine-tuned is worth a second question.

Common questions

Do I need a vector database?

No. Vector search can improve retrieval when customers use vocabulary your documents avoid, and it is a genuine upgrade for large content sets. It is not what makes a chatbot trustworthy. Serochat ranks passages with plain full-text keyword search on the database we already run, and we would rather say that plainly than describe it as semantic.

Is a RAG chatbot the same as a grounded chatbot?

Not automatically. RAG describes the retrieval step; grounding describes the guarantee about what reaches the customer. A bot can retrieve well and still print an unverified answer with a fabricated reference underneath it.

Do I need to train a model to use RAG?

No, and that is the main practical appeal. You connect documents, the search index is built for you, and updating an answer means editing a document rather than preparing training data.

The acronym matters far less than two behaviours you can check in five minutes: does the bot name the source of its answers, and does it admit when your documents do not contain one. If it does both, the machinery underneath is doing its job. If it does neither, no amount of retrieval architecture will save the answer that reaches your customer.