RAG
RAG stands for Retrieval-Augmented Generation. The model retrieves relevant sources before answering and assembles the answer from them, instead of drawing only on what it learned during training. This mechanism is precisely why a brand's visibility in AI answers can be influenced by content you publish today.
In short
| What it is | Generating an answer from retrieved sources rather than from the model's memory |
| Why marketers care | Without RAG you could only influence what was in the training data. With RAG, current content and its availability decide |
| Who uses it | Effectively every assistant with web access: ChatGPT, Perplexity, Gemini, Claude, Copilot |
| What RAG rewards | Clear statements that make sense out of context, and original data |
| Where it breaks | On technical availability. What a crawler cannot fetch never enters an answer |
How RAG works
It runs in three steps. The query is first turned into something searchable, often by splitting it into several sub-questions. Passages answering them are then pulled from an index. Finally the model assembles an answer from those passages and attaches source links.
Retrieval works on passages, not pages
This is the most important consequence for content. The unit is not a URL, it is a paragraph or a section. A long article with no part that answers standalone has little chance, however well it matches the topic. A short passage carrying a clear definition gets picked up easily.
Two different crawlers, two different kinds of damage
Model providers run separate crawlers. One collects content into the index, the other fetches a page only when it comes up inside a specific answer. Each can be blocked on its own and the consequences differ: without the indexing crawler the site never enters the candidate set at all, without the second one the model cannot verify the current wording of the page.
The model prefers what it can verify
A claim backed by a number, a date or a source enters answers more often than a general statement. Marketing copy with no data gives the model nothing to hold on to.
From our own practice: crawlers blocked by the firewall
In technical audits we test language model and search crawlers directly, each with a request carrying its user agent. Some routinely receive HTTP 403 even though robots.txt explicitly allows them. They are blocked by the firewall at hosting level or by the CDN, a layer most companies never check.
The split between two bots of the same provider is instructive: the crawler that fetches a page during a live answer passes without trouble, while the indexing crawler gets a 403. In practice that means the model can open the page once it knows about it, but will never discover it on its own.
Hence the check that belongs in routine maintenance: go through the crawler list and verify the response code directly. Reading robots.txt is not enough, because robots.txt can contradict a block sitting at firewall level.
Common mistakes
- Assuming the model reads your site directly. It reads an index. A new page shows up in answers after indexing, not right after publication.
- Optimizing whole pages instead of passages. RAG works with sections, not URLs.
- Trusting robots.txt. Real availability is decided by the server, not by a file.
- Publishing content with no numbers. The model favors verifiable claims and passes over generic text.
What it means for content
- Write in blocks that work standalone. Each section should answer one question in a complete sentence.
- Put the answer right under the heading. The first two sentences decide whether the passage is usable.
- State the source and the date. Verifiable figures take precedence over phrasing.
- Check availability directly. A monthly crawler test belongs in the routine, exactly like uptime monitoring.
Related terms
See also AI Mode, AI Overviews, AEO, GEO, llms.txt and AI visibility.
Frequently asked questions
Does RAG mean the model stops making things up?
No. It lowers the chance, because the model has sources at hand. But when the sources are wrong or contradict each other, the error carries into the answer.
How long before new content appears in answers?
Weeks, because it has to be indexed first. Bing tends to be faster, several assistants draw from it, and indexing can be accelerated with the IndexNow protocol.
Can our own content fix wrong claims about the brand?
Yes, provided it is consistent and machine-readable. The model resolves contradictions between sources on its own terms, so identical numbers on the site, in profiles and in directories are cheap insurance.
Is schema enough?
Structured data aids comprehension, but RAG retrieves from the text. Without a usable passage, schema alone will not carry you.
How we can help
Technical availability for crawlers, structured data and content prepared to be cited are one service for us. Details are on the AI visibility agency page.