RankEcho

Crawling and indexing

What is the difference between a training crawler and a search crawler?

An AI training crawler, such as GPTBot or ClaudeBot, collects pages to train future models. An AI search crawler, such as OAI-SearchBot, fetches pages so an assistant can find and cite them in answers. Blocking a training crawler does not remove you from AI answers; blocking a search crawler can.

How to read it

Most AI companies now run separate crawlers for separate jobs: one for training, one for search, and a user fetcher that reads a page when a person asks the assistant to.

Decide crawler by crawler. A publisher can block training and still be cited in answers, while blocking the search crawler or the user fetcher can drop the site from that assistant.

Example from rankecho.io

OpenAI's three crawlers on rankecho.io

In 30 days Cloudflare verified 134 requests from GPTBot, the training crawler, 166 from OAI-SearchBot, the search crawler, and 448 from ChatGPT-User, which fetches pages when someone in ChatGPT asks.

ChatGPT-User was the busiest, with 341 requests on a single day, 12 November. Fetches made for people asking about you matter more than training crawls.

Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.

Questions people ask

What is the difference between a training crawler and a search crawler?

An AI training crawler, such as GPTBot or ClaudeBot, collects pages to train future models. An AI search crawler, such as OAI-SearchBot, fetches pages so an assistant can find and cite them in answers. Blocking a training crawler does not remove you from AI answers; blocking a search crawler can.

If I block GPTBot, can ChatGPT still cite me?

Yes. Citations come from OAI-SearchBot and ChatGPT-User fetches, which GPTBot rules do not cover.

Does robots.txt bind user fetchers?

Not always. Some operators say a fetch made for a user is not crawling and does not follow robots.txt.

Related terms

Read more on rankecho.io

See it on your own site

RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.

Definition and example checked 2026-11-16.