AI crawler
What is CCBot?
CCBot builds Common Crawl, a free public archive of the web that many AI model makers and researchers train on. Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives.
- Operator
- Common Crawl
- What it does
- Collects pages for AI training
- robots.txt token
CCBot- Verified on rankecho.io, 30 days
- 0 requests
What CCBot did on rankecho.io
Cloudflare verified no request from CCBot to rankecho.io from 17 October to 15 November 2026. In the same days 112 requests used CCBot's user agent without Cloudflare verifying it, and 16 of those asked for secret files such as /.env or compose.yml, which no real crawler needs. Cloudflare checks requests against the operator's published addresses, so an unverified request may come from an address Cloudflare does not know yet, or from someone borrowing the name. Treat them as unconfirmed.
Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16.
How many sites block CCBot
2 of 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block CCBot from the whole site. The sample is small and grows with every audit, so read it as a first look rather than a rate for the web.
Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives. Copies already archived stay in older crawls.
Allow CCBot
User-agent: CCBot Allow: /
Block CCBot
User-agent: CCBot Disallow: /
Put the lines in the robots.txt file at the root of your domain. A more specific group for CCBot replaces the rules under User-agent: * for that crawler.
How to check a visit is really CCBot
Common Crawl publishes the IP ranges CCBot uses. A request outside them is not CCBot.
Source: Common Crawl: CCBot.
Questions about CCBot
What is CCBot?
CCBot builds Common Crawl, a free public archive of the web that many AI model makers and researchers train on.
Should I block CCBot?
Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives. Copies already archived stay in older crawls.
How do I know a visit is really from CCBot?
Common Crawl publishes the IP ranges CCBot uses. A request outside them is not CCBot.
How often does CCBot visit a site?
On rankecho.io, Cloudflare verified no request from CCBot from 17 October to 15 November 2026, although 112 requests used its user agent.
How many sites block CCBot?
2 of the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block CCBot from the whole site. It is a small sample, so read it as a first look, not a web-wide rate.
Related crawlers
Check what your robots.txt allows
The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.
Facts checked 2026-11-16 against Common Crawl's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.
