RankEcho

AI crawler

What is CCBot?

CCBot builds Common Crawl, a free public archive of the web that many AI model makers and researchers train on. Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives.

Operator
Common Crawl
What it does
Collects pages for AI training
robots.txt token
CCBot
Verified on rankecho.io, 30 days
0 requests

What CCBot did on rankecho.io

Verified requests030 days
Requests using its name112not verified by Cloudflare
Secret-file probes among them16/.env, compose.yml and similar

Cloudflare verified no request from CCBot to rankecho.io from 17 October to 15 November 2026. In the same days 112 requests used CCBot's user agent without Cloudflare verifying it, and 16 of those asked for secret files such as /.env or compose.yml, which no real crawler needs. Cloudflare checks requests against the operator's published addresses, so an unverified request may come from an address Cloudflare does not know yet, or from someone borrowing the name. Treat them as unconfirmed.

Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16.

How many sites block CCBot

2 of 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block CCBot from the whole site. The sample is small and grows with every audit, so read it as a first look rather than a rate for the web.

Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives. Copies already archived stay in older crawls.

Allow CCBot

User-agent: CCBot
Allow: /

Block CCBot

User-agent: CCBot
Disallow: /

Put the lines in the robots.txt file at the root of your domain. A more specific group for CCBot replaces the rules under User-agent: * for that crawler.

How to check a visit is really CCBot

Common Crawl publishes the IP ranges CCBot uses. A request outside them is not CCBot.

Source: Common Crawl: CCBot.

Questions about CCBot

What is CCBot?

CCBot builds Common Crawl, a free public archive of the web that many AI model makers and researchers train on.

Should I block CCBot?

Blocking CCBot keeps future pages out of new Common Crawl archives, and with them out of models trained on those archives. Copies already archived stay in older crawls.

How do I know a visit is really from CCBot?

Common Crawl publishes the IP ranges CCBot uses. A request outside them is not CCBot.

How often does CCBot visit a site?

On rankecho.io, Cloudflare verified no request from CCBot from 17 October to 15 November 2026, although 112 requests used its user agent.

How many sites block CCBot?

2 of the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block CCBot from the whole site. It is a small sample, so read it as a first look, not a web-wide rate.

Related crawlers

Check what your robots.txt allows

The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.

Facts checked 2026-11-16 against Common Crawl's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.