RankEcho

Search engine crawler

What is Googlebot?

Googlebot crawls pages for Google Search, which includes AI Overviews and AI Mode. Blocking Googlebot removes your pages from Google Search, and with it from AI Overviews and AI Mode.

Operator
Google
What it does
Search engine crawler
robots.txt token
Googlebot
Verified on rankecho.io, 30 days
1,751 requests

What Googlebot did on rankecho.io

From 17 October to 15 November 2026, Cloudflare verified 1,751 requests from Googlebot, on 30 of the 30 days. It fetched 217 different pages, read robots.txt 379 times, sitemaps 15 times and llms.txt 1 time. 182 of its requests got an error status (4xx or 5xx).

Verified requests1,75130 of 30 days
Different pages fetched2171,230 page requests
robots.txt reads379sitemaps 15, llms.txt 1
Requests faking its name847same user agent, not verified
  • Verified requests
Verified Googlebot requests per day to rankecho.io, 17 October to 15 November 2026
DateVerified requests
Oct 1740
Oct 1810
Oct 1950
Oct 2020
Oct 2160
Oct 2230
Oct 2340
Oct 2434
Oct 2571
Oct 2622
Oct 2783
Oct 2821
Oct 2915
Oct 3075
Oct 3157
Nov 183
Nov 288
Nov 366
Nov 4128
Nov 560
Nov 696
Nov 763
Nov 8102
Nov 955
Nov 1065
Nov 1164
Nov 1258
Nov 1350
Nov 1486
Nov 1559
Pages Googlebot fetched most
PathRequests
/product60
/47
/login43
/tools/geo-fix-preview31
/benchmark/category/pastry/nyc23

Also in those days, 847 requests used Googlebot's user agent without Cloudflare verifying it, and 114 of those asked for secret files such as /.env or compose.yml, which no real crawler needs.

Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16. Cloudflare sampled 295 of the verified requests, so those counts are estimates.

How many sites block Googlebot

0 of 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block Googlebot from the whole site. The sample is small and grows with every audit, so read it as a first look rather than a rate for the web.

Blocking Googlebot removes your pages from Google Search, and with it from AI Overviews and AI Mode. To keep Search but opt out of Gemini model use, block Google-Extended instead.

Allow Googlebot

User-agent: Googlebot
Allow: /

Block Googlebot

User-agent: Googlebot
Disallow: /

Put the lines in the robots.txt file at the root of your domain. A more specific group for Googlebot replaces the rules under User-agent: * for that crawler.

How to check a visit is really Googlebot

Run a reverse DNS lookup on the IP: it should end in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name should return the same IP. Google also publishes its crawler IP ranges.

Source: Google: Google's common crawlers.

Questions about Googlebot

What is Googlebot?

Googlebot crawls pages for Google Search, which includes AI Overviews and AI Mode.

Should I block Googlebot?

Blocking Googlebot removes your pages from Google Search, and with it from AI Overviews and AI Mode. To keep Search but opt out of Gemini model use, block Google-Extended instead.

How do I know a visit is really from Googlebot?

Run a reverse DNS lookup on the IP: it should end in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name should return the same IP. Google also publishes its crawler IP ranges.

How often does Googlebot visit a site?

On rankecho.io, Googlebot made 1,751 Cloudflare-verified requests from 17 October to 15 November 2026, on 30 of the 30 days. Your site will differ with its size and how often it changes.

How many sites block Googlebot?

0 of the 13 sites whose robots.txt RankEcho read in audits from 3 November 2026 to 16 November 2026 block Googlebot from the whole site. It is a small sample, so read it as a first look, not a web-wide rate.

Is there a separate crawler for AI Overviews?

No. AI Overviews and AI Mode use pages Googlebot crawled for Search. Google-Extended does not control them.

Related crawlers

Check what your robots.txt allows

The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.

Facts checked 2026-11-16 against Google's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.