Crawling and indexing
How often do crawlers visit a site?
Crawl frequency is how often a crawler fetches your site and each page. It depends on the crawler, how often your pages change, how popular they are and how fast your server answers. Search engines visit most sites daily; AI crawlers come in waves and may stop for weeks.
How to read it
Frequency is per page as much as per site. Your home page and fresh pages may be fetched daily while old pages wait weeks. A sitemap with honest lastmod dates and internal links to new pages speed them up.
For AI crawlers, a quiet spell is normal. Training crawlers run in campaigns, and fetchers come only when a person asks. Judge them over a month, not a week.
Example from rankecho.io
Googlebot every day, ClaudeBot in one burst
Googlebot made verified requests to rankecho.io on 30 of 30 days, 1,751 in all. Amazonbot came on 30 days with 5,914 requests.
ClaudeBot came on 16 days, then sent no verified request after 5 November. That is a crawl campaign ending, not a block: its robots.txt group allows it.
Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.
Questions people ask
How often do crawlers visit a site?
Crawl frequency is how often a crawler fetches your site and each page. It depends on the crawler, how often your pages change, how popular they are and how fast your server answers. Search engines visit most sites daily; AI crawlers come in waves and may stop for weeks.
How can I get Google to crawl a page sooner?
Link to it from pages Google already visits often, list it in your sitemap with its real lastmod, and request indexing in Search Console for important pages.
Why did an AI crawler stop visiting?
Most often its campaign ended. Check your robots.txt and firewall too, but a pause of weeks is common.
Related terms
Read more on rankecho.io
See it on your own site
RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.
Definition and example checked 2026-11-16.
