RankEcho

SEO tool crawler

What is SemrushBot?

SemrushBot crawls the web to collect the link and page data behind Semrush's SEO tools. Semrush says SemrushBot obeys robots.txt and accepts a crawl-delay of up to 10 seconds.

Operator
Semrush
What it does
Builds an SEO tool's index
robots.txt token
SemrushBot
Verified on rankecho.io, 30 days
883 requests

What SemrushBot did on rankecho.io

From 17 October to 15 November 2026, Cloudflare verified 883 requests from SemrushBot, on 30 of the 30 days. It fetched 252 different pages, read robots.txt 345 times, sitemaps 1 time and llms.txt 0 times.

Verified requests88330 of 30 days
Different pages fetched252536 page requests
robots.txt reads345sitemaps 1, llms.txt 0
Requests faking its name0same user agent, not verified
  • Verified requests
Verified SemrushBot requests per day to rankecho.io, 17 October to 15 November 2026
DateVerified requests
Oct 1740
Oct 1840
Oct 1940
Oct 2030
Oct 2160
Oct 2210
Oct 2340
Oct 2430
Oct 2532
Oct 2623
Oct 2744
Oct 2818
Oct 2962
Oct 3051
Oct 3156
Nov 116
Nov 237
Nov 339
Nov 44
Nov 515
Nov 623
Nov 724
Nov 823
Nov 98
Nov 107
Nov 1114
Nov 1213
Nov 1329
Nov 1436
Nov 1519
Pages SemrushBot fetched most
PathRequests
/compare/agencyanalytics-alternatives24
/learn/best-ai-visibility-tools17
/compare/peec-alternatives16
/compare/semrush-aio-alternatives16
/compare/rankecho-vs-agencyanalytics14

Also in those days, 0 requests used SemrushBot's user agent without Cloudflare verifying it.

Source: rankecho.io's Cloudflare logs, visitor requests only, queried 2026-11-16. Cloudflare sampled 300 of the verified requests, so those counts are estimates.

Allowing or blocking SemrushBot

RankEcho's audits do not read the SemrushBot token yet, so there is no count of sites blocking it. The audit checks the AI and search crawlers that decide whether you can be cited.

Semrush says SemrushBot obeys robots.txt and accepts a crawl-delay of up to 10 seconds. Other Semrush tools, such as Site Audit, use their own tokens, so allow or block each one.

Allow SemrushBot

User-agent: SemrushBot
Allow: /

Block SemrushBot

User-agent: SemrushBot
Disallow: /

Put the lines in the robots.txt file at the root of your domain. A more specific group for SemrushBot replaces the rules under User-agent: * for that crawler.

How to check a visit is really SemrushBot

Cloudflare checks requests against Semrush's known addresses and marks real ones as verified bots, which is how this page tells real visits from borrowed names. The user agent alone proves nothing.

Source: Semrush: SemrushBot.

Questions about SemrushBot

What is SemrushBot?

SemrushBot crawls the web to collect the link and page data behind Semrush's SEO tools.

Should I block SemrushBot?

Semrush says SemrushBot obeys robots.txt and accepts a crawl-delay of up to 10 seconds. Other Semrush tools, such as Site Audit, use their own tokens, so allow or block each one.

How do I know a visit is really from SemrushBot?

Cloudflare checks requests against Semrush's known addresses and marks real ones as verified bots, which is how this page tells real visits from borrowed names. The user agent alone proves nothing.

How often does SemrushBot visit a site?

On rankecho.io, SemrushBot made 883 Cloudflare-verified requests from 17 October to 15 November 2026, on 30 of the 30 days. Your site will differ with its size and how often it changes.

Related crawlers

Check what your robots.txt allows

The free crawler policy check reads your robots.txt and says which AI crawlers it lets in. A free audit also asks ChatGPT, Perplexity and Gemini whether they cite you, and marks a missing answer as crawler access when a crawler is blocked.

Facts checked 2026-11-16 against Semrush's documentation. Log numbers from 17 October to 15 November 2026. Read our crawler log study for the method.