RankEcho

Crawling and indexing

What is a user-agent token?

A user-agent token is the short name a crawler answers to in robots.txt, such as GPTBot or Googlebot. It is often only part of the full user-agent string the crawler sends. Some tokens, such as Google-Extended, control a use of your content and never appear in a request at all.

How to read it

A crawler follows the most specific group in robots.txt that names its token, and ignores the rest, including the rules under User-agent: *. So a group for GPTBot replaces your general rules for GPTBot; it does not add to them.

Look up each token in the operator's own documentation. A typo in a token makes the group match nothing, silently.

Example from rankecho.io

Tokens seen and unseen on rankecho.io

Google-Extended never shows up in rankecho.io's logs, because Googlebot does the fetching and Google applies the token to how the content is used afterwards.

Crawlers check the rules often. In 30 days Googlebot read rankecho.io's robots.txt 379 times and OAI-SearchBot 94 times, so a change to robots.txt is seen within hours.

Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.

Questions people ask

What is a user-agent token?

A user-agent token is the short name a crawler answers to in robots.txt, such as GPTBot or Googlebot. It is often only part of the full user-agent string the crawler sends. Some tokens, such as Google-Extended, control a use of your content and never appear in a request at all.

Is the user-agent token case-sensitive?

No. Google and most crawlers match tokens without regard to case, but copy the operator's spelling anyway.

Can I block a crawler that has no token?

Not with robots.txt. Use your firewall or CDN rules, matched on verified bot identity rather than the user-agent string.

Related terms

Read more on rankecho.io

See it on your own site

RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.

Definition and example checked 2026-11-16.