Titles and markup
What is the X-Robots-Tag header?
X-Robots-Tag is an HTTP response header that gives search engines the same instructions as a robots meta tag, such as noindex or nofollow. Because it is a header, it works for files that have no HTML, like PDFs and images, and can be set for whole paths at the server.
How to read it
The crawler must be allowed to fetch the URL to see the header. Blocking the same URL in robots.txt hides the noindex and can leave the page indexed.
Headers are easy to set too widely. Check the response headers of a few public pages after any change.
Example from rankecho.io
Googlebot fetched /login 43 times
rankecho.io's /login page answers with X-Robots-Tag: noindex, follow. In 30 days, Googlebot fetched it 43 times, and each time the header told Google not to index it while still following its links.
That is the intended trade: the page stays crawlable, so Google can read the instruction, and stays out of results.
Source: rankecho.io's Cloudflare logs, verified requests only, 17 October to 15 November 2026.
Questions people ask
What is the X-Robots-Tag header?
X-Robots-Tag is an HTTP response header that gives search engines the same instructions as a robots meta tag, such as noindex or nofollow. Because it is a header, it works for files that have no HTML, like PDFs and images, and can be set for whole paths at the server.
X-Robots-Tag or meta robots: which is better?
Same effect for HTML pages. Use the header for PDFs and other files, or when setting a rule for many paths at once.
Can I use both?
Yes. Search engines combine them and follow the most restrictive.
Related terms
Read more on rankecho.io
See it on your own site
RankEcho reads your Search Console data and asks AI assistants whether they cite you, then shows each of these numbers for your pages. The audit is free.
Definition and example checked 2026-11-16.
