Robots.txt Tester

Check whether a crawler is allowed to visit a URL. Fetch a site's robots.txt or paste your own, then test paths against Googlebot, Bingbot, AI crawlers or any user agent.

  • Checked liveReal-time results fetched from the public internet
  • Nothing storedWe don't keep what you check or the results
  • 100% freeNo signup, no trial, no hidden limits
  • No accountOpen the page and get your answer

How we protect your privacy →

1. Load robots.txt

Try it: We fetch /robots.txt from the site's root. Or skip this and paste or edit the file below.Fair use: up to 20 checks a minute per visitor, so the checks stay fast and free for everyone.

2. robots.txt content

10 lines · 152 B

3. Test URLs

Full URLs work too — only the path and query string are matched.
Rules for Googlebot come from the group on line 1 (User-agent: *). Other groups are ignored for this crawler.

Results (4)

URL pathResultMatching rule
/AllowedNo rule matches (allowed by default)
/admin/settingsBlockedDisallow: /admin/ · line 2
/admin/public/helpAllowedAllow: /admin/public/ · line 4
/products?sessionid=42BlockedDisallow: /*?sessionid= · line 3
Groups
2
Rules
4
Sitemaps
1
Errors
0
Warnings
0

Groups

LineUser-agentRulesCrawl-delay
1*2 disallow · 1 allow—
6gptbot1 disallow · 0 allow—

Sitemaps

LineSitemap URL
9https://example.com/sitemap.xml

How to use the Robots.txt Tester

  1. Enter a website to fetch its live robots.txt, or paste and edit rules in the text box.
  2. Pick a crawler such as Googlebot or GPTBot, or type a custom user-agent token.
  3. List the URLs or paths you want to test, one per line.
  4. Read the Allowed / Blocked result and the rule and line that decided it, then fix any syntax warnings.

Guides for Robots.txt Tester

All →

Frequently asked questions

How does Google decide which robots.txt rule applies?

Google picks the one group whose User-agent best matches the crawler (falling back to *), then uses the most specific rule — the one with the longest path. If an Allow and a Disallow rule are equally long, Allow wins.

What do * and $ mean in robots.txt?

* matches any sequence of characters, so Disallow: /*.pdf blocks every URL containing .pdf. $ marks the end of the URL, so Disallow: /*.pdf$ blocks URLs that end in .pdf but not /file.pdf?download=1.

What happens if robots.txt returns 404 or 500?

A 4xx response (like 404) means there is no robots.txt, so everything may be crawled. A 5xx or 429 response makes Google treat the whole site as disallowed and pause crawling until the file is reachable again.

Does blocking a page in robots.txt remove it from Google?

No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of search, allow crawling and add a noindex meta tag or X-Robots-Tag header.

How do I block AI crawlers like GPTBot?

Add a group for each bot, for example "User-agent: GPTBot" followed by "Disallow: /". Use the Google-Extended token to opt out of Google's AI training without affecting Google Search.

Is there a size limit for robots.txt?

Yes. Google reads only the first 500 KiB of the file and ignores the rest, so keep it short and put the most important rules first.