Robots.txt Tester
Check whether a crawler is allowed to visit a URL. Fetch a site's robots.txt or paste your own, then test paths against Googlebot, Bingbot, AI crawlers or any user agent.
Checked live from our server — no signup Free, no limits
1. Load robots.txt
2. robots.txt content
3. Test URLs
Results (4)
| URL path | Result | Matching rule |
|---|---|---|
| / | Allowed | No rule matches (allowed by default) |
| /admin/settings | Blocked | Disallow: /admin/ · line 2 |
| /admin/public/help | Allowed | Allow: /admin/public/ · line 4 |
| /products?sessionid=42 | Blocked | Disallow: /*?sessionid= · line 3 |
- Groups
- 2
- Rules
- 4
- Sitemaps
- 1
- Errors
- 0
- Warnings
- 0
Groups
| Line | User-agent | Rules | Crawl-delay |
|---|---|---|---|
| 1 | * | 2 disallow · 1 allow | — |
| 6 | gptbot | 1 disallow · 0 allow | — |
Sitemaps
| Line | Sitemap URL |
|---|---|
| 9 | https://example.com/sitemap.xml |
How to use the Robots.txt Tester
- Enter a website to fetch its live robots.txt, or paste and edit rules in the text box.
- Pick a crawler such as Googlebot or GPTBot, or type a custom user-agent token.
- List the URLs or paths you want to test, one per line.
- Read the Allowed / Blocked result and the rule and line that decided it, then fix any syntax warnings.
Frequently asked questions
How does Google decide which robots.txt rule applies?
Google picks the one group whose User-agent best matches the crawler (falling back to *), then uses the most specific rule — the one with the longest path. If an Allow and a Disallow rule are equally long, Allow wins.
What do * and $ mean in robots.txt?
* matches any sequence of characters, so Disallow: /*.pdf blocks every URL containing .pdf. $ marks the end of the URL, so Disallow: /*.pdf$ blocks URLs that end in .pdf but not /file.pdf?download=1.
What happens if robots.txt returns 404 or 500?
A 4xx response (like 404) means there is no robots.txt, so everything may be crawled. A 5xx or 429 response makes Google treat the whole site as disallowed and pause crawling until the file is reachable again.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of search, allow crawling and add a noindex meta tag or X-Robots-Tag header.
How do I block AI crawlers like GPTBot?
Add a group for each bot, for example "User-agent: GPTBot" followed by "Disallow: /". Use the Google-Extended token to opt out of Google's AI training without affecting Google Search.
Is there a size limit for robots.txt?
Yes. Google reads only the first 500 KiB of the file and ignores the rest, so keep it short and put the most important rules first.