Robots.txt Tester
Check whether a crawler is allowed to visit a URL. Fetch a site's robots.txt or paste your own, then test paths against Googlebot, Bingbot, AI crawlers or any user agent.
- Checked liveReal-time results fetched from the public internet
- Nothing storedWe don't keep what you check or the results
- 100% freeNo signup, no trial, no hidden limits
- No accountOpen the page and get your answer
1. Load robots.txt
2. robots.txt content
3. Test URLs
Results (4)
| URL path | Result | Matching rule |
|---|---|---|
| / | Allowed | No rule matches (allowed by default) |
| /admin/settings | Blocked | Disallow: /admin/ · line 2 |
| /admin/public/help | Allowed | Allow: /admin/public/ · line 4 |
| /products?sessionid=42 | Blocked | Disallow: /*?sessionid= · line 3 |
- Groups
- 2
- Rules
- 4
- Sitemaps
- 1
- Errors
- 0
- Warnings
- 0
Groups
| Line | User-agent | Rules | Crawl-delay |
|---|---|---|---|
| 1 | * | 2 disallow · 1 allow | — |
| 6 | gptbot | 1 disallow · 0 allow | — |
Sitemaps
| Line | Sitemap URL |
|---|---|
| 9 | https://example.com/sitemap.xml |
How to use the Robots.txt Tester
- Enter a website to fetch its live robots.txt, or paste and edit rules in the text box.
- Pick a crawler such as Googlebot or GPTBot, or type a custom user-agent token.
- List the URLs or paths you want to test, one per line.
- Read the Allowed / Blocked result and the rule and line that decided it, then fix any syntax warnings.
Guides for Robots.txt Tester
All →
SEO & MarketingHow to make a Wi-Fi QR code so guests join without typing
Make a WiFi QR code so guests join your network by pointing a phone camera at it. The format, hidden networks, special characters, safety…
12 Nov 20266 min read
SEO & MarketingWhy WhatsApp link previews show no image, and the fix
WhatsApp link preview not showing image? See the usual causes (missing og:image, a heavy file, a relative URL, a blocked bot) and how to fix…
12 Nov 20265 min read
SEO & MarketingOpen Graph vs Twitter Card tags: do you need both?
Twitter Card vs Open Graph: X reads your og: tags, so you only need one twitter:card line on top. See which tags to keep, with a…
12 Nov 20265 min readFrequently asked questions
How does Google decide which robots.txt rule applies?
Google picks the one group whose User-agent best matches the crawler (falling back to *), then uses the most specific rule — the one with the longest path. If an Allow and a Disallow rule are equally long, Allow wins.
What do * and $ mean in robots.txt?
* matches any sequence of characters, so Disallow: /*.pdf blocks every URL containing .pdf. $ marks the end of the URL, so Disallow: /*.pdf$ blocks URLs that end in .pdf but not /file.pdf?download=1.
What happens if robots.txt returns 404 or 500?
A 4xx response (like 404) means there is no robots.txt, so everything may be crawled. A 5xx or 429 response makes Google treat the whole site as disallowed and pause crawling until the file is reachable again.
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of search, allow crawling and add a noindex meta tag or X-Robots-Tag header.
How do I block AI crawlers like GPTBot?
Add a group for each bot, for example "User-agent: GPTBot" followed by "Disallow: /". Use the Google-Extended token to opt out of Google's AI training without affecting Google Search.
Is there a size limit for robots.txt?
Yes. Google reads only the first 500 KiB of the file and ignores the rest, so keep it short and put the most important rules first.


