All Tools

Robots.txt Generator

Build a robots.txt file from presets and simple choices: block AI crawlers, keep admin and cart pages out of search, add your sitemap, and test any path before you upload it.

  • PrivateYour content never leaves this device
  • No uploadsProcessed in your browser, not on a server
  • 100% freeNo signup, no watermark, no trial
  • UnlimitedUse it as often as you like, no daily caps

How we protect your privacy →

Start from a preset

Rules for all crawlers

One per line, e.g. /admin/ or /*.pdf$ (a trailing / blocks the whole folder).
Exceptions inside blocked folders.
Optional. Bing and Yandex slow down; Google ignores it.
Full addresses, one per line.

robots.txt

User-agent: *
Allow: /
Valid robots.txt: no problems found.
Upload it to the root of your site so it opens at https://yoursite.com/robots.txt, then check the live file with the Robots.txt Tester. Robots.txt asks crawlers to stay out; it doesn't hide pages from people or remove them from Google (use noindex for that).

Test a path

Allowed by “Allow: /” (line 2)

How to use the Robots.txt Generator

  1. Start from a preset, such as Allow everything, Block AI crawlers or WordPress.
  2. Add the paths to block, any exceptions and your sitemap URL.
  3. Check the validator and test a few paths, then copy or download robots.txt and upload it to your site's root.

Frequently asked questions

How do I block AI crawlers like GPTBot and ClaudeBot?

Tick “Block AI crawlers”. It adds a group for the published AI user agents (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others) with Disallow: /. Well-behaved crawlers respect it; it isn't a technical lock.

Where does robots.txt go?

In the root of your domain, so it opens at https://yoursite.com/robots.txt. Each subdomain needs its own file.

Will Disallow remove a page from Google?

No. It stops crawling, but a blocked page can still be listed if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag.

Does Google support Crawl-delay?

No. Google ignores Crawl-delay; Bing and Yandex use it. Google adjusts its crawl speed to how fast your server responds.