Robots.txt
generator.
Build a standards-compliant robots.txt with full control over search engine and AI crawlers, or paste an existing one to check for errors and warnings. Free, and everything stays in your browser.
Optional. Google ignores Crawl-delay; Bing and Yandex respect it.
Upload this file to the root of your domain (e.g. example.com/robots.txt). The file must be served as plain text at the exact path /robots.txt. Changes take effect next time crawlers request it.
What is robots.txt?
Robots.txt is a plain-text file placed at the root of a website that tells search engine crawlers which URLs they are and aren't allowed to access. It follows the Robots Exclusion Protocol, a de-facto standard every major search engine respects. While it does not prevent pages from being indexed (use a noindex meta tag for that), it is the primary mechanism for managing crawl budget and keeping private or low-value paths out of a crawler's queue.
A well-configured robots.txt helps search engines spend their crawl budget on the pages that matter most to your rankings. Blocking paths like /admin/, /search, and duplicate filtered views prevents crawl waste, while leaving CSS and JavaScript accessible ensures Google can render your pages correctly. Including a Sitemap directive gives crawlers a direct path to your most important URLs.
With the rise of AI-powered search and LLM training, robots.txt has gained a new role: controlling access by AI crawlers like GPTBot, CCBot, and Google-Extended. Blocking training crawlers while allowing AI search bots (like ClaudeBot or PerplexityBot) lets your content appear in AI-powered answers without being used to train models. This generator helps you set up both traditional and AI crawler rules in seconds.
Need help with technical SEO?
Robots.txt is just one piece of the crawlability puzzle. I audit and fix crawl issues, indexation gaps, site speed problems and more so search engines see your best content first.
Technical SEO services ↗ Get in touch ↗