Robots.txt Generator
Generate a robots.txt file to control how search engines crawl your site.
Worth Knowing
A robots.txt file sits at your domain's root and politely asks crawlers which paths to avoid, but the key word is 'asks' — it's a voluntary convention that well-behaved search engines honor and malicious bots ignore entirely. The dangerous misconception is treating it as security: listing a private directory in robots.txt actually advertises its existence to anyone who reads the file. Use it to manage crawl budget and keep low-value pages out of the index, never to hide sensitive content.
How to Use
- Choose your rule into the Robots.txt Generator above.
- Set any additional options: Paths to disallow (one per line), Sitemap URL (optional).
- The output updates instantly and is ready to copy.