t04 · SEO~20 min saved per pass

Robots.txt Generator

Build a valid robots.txt file to control how search engines crawl your site, then copy or download it.

Runs entirely in your browser. Nothing is uploaded.

Preset

Let every crawler index everything

Block AI crawlersAdds rules for 13 known AI training and answer-engine bots (GPTBot, ClaudeBot, PerplexityBot, CCBot and more). Google-Extended and Applebot-Extended cover AI training only. Your site stays in Google Search and Siri.

Sitemap URL (optional)

Crawl-delay (seconds) (optional)

robots.txt

User-agent: *
Allow: /

About this tool

/tools/robots-txt-generator

A robots.txt file lives at the root of your domain and tells search engine crawlers which URLs they can request. Get it wrong and you can accidentally hide your whole site, or expose pages you meant to keep private, so it's worth generating cleanly.

Pick a preset, optionally block specific paths, add your sitemap, and download the result straight to your project's root directory.

Frequently asked questions

4 answers

What is a robots.txt file?

robots.txt is a plain-text file placed at the root of your domain (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not access. It's a key part of technical SEO.

Where do I put the robots.txt file?

It must live at the root of your domain, like https://yoursite.com/robots.txt. Crawlers look for it there and nowhere else. Download the file from this tool and upload it to your site's root directory.

Does robots.txt hide pages from Google?

Not exactly. Disallow stops compliant crawlers from fetching a page, but a blocked URL can still appear in results if it's linked elsewhere. To keep a page out of the index entirely, use a noindex meta tag or HTTP header instead.

Should I add my sitemap to robots.txt?

Yes. Adding a Sitemap line helps search engines discover all your important URLs. This generator includes it automatically when you provide a sitemap URL.

More free tools

All tools

Get the next tool first.

We publish a tool when one of our own builds throws off something worth giving away. One email per release; nothing else.