Free Robots.txt Generator
Need a robots.txt file for your site? This free robots.txt generator builds one in seconds. Choose what crawlers can access, block private folders, add your sitemap and copy the finished file.
What Is A Robots.txt File?
A robots.txt file is a small text file placed in the root of your website, for example example.com/robots.txt. It tells search engine crawlers which pages or folders they may crawl. It helps you keep crawlers out of admin areas, cart pages and other low-value URLs, and it points them to your sitemap.
How To Create A Robots.txt File
- Choose the default rule. Most sites should allow all pages.
- Add any folders or pages you want to block, one per line, each starting with a slash.
- Paste your sitemap URL.
- Click Generate Robots.txt, then click Copy Robots.txt.
- Save the text as robots.txt and upload it to the root folder of your site (public_html on most hosts).
Robots.txt Tips
- Do not block CSS or JavaScript files that Google needs to render your pages.
- Robots.txt controls crawling, not indexing. Use a noindex tag to keep a page out of search results.
- Google ignores the crawl-delay rule, although some other search engines read it.
- Test your file in Google Search Console after you upload it.
- Only block AI crawlers if you want to opt out of those bots. They are separate from Google Search.
More Free SEO Tools
Try these tools next: Meta Tag Generator, Open Graph Tag Generator. You can also browse all our free SEO tools. If you want expert help with the rest of your site, see our on-page SEO services.
Frequently Asked Questions
Where do I upload my robots.txt file?
Upload it to the root folder of your domain so it loads at yourdomain.com/robots.txt. Crawlers do not look for it anywhere else.
Will robots.txt remove a page from Google?
No. It only stops crawling. A blocked page can still appear in results if other sites link to it. Use a noindex tag instead.
Should I block AI crawlers?
That is your choice. Blocking them asks those bots not to crawl your content, but it does not affect how Google Search crawls your site.
Does Google use the crawl-delay rule?
No. Google ignores crawl-delay, but Bing and some other crawlers may follow it.
