Robots.txt Generator
Create a valid robots.txt in a few clicks.
What robots.txt controls
Robots.txt tells well-behaved crawlers which parts of your site they may request. It is the first thing search engines read, and it is also where you declare your sitemap location — a line every site should have.
Start from "allow everything" and only add disallow rules for areas that add no search value: admin panels, cart or checkout URLs, internal search results, and API endpoints. Blocking everything is meant for staging sites only — note that disallowed pages can still be linked to by others.
Blocking AI crawlers such as GPTBot or CCBot is increasingly common for site owners who do not want their content used for model training. List them in the "block bots" field and separate rules will be generated.
FAQ
Does robots.txt remove pages from Google?
No — it prevents crawling, not indexing. To truly de-index a page use a noindex meta tag or the URL removal tool. Pages blocked by robots.txt can still appear in results if other pages link to them.
Where does robots.txt go?
Upload it to your domain root so it is served at https://example.com/robots.txt. One file per subdomain.
Is crawl-delay respected?
Google ignores it (it manages load automatically); Bing and some other crawlers honor it. Use it only if you see real server-load problems.