What Robots.txt Generator does
robots.txt is the file crawlers read before anything else on your site, and it decides what they are allowed to fetch. This generator builds one from the paths you want kept out of crawling, the sitemap you want advertised, and — optionally — rules for the AI training crawlers that have appeared in the last few years. The most important thing to understand before using it is that robots.txt controls crawling and not indexing, a distinction that catches out a great many sites and is explained in full below.
Step by step
- 1
Choose whether crawlers are allowed by default, or blocked entirely for a staging site.
- 2
List the paths you do not want crawled, one per line.
- 3
Add your sitemap URL so crawlers can discover it without you submitting it anywhere.
- 4
Tick the AI crawler option if you want to opt out of training datasets, then upload the file to your site root.
Common uses
Keeping admin and checkout pages out of crawls
Pages behind a cart or login have no search value and waste the crawl budget that should go to real content.
Blocking a staging site
A full disallow keeps a staging environment out of search results, though it is not a substitute for real access control.
Opting out of AI training
Rules for GPTBot, CCBot, Google-Extended and similar agents are honoured by the major operators that publish them.
Advertising your sitemap
A Sitemap directive costs nothing and lets any crawler find your sitemap without a manual submission.
Disallow stops crawling, not indexing
This is the single most misunderstood thing about robots.txt, and getting it backwards produces exactly the outcome people were trying to avoid. Disallow tells a crawler not to fetch a URL. It does not tell a search engine not to list it. If other pages link to a disallowed URL, Google can index the URL on the strength of those links alone and show it in results with no title or description, because it was never permitted to fetch the page and see what is there. Worse, because it cannot fetch the page, it can never see a noindex tag you added — so blocking a page in robots.txt actively prevents the mechanism that would have removed it. The correct approach for keeping a page out of search results is the opposite of the intuitive one: allow crawling, and serve a noindex meta tag or X-Robots-Tag header. Reserve Disallow for pages that are genuinely wasteful to crawl, such as faceted search URLs, infinite calendars and internal endpoints.
What robots.txt is not
It is a request, not a control. Nothing in the protocol enforces compliance, and the file is publicly readable by anyone, which means listing a sensitive path in it advertises that path to precisely the people you least want finding it. A directory listed as Disallow: /internal-admin/ is a signpost. Anything that genuinely must not be reached needs authentication or network-level restriction, and robots.txt should never appear in a threat model as a protective measure. Scope is the other common trap: the file applies only to the exact host and protocol it is served from, so a file at example.com/robots.txt has no effect on blog.example.com, and a file placed anywhere other than the root is ignored entirely. Matching rules are also more subtle than they look — the most specific rule wins rather than the first or last one, and a Disallow with an empty value means allow everything rather than block everything.
Practical notes
- To keep a page out of search results, allow crawling and use a noindex tag — blocking it in robots.txt prevents the noindex from ever being seen.
- Never list sensitive paths here; the file is public, so it advertises exactly what you were hoping to hide.
- Test changes in Google Search Console's robots.txt report before deploying — a stray Disallow: / can deindex an entire site.