Robots.txt Generator

List the paths crawlers should stay out of, add your sitemap, and download a finished robots.txt. One tick box adds blocking rules for GPTBot, ClaudeBot, CCBot and nine other AI crawlers.

Runs in your browserFree, no sign-upLast reviewed October 8, 2026

robots.txt

# robots.txt — generated by PlainToolbox

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml

Disallow stops a page being crawled, not indexed. A blocked URL that other sites link to can still appear in results with no description. To keep a page out of the index, allow crawling and serve a noindex meta tag instead.

Quick answers

What should a basic robots.txt contain?: Three lines are enough for most sites: User-agent: * on the first, an empty Disallow: on the second, and Sitemap: followed by your full sitemap URL. That allows every crawler to fetch everything and tells them where the sitemap is. Add Disallow lines only for areas with no search value.
How do I block all crawlers with robots.txt?: Use two lines: User-agent: * then Disallow: /. Every compliant crawler will stop fetching the site. Use this for staging and internal sites only. On a live site it leads to pages dropping out of search results, and it is not access control, so add a password for anything private.
How do I block AI crawlers such as GPTBot and ClaudeBot?: Give each one its own group: User-agent: GPTBot followed by Disallow: /, then the same for ClaudeBot, CCBot, Google-Extended and any others. Ticking Block AI training crawlers here writes twelve such groups. It works only for crawlers that choose to obey robots.txt.
Does Disallow in robots.txt remove a page from Google?: No. Disallow stops the page being crawled, not indexed. If other pages link to it, the URL can still appear in results without a description. To keep a page out of results, leave it crawlable and add a noindex meta tag or an X-Robots-Tag: noindex header.
Where do I put the robots.txt file?: In the root of the host, so it loads at https://example.com/robots.txt. It must be named robots.txt in lowercase and served as plain text. Each subdomain and each protocol needs its own file: the one on example.com does not cover blog.example.com.
Does Google obey Crawl-delay?: No. Googlebot ignores the Crawl-delay line and sets its own crawl rate based on how your server responds. Bing does honour it. Leave Crawl delay at 0 here to omit the directive unless you have a specific crawler in mind that supports it.

What Robots.txt Generator does

Fill in the form and this robots.txt generator writes the complete file: a User-agent: * group with your Allow and Disallow lines, an optional Crawl-delay, an optional block of AI crawler rules, and a Sitemap line. You can copy the result or download it as robots.txt and upload it to the root of your site. The file tells well-behaved crawlers which URLs not to fetch. It does not remove pages from search results and it does not protect anything: a disallowed URL can still be indexed if other sites link to it, and anyone can read the file. Rules here apply to all crawlers at once; the only per-bot groups it writes are the AI crawler ones.

Robots.txt examples generated with this tool

Each result is the file the generator produced for the settings shown. Line breaks are written as vertical bars, and the comment line the tool puts at the top is left out.

Allow everything and list the sitemap

Default policy: Allow all. Paths to disallow: empty. Sitemap URL: https://example.com/sitemap.xml

User-agent: * | Disallow: | Sitemap: https://example.com/sitemap.xml

The Disallow line is empty on purpose. An empty value means nothing is blocked.

The default shop setup

Paths to disallow: /admin/, /cart, /checkout, /*?sessionid= (the values the page loads with)

User-agent: * | Disallow: /admin/ | Disallow: /cart | Disallow: /checkout | Disallow: /*?sessionid= | Sitemap: https://example.com/sitemap.xml

WordPress admin with one exception

Paths to disallow: /wp-admin/. Paths to allow: /wp-admin/admin-ajax.php. Sitemap URL: https://example.com/sitemap_index.xml

User-agent: * | Allow: /wp-admin/admin-ajax.php | Disallow: /wp-admin/ | Sitemap: https://example.com/sitemap_index.xml

The Allow line wins for that one file because it is the longer, more specific rule.

Staging site

Default policy: Block all crawlers entirely. Sitemap URL: empty

User-agent: * | Disallow: /

In this mode the path lists and Crawl delay are ignored, and the tool shows a warning under the output.

Faceted search and a crawl delay

Paths to disallow: search and /*.pdf$. Crawl delay: 10. Preferred host: https://www.example.com. Sitemap URL: empty

User-agent: * | Disallow: /search | Disallow: /*.pdf$ | Crawl-delay: 10 | Host: www.example.com

The slash before search was added and https:// was stripped from the host automatically.

Open to search engines, closed to AI crawlers

Paths to disallow: empty. Block AI training crawlers: ticked

User-agent: * | Disallow: | then twelve groups, from User-agent: GPTBot | Disallow: / through to User-agent: Meta-ExternalAgent | Disallow: / and finally the Sitemap line.

In the first group Disallow has no value (allow all); in the AI groups it is followed by a slash (block all).

AI crawlers the Block AI training crawlers option adds

Ticking the box writes a User-agent group with Disallow: / for each of these twelve names. They do not all do the same job, so read the last column before blocking them wholesale.

User-agent	Operator	What it is used for
GPTBot	OpenAI	Collects pages for training OpenAI models
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user asks for it; not a training crawler
OAI-SearchBot	OpenAI	Builds the index behind ChatGPT search results; not a training crawler
ClaudeBot	Anthropic	Collects pages for training Claude models
Claude-Web	Anthropic	Older Anthropic agent name, kept for coverage
anthropic-ai	Anthropic	Older Anthropic agent name, kept for coverage
CCBot	Common Crawl	Builds the open Common Crawl dataset, which many AI projects train on
Google-Extended	Google	A control token, not a separate bot: opts the site out of Gemini training without affecting Google Search
PerplexityBot	Perplexity	Indexes pages so Perplexity can cite them in answers
Applebot-Extended	Apple	A control token: opts the site out of Apple's AI training while Applebot still crawls for search
Bytespider	ByteDance	ByteDance's crawler, associated with model training
Meta-ExternalAgent	Meta	Collects pages for Meta's AI products

The option is all or nothing. If you want to stay visible in ChatGPT search or Perplexity answers, delete the OAI-SearchBot, ChatGPT-User and PerplexityBot groups from the output before uploading it. Operators add and rename agents often, so check their documentation for names not on this list.

Robots.txt directives and who supports them

Directive	What it does	Support
User-agent	Starts a group and names the crawler it applies to; * means any crawler	All crawlers that read robots.txt
Disallow	A path prefix the crawler should not fetch. Empty means nothing is blocked; / means everything is	All
Allow	An exception inside a disallowed path	Google, Bing and other crawlers following RFC 9309
* and $ in paths	* matches any run of characters; $ anchors the end of the URL	Google and Bing; not guaranteed elsewhere
Sitemap	Full URL of a sitemap or sitemap index. Sits outside the groups and can be repeated	Google, Bing and most search engines
Crawl-delay	Seconds a crawler should wait between requests	Bing and some others; ignored by Google
Host	Names a preferred domain. A non-standard line introduced by Yandex	Ignored by Google and Bing
# comment	Everything after # on a line is ignored	All

Noindex is not a robots.txt directive. Google stopped acting on Noindex lines in robots.txt in 2019.

How crawlers choose which rules to follow

A crawler reads the file, finds the one group whose User-agent line matches it most specifically, and follows only that group. This matters when you add AI rules. GPTBot, seeing a group addressed to GPTBot, obeys that group and ignores everything under User-agent: *. The groups this generator writes for AI crawlers contain only Disallow: /, so that is the complete rule set for those bots.

Inside a group, order does not matter. For Google and Bing the rule with the longest matching path wins, and if an Allow and a Disallow match equally, Allow wins. That is why Allow: /wp-admin/admin-ajax.php overrides Disallow: /wp-admin/ wherever the two lines sit. The generator lists Allow lines first purely so the exceptions are easy to spot.

Paths are matched from the start of the URL path and are case-sensitive. Disallow: /cart blocks /cart, /cart/ and /cart-items alike, because it is a prefix. Add a trailing slash, /cart/, to limit the rule to the folder. Query strings are part of the match, which is what makes a rule such as /*?sessionid= work.

Mistakes that cause real damage

Leaving Disallow: / in place after launch. It is the standard staging rule and the most common reason a new site never appears in search. Check the live file on launch day.
Disallowing a page you want removed from search. The crawler can no longer fetch it, so it never sees your noindex tag and the URL can stay listed. Allow crawling and use noindex.
Blocking CSS and JavaScript folders. Search engines render pages; if they cannot load your styles and scripts they may judge the page as broken or not mobile friendly.
Listing secret paths. The file is public, so Disallow: /internal-reports/ tells everyone where to look. Protect private areas with a login.
Assuming one file covers every subdomain. www.example.com, example.com and shop.example.com are separate hosts and each serves its own file.
Forgetting the trailing slash. Disallow: /blog also blocks /blog-archive and /blogroll.

What this generator does not do

Separate rules per search engine. All your paths go under User-agent: *; there is no way to give Googlebot or Bingbot a group of its own. Add those groups by hand after downloading.
Choosing individual AI crawlers. The tick box adds all twelve or none.
More than one sitemap. The Sitemap URL box produces a single line; add further Sitemap lines to the file yourself.
Testing. It does not fetch your site or check whether a given URL would be blocked. After uploading, open the file in a browser and check it in your search engine's webmaster tools.
Validation of paths. Lines are written as typed, apart from adding a leading slash, so check wildcards and spelling.

Step by step

1
Set Default policy. Allow all crawlers, then add exceptions is right for a live site. Block all crawlers entirely writes Disallow: / and is meant for staging or internal sites.
2
In Paths to disallow, enter one path per line, such as /admin/ or /*?sessionid=. A missing leading slash is added for you.
3
In Paths to allow, add any exceptions inside a disallowed folder, such as /wp-admin/admin-ajax.php.
4
Enter your Sitemap URL in full, including https://. Leave Preferred host blank unless you specifically need a Host line.
5
Tick Block AI training crawlers if you want the twelve AI user-agent groups added, and set Crawl delay above 0 only if you need the directive.
6
Copy or download the robots.txt output and upload it so that it loads at https://yourdomain/robots.txt.

Frequently asked questions

Is a robots.txt file required?

No. If the file is missing and the server returns a 404, crawlers assume everything may be crawled. A file is still worth having to point to your sitemap and to keep crawlers out of carts, internal search results and other endless or duplicate URL spaces.

What is the difference between Disallow: and Disallow: /?

They are opposites. Disallow: with nothing after it blocks nothing, so the whole site may be crawled. Disallow: / blocks every URL on the host. The single slash is the difference between a fully open site and a fully closed one.

Will blocking AI crawlers hide my site from search engines?

Blocking GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended does not change how you appear in ordinary web search, because none of them feed a search index. Blocking OAI-SearchBot and PerplexityBot does remove your pages from ChatGPT search and Perplexity answers, which may be traffic you want.

Can I use wildcards in robots.txt paths?

Yes, for Google and Bing. An asterisk matches any sequence of characters and a dollar sign marks the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks any URL with a sort parameter. Full regular expressions are not supported.

Should I fill in Preferred host?

Almost always no. It writes a Host line, a non-standard directive that Google and Bing ignore. To tell search engines which version of your domain is the main one, use 301 redirects and canonical tags.

How long does it take for a new robots.txt to take effect?

Crawlers cache the file. Google generally refreshes its copy within 24 hours, so a change may take up to a day to be noticed, and longer before its effect shows in search results. Other crawlers keep their own schedules.

Does robots.txt stop scrapers and bad bots?

No. It is a published request that reputable crawlers follow voluntarily. A scraper can ignore it or pretend to be a browser. To actually stop unwanted traffic you need server-side measures such as rate limiting, a firewall rule or authentication.

Is there a size limit for robots.txt?

Google reads the first 500 kibibytes of the file and ignores anything beyond that. Files produced here are a few hundred bytes to a few kilobytes, so the limit only matters if you maintain thousands of rules by hand.