Robots.txt Generator

How it works

robots.txt is a plain text file in the root folder of your site that tells search engine crawlers which paths they should not crawl. Type the paths you want to block into the left pane, one per line. /cart/ stays /cart/, a full address such as https://www.example.com/cart/ is cut down to its path, and a bare name such as cart becomes /cart, which also blocks /cart-help and anything else starting with those letters, so add the trailing slash when you mean the folder. Lines starting with # are comments and do not go into the file. To open one page inside a blocked folder, start the line with Allow:, for example Allow: /cart/deals. If you enter your full sitemap address, a Sitemap line is added at the end; an address without https:// is rejected because the standard requires a full URL. The option to block AI training crawlers writes a separate blocking group for each of seven bots, including GPTBot, ClaudeBot and Google-Extended. Download the file, keep the name robots.txt and upload it to the top folder of your site; on WordPress you can also paste it into the robots.txt editor of an SEO plugin.

This tool is also known as robots.txt generator, robots txt generator, create robots.txt, robots.txt example, block gptbot robots.txt.

What is robots.txt?

robots.txt is a plain text file at the root address of a site that tells crawlers which paths not to visit. It has been in use since 1994 and became a standard in 2022 as RFC 9309. The file is made of groups that start with a User-agent line; each group holds Disallow (block) and Allow (open) rules. Google applies the longest, most specific matching rule.

What is Disallow and Allow?

The line Disallow: /cart/ blocks every address that starts with /cart/; an empty Disallow: line blocks nothing. Allow uses the same form to open a path and serves as an exception inside a blocked folder. * stands for any run of characters and a $ at the end marks where the address ends: Disallow: /*.pdf$ blocks all PDF files.

What is the difference between robots.txt and noindex?

robots.txt stops a crawler from reading a page, but on its own it does not keep the page out of search results: a blocked address that gets links can still be listed without its content. noindex is a tag inside the page that tells search engines not to list it, and the crawler must be able to read the page to see it. Use robots.txt for cart and filter pages that waste crawl budget, and noindex for a thank-you or draft page that should never appear in search; do not use both on the same page.

What robots.txt does and does not do

robots.txt is a request, not a lock. The crawlers of Google, Bing and the large AI companies follow it; a badly behaved bot can download your pages without ever reading the file. Protect a page that must stay private (a customer area, an invoice folder) with a password, not with robots.txt.

A blocked page can still show up in Google: an address that other sites link to can be listed by its URL alone, even though its content was never read. To remove a page from search results completely, put a noindex tag on it and do not block it in robots.txt; if the crawler cannot read the page, it cannot see the noindex either.

Which setting for which situation

For most sites the right file is short: cart, account, search results and admin folders blocked, everything else open. Do not block CSS and JavaScript folders; Google reads them to render your page, and if they are blocked it sees an incomplete page, which can hurt rankings.

On a site under construction, block the whole site, and undo that setting when you launch; a forgotten Disallow: / line is the most common reason a site drops out of search entirely. Blocking AI training crawlers limits the use of your content as training data; the bots in the list are separate from search crawlers, so your place in Google Search is not affected. AI search crawlers such as ChatGPT search and Perplexity are left out of the list on purpose: if you block them, your site is not cited as a source in their answers.

• Blog or brochure site: only /wp-admin/ blocked, Allow: /wp-admin/admin-ajax.php open
• Online shop: /cart/, /account/, /checkout/ and filter addresses (/search?) blocked
• Site under construction: All bots → May not crawl the site

Frequently asked questions

Where do I upload the robots.txt file?

To the top (root) folder of your site, so that it opens at www.example.com/robots.txt. A robots.txt placed in a subfolder (www.example.com/blog/robots.txt) is not read by crawlers. After uploading, open the address in your browser and check that the file shows up.

Do I need to add my sitemap to robots.txt?

It is not required but it is recommended: the Sitemap line lets search engines you have not set up in their webmaster tools find your sitemap too. The address must be complete, starting with https://. On WordPress the sitemap is usually at /wp-sitemap.xml or at the /sitemap_index.xml address given by your SEO plugin.

How do I block GPTBot and other AI crawlers?

Set AI training crawlers to Block. The tool writes separate User-agent groups for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), Applebot-Extended, CCBot (Common Crawl), Meta-ExternalAgent and Bytespider. The list is based on the companies' own documentation as of September 2026 and needs updating as new bots appear.

Is what I type sent to a server?

No. The robots.txt file is built entirely in your browser; your folder names and site address never leave this page.