Robots.txt Generator

This robots.txt generator builds a valid robots.txt file that lets search engines in while allowing or blocking AI crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended. Pick a preset, adjust individual bots, add your sitemap, then copy or download the file.

robots.txt

# robots.txt
# Generated with the Devma robots.txt generator (devmaagency.com/tools/).
# These rules are advisory: reputable crawlers follow them, others may not.

# All crawlers not named in a group below
User-agent: *
Allow: /

# Following the * rules above: Googlebot, Bingbot, Applebot, DuckDuckBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot

# AI training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

Upload the file to the root of your site so it loads at https://yourdomain.com/robots.txt. Crawlers only read it from the root, and each subdomain needs its own.

robots.txt is a request, not a lock. Reputable crawlers follow it, but some bots ignore it or change their name. To actually stop unwanted crawlers, block them at the server with AI Shield from Devma Hosting.

How to use the Robots.txt Generator

1Choose a preset: allow all, block AI training, block all AI crawlers, or block everything.
2Fine-tune individual search engines and AI bots with the checkboxes.
3Add any paths you want kept out of crawlers (one per line), an optional crawl-delay and your sitemap URLs.
4Copy or download the file and upload it to the root of your site as /robots.txt.
5Check the live file with the AI Crawler Checker to confirm each bot is allowed or blocked as intended.

How robots.txt rules are read

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules for URL paths. A crawler looks for the group that names it most specifically and follows only that group. If no group names it, it falls back to the User-agent: * group.

That detail matters: if you give GPTBot its own group, it ignores everything in the * group, so any paths you want protected must be repeated there. This generator handles that for you. Within a group, the longest matching path wins, so Allow: / plus Disallow: /admin/ opens the whole site except the admin area. Sitemap lines can go anywhere and apply to every crawler.

AI training crawlers versus AI search crawlers

AI companies now run separate bots for separate jobs, and robots.txt lets you treat them differently. Training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect pages to build future models. Search and assistant bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI tools can cite and link to you in answers, while ChatGPT-User and Claude-User fetch a page only when a person asks the assistant to.

Google-Extended and Applebot-Extended are not separate crawlers. They are control tokens: Googlebot and Applebot still crawl your site for search, and the token tells Google or Apple whether that content may be used for their AI models. Blocking them does not affect your search rankings. The Block AI training preset uses these distinctions to keep you visible in search and AI answers while opting out of training.

What robots.txt cannot do

robots.txt is a public, voluntary convention. Reputable crawlers follow it, but nothing enforces it: scrapers can ignore the file, rotate user-agent names or pretend to be a browser. It also does not hide anything. Because anyone can read your robots.txt, listing a secret path there advertises it.

Blocking a URL in robots.txt also does not guarantee it disappears from search. Google can still index a disallowed URL without its content if other sites link to it. To keep a page out of results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header instead. For private content, use authentication. To actually stop unwanted bots, block them at the server or firewall by user agent, IP range and behavior.

Common robots.txt mistakes

The most damaging mistake is a leftover Disallow: / from a staging site, which tells every crawler to skip the whole domain. Check your live file after every launch or migration.

Other frequent problems: blocking CSS and JavaScript folders, which stops Google rendering your pages properly; putting the file in a subfolder instead of the root; and forgetting that each subdomain and protocol needs its own file. Paths are case-sensitive, so /Admin/ and /admin/ are different rules. Google only processes the first 500 KiB of a robots.txt file, so keep it short. Finally, remember that Crawl-delay is ignored by Google; Googlebot adjusts its crawl rate automatically based on how your server responds.

Questions? Answers.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects content for training OpenAI models. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a user asks for them. Blocking GPTBot stops future training crawls but does not remove content that was already collected.

Will blocking Google-Extended hurt my Google rankings?

No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether content crawled by Googlebot can be used to train and ground Gemini models.

Where do I put my robots.txt file?

At the root of each host, so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored, and subdomains such as shop.yourdomain.com need their own file.

Do all bots respect robots.txt?

No. Major search engines and the large AI companies say they honour it, but robots.txt has no enforcement. Some crawlers ignore it or disguise themselves, so use server-side bot blocking for anything you need to stop.

How do I stop Google indexing a page?

Use a noindex robots meta tag or X-Robots-Tag header and leave the page crawlable so Google can see it. A robots.txt Disallow only stops crawling, and the URL can still appear in results if other pages link to it.

More free tools

All free tools →