robots.txt for AI Crawlers: Control What ChatGPT, Claude & Perplexity Can Read (and How to Check It)

A 2026 guide to robots.txt for AI crawlers: what GPTBot, ClaudeBot, PerplexityBot and Google-Extended are, whether to allow or block them, mistakes that make you invisible to AI, and how to check your robots.txt with a free checker.

robots.txt is one of the oldest files on the web, a plain-text list of instructions at the root of your site that tells crawlers where they can and cannot go. For twenty years it was an SEO housekeeping detail. In 2026 it quietly became one of the most important files for a different reason: it is the first gate that decides whether AI models like ChatGPT, Claude, Gemini and Perplexity are even allowed to read your site. Block the wrong line and you can make your brand invisible to the exact engines shoppers now ask what to buy.

What robots.txt actually does (and what it doesn’t)

robots.txt is a set of polite instructions, not a lock. It lives at one fixed location, https://yourdomain.com/robots.txt, and any crawler that visits reads it first to see which paths it is asked to avoid. Well-behaved crawlers, including the major AI bots, honor it. But it is not security: it does not stop a determined scraper, it does not hide a page from anyone who has the URL, and disallowing a page does not remove it from an index it is already in. Think of it as a sign on the door, not a deadbolt.

The AI crawlers you need to know in 2026

Each major AI company runs one or more named crawlers, and they split roughly into three jobs: crawling the web to train models, indexing pages so the assistant can answer from them, and fetching a page live when a user asks about it. You can allow or block each one independently in robots.txt.

Two things to keep straight. First, Google-Extended is not a crawler that visits your site. It is a token you disallow to opt out of Google using your content for Gemini and Vertex AI, and it does not affect your Google Search ranking. Second, you cannot block Google’s AI Overviews separately from Search: content Googlebot can read may appear in an AI Overview. User-agent tokens change as providers add bots, so always confirm the current names against each provider’s own documentation.

How to read your robots.txt in 60 seconds

The syntax is small. A block starts with one or more User-agent lines naming who it applies to, followed by Disallow and Allow rules for paths. A star matches every crawler. A Sitemap line points crawlers to your sitemap. Here is a minimal file:

User-agent: *
Disallow: /cart
Disallow: /checkout
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Read it top to bottom: the block applies to every crawler, asks them to avoid /cart and /checkout, allows everything else, and tells them where the sitemap is. Rules are path-prefix matches and are case-sensitive. The most common misreading is assuming Disallow means “hide.” It only means “please don’t crawl.”

Should you allow or block AI crawlers?

This is the real decision, and it comes down to your goal. If you want AI assistants to recommend your brand, which is the whole point of answer-engine and generative-engine optimization, you almost always want to allow the AI crawlers, because a model can only cite and recommend what it is allowed to read. Blocking GPTBot or PerplexityBot to “protect” your content also removes you from the answers your buyers are reading. Publishers who sell content, not products, sometimes make the opposite call: block the training crawlers (GPTBot, ClaudeBot, Google-Extended) but keep the live user-fetch bots open so they can still be cited. For a brand that sells things, invisibility is the bigger risk than training.

Common robots.txt mistakes that make you invisible to AI

• A leftover Disallow: / from a staging site shipped to production. This blocks everyone, including every AI crawler, from your entire site.
• Blocking GPTBot, ClaudeBot or PerplexityBot without realizing it removes you from ChatGPT, Claude and Perplexity answers.
• Disallowing /blog, /products or other key content that is exactly what AI needs to read to recommend you.
• Putting robots.txt anywhere but the root. It only works at https://yourdomain.com/robots.txt, and only for that exact host and protocol.
• Blocking CSS or JavaScript that pages need to render, so crawlers see a broken page.
• Assuming robots.txt removes a page from results. To keep a page out of an index, you need a noindex tag, not a Disallow.

A sensible robots.txt that welcomes AI (and hides the junk)

For most product brands, the right default is: let every crawler, search and AI alike, read your content, keep them out of transactional and account pages, and point them to your sitemap. Something like:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /admin
Disallow: /search
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This lets GPTBot, ClaudeBot, PerplexityBot, Googlebot and the rest read your public pages while keeping crawlers out of the pages that only waste crawl budget. If you specifically want to opt out of model training but stay citable, add targeted blocks for the training bots below this. But for most brands that want to be recommended, the open default is the right one.

How to check your robots.txt

The fastest manual check is to open https://yourdomain.com/robots.txt in a browser and read it. If you see Disallow: / on its own, or your key sections blocked, that is a red flag. But a manual read misses the AI-specific question: is each of the AI crawlers actually allowed? A robots.txt checker, sometimes called a robots.txt tester, parses your file, tests it against the real user-agents, and tells you exactly which bots can and cannot reach which paths. Our free robots.txt checker at useauspex.ai/tools/robots-txt-checker does this with the AI crawlers built in. Paste your domain and it flags whether GPTBot, ClaudeBot, PerplexityBot and Google’s bots can read your site, so an accidental block does not quietly cost you AI visibility.

robots.txt is the gate, being recommended is the goal

Getting robots.txt right only earns you the right to be read. It does not make an AI recommend you. Plenty of brands are perfectly crawlable and still never named when a shopper asks what to buy, because being readable and being recommended are different problems. Once you have confirmed the door is open, the next question is whether ChatGPT, Perplexity and Gemini actually name you, and who they name instead. That is what Auspex measures: run a free instant check to see whether AI recommends your brand today, and where the gaps are.