Enter your domain to see if the AI crawlers behind ChatGPT, Claude, Perplexity and Gemini can access your site. If they’re blocked, you can’t be recommended.
Tip: leave the path blank to check your whole site, or enter a path like /blog to test one section.
Paste your domain and this free robots.txt checker (or tester) fetches your live robots.txt, parses every rule the way a real crawler does — most-specific User-agent group, Allow overrides, and wildcards (* and $) — and tests it against 13 crawlers: the AI answer engines (GPTBot and OAI-SearchBot from OpenAI, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Meta’s meta-externalagent and more) plus classic search (Googlebot, Bingbot). For each bot you get a clear allowed, partial or blocked verdict — and the exact rule that caused it — so an accidental Disallow doesn’t quietly cost you AI visibility.
AI answer engines can only recommend what they are allowed to read. If your robots.txt blocks GPTBot or PerplexityBot, ChatGPT and Perplexity can’t crawl your pages — so they can’t cite or recommend your brand. A single leftover Disallow: / from a staging site, or an over-broad rule, is enough to make you invisible in AI answers while your Google rankings still look fine.
Enter a path like /blog or /products in the optional field to check whether a crawler is allowed to reach that exact section. This matters when your robots.txt allows the homepage but blocks a subfolder — the tool applies real longest-match rules to that path and shows the governing Allow or Disallow line, so you know precisely what each engine can and can’t read.
When the checker flags a bot as blocked, it generates a recommended robots.txt block you can copy straight into yourdomain.com/robots.txt to open your site to every AI answer engine while keeping Googlebot and Bingbot allowed. Merge it with any rules you already have, add your Sitemap: line, re-upload the file, then re-run the check to confirm. For the full walkthrough of every AI crawler, see the guide below.
If your robots.txt blocks an AI crawler like GPTBot or PerplexityBot, that engine can’t read your site — so it can’t recommend or cite you. Blocking the wrong bots quietly removes you from AI answers.
For maximum visibility, allow GPTBot & OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (Gemini/AI Overviews). Block only what you have a specific reason to.
It stops OpenAI from crawling your site for training and, in some cases, live answers. Live-browsing features may still reach you, but blocking reduces how reliably ChatGPT can find and cite your content.
Edit your robots.txt to remove the Disallow rule under that crawler’s User-agent (or add Allow: /). Re-run this checker afterward to confirm.
Allow GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini/AI Overviews), Applebot-Extended, Amazonbot and meta-externalagent — each with Allow: / — and keep Googlebot and Bingbot allowed for classic search. This checker generates that exact copy-paste block for your domain after you run it.
Yes. Enter a path such as /blog or /products in the optional path field and the checker applies your robots.txt’s real longest-match rules to that URL, showing whether each crawler is allowed and which Allow or Disallow line governs it.
Competitors, citations and the fixes that move you into the answer — see it in a live demo.
Book a demo