Tools · Free, no signup
AI crawler checker: can ChatGPT see your site?
Put in a website and see which AI crawlers its robots.txt lets in: the ones that decide whether ChatGPT, Perplexity, Claude, Google and Copilot can cite you, and the ones that only train. Then whether the page can be quoted at all.
Put in a website and press the button. It reads the site’s robots.txt and the page, the way the crawlers do, and tells you who can read it.
What it checks
It reads the site's robots.txt and works out, for 15 crawlers, whether each one may fetch the page you entered. Then it reads the page itself, for a noindex or nosnippet instruction and for how much of its text is there before any JavaScript runs.
The crawlers come in three groups, because blocking each one costs something different. Search crawlers decide whether an assistant can cite you. User fetchers read a page when a person asks an assistant about it. Training crawlers collect pages for future models, and blocking them doesn't affect citations at all.
Search crawlers and training crawlers are different bots
OpenAI runs OAI-SearchBot for ChatGPT search and GPTBot for training, and treats them as separate settings, so a site can stay in ChatGPT's answers while keeping its pages out of training. Perplexity says PerplexityBot is for search and isn't used to train models. Anthropic runs Claude-SearchBot for search and ClaudeBot for training.
Google is the odd one out. Its AI Overviews and AI Mode use the same Googlebot crawl as the rest of Search. Google-Extended, the control people block to opt out of Google's AI, covers training Gemini, and Google says it doesn't affect a site's inclusion in Search.
How a verdict is worked out
The same way the crawlers do it, following the robots.txt standard, RFC 9309. A crawler obeys the groups that name it, or the group for every crawler (User-agent: *) if none does. Within those, the longest matching rule wins, and an allow beats a disallow of the same length. No matching rule means allowed.
How the file answers matters too. A missing file, or any other 4xx, means no rules, so everything is allowed. A server error, a 429 or no answer at all tells a careful crawler to stay out until it can read the file, so the checker reports every crawler as blocked.
What it can't see
A firewall can block a crawler that robots.txt allows. The checker only sees how the site treats its own request, so if the page turns it away, it says so, and that's often a sign AI crawlers are turned away too. It can't see the IP-based rules a CDN applies to each named crawler.
Being allowed in isn't being cited. Whether an assistant quotes you depends on whether the page ranks for the questions it searches and says something worth quoting. For whether it's actually happening, Search Console has a Generative AI performance report for Google, and Bing Webmaster Tools has an AI Performance report for Copilot.
What 229 sites' files showed
Using the same rules, I read the robots.txt files of 229 sites on 16 October 2026. Not one blocked an AI search crawler. Thirteen blocked at least one training crawler, and all six that blocked GPTBot still let OAI-SearchBot in. The full study is in the guide to getting cited by AI.
Questions people actually ask
answered properly,
not padded out
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two as independent settings. Block GPTBot to stay out of training; leave OAI-SearchBot allowed to stay in ChatGPT's answers.
If I block Google-Extended, will I disappear from AI Overviews?
No. AI Overviews and AI Mode use Googlebot, the same crawler as the rest of Google Search. Google says Google-Extended doesn't affect a site's inclusion in Search. It controls whether your pages train Gemini and ground some of Google's other AI products.
How do I let OAI-SearchBot or PerplexityBot in?
If your robots.txt blocks every bot with a rule under User-agent: *, give each search crawler its own group above it, such as "User-agent: OAI-SearchBot" followed by "Allow: /". A crawler with a group of its own ignores the * group. OpenAI says a change can take about a day to reach its systems.
Why does it say my page turned the check away?
The site answered our request with an error such as 403, usually because a firewall or bot protection blocks visitors it doesn't recognise. That doesn't prove AI crawlers are blocked, but firewalls that do this often block them too. Check your CDN's bot settings. OpenAI and Perplexity publish the IP ranges of their search crawlers so they can be let through.
Do I need an llms.txt file?
No. Google says Search ignores llms.txt and that creating one neither helps nor harms, and no AI company has said it reads one for answers or citations. The checker notes whether you have one, for information only.
Why does the text-without-JavaScript check matter?
Because most AI crawlers don't run JavaScript. Vercel studied the crawlers on its network in December 2024 and found that none of the major ones from OpenAI, Anthropic and Perplexity did. If a page's text only appears after a script runs, those crawlers may see almost nothing. Google's crawler does run JavaScript.
Does the checker store the sites I check?
No database, no account. A result is kept in the server's memory for ten minutes so a repeat check is instant, and like every request to this site, the check passes through Vercel's logs. The site you checked goes in the page address, so you can share the result.