Which AI crawlers should I allow in robots.txt?

At minimum: GPTBot and OAI-SearchBot (ChatGPT), Google-Extended (Gemini), ClaudeBot and anthropic-ai (Claude), and PerplexityBot (Perplexity). Grok doesn't publish a dedicated crawler — it draws on live X and web search, so staying crawlable by Bing and Google covers it.

How do I check if my site is crawlable by AI engines?

Fetch your robots.txt and test each AI user-agent against its rules, confirm your pages return 200 to those agents without JavaScript, and check that firewalls or bot protection aren't silently blocking them. AEO Track's Crawlability page runs these checks for your tracked site and its AI crawler setup page generates the exact robots.txt block to paste.

Does blocking AI crawlers protect my content?

It keeps your pages out of AI answers — including the ones your buyers ask. Blocking is a legitimate editorial choice, but for most businesses the trade is invisibility at the exact moment someone is deciding what to buy. Decide it deliberately per crawler, not by accident through a stale robots.txt.

← Back to Blog

Guide August 31, 2026 6 min read

AI Crawlability: Can ChatGPT and Claude Actually Read Your Site?

Before any AI engine can cite you, its crawler has to be able to fetch your pages. The robots.txt allowlist, the checks that matter, and how to verify GPTBot, ClaudeBot and PerplexityBot access.

Every conversation about AEO — schema, content structure, citations — rests on one unglamorous precondition: the AI engine's crawler can fetch your pages at all. If GPTBot gets a 403 from your firewall, nothing else you do matters to ChatGPT.

The failure is silent

A blocked human visitor complains. A blocked crawler just leaves. Sites routinely block AI crawlers by accident — a CDN's bot protection with default settings, a robots.txt written years ago with a blanket Disallow, a WAF rule someone added during a scrape scare — and the only symptom is an absence: you never show up in AI answers, and there's no error to find.

The crawlers that matter

Crawler	Engine	Why allow it
GPTBot, OAI-SearchBot	ChatGPT	Training and live search retrieval
Google-Extended	Gemini / AI Overviews	Gating for Google's generative answers
ClaudeBot, anthropic-ai	Claude	Retrieval for Claude's web answers
PerplexityBot	Perplexity	The most citation-forward engine there is

Grok publishes no dedicated crawler — it leans on live X and web search, so ordinary Bing/Google crawlability covers it.

A five-minute verification

1. Read your own robots.txtFetch /robots.txt and check each agent above against its rules. An old blanket rule outranks your good intentions.

2. Check what a crawler receivesAI crawlers don't run JavaScript. If your content only exists after a framework boots, the crawler reads an empty shell. Server-rendered or static HTML wins here.

3. Look for the invisible wallCDN bot management and WAF rules act before robots.txt is even consulted. Verify the AI user-agents get 200s, not challenge pages.

4. Watch them actually arriveCrawler visits appear in server logs, not analytics — JavaScript-based analytics can't see them at all. AEO Track's Crawlability page checks your tracked site's access rules, and its AI crawler setup page generates the exact robots.txt allowlist to paste.

The analytics blind spot

GA4 and similar tools run in the browser; crawlers don't execute them. "No AI crawler traffic in analytics" is what you'd see whether crawlers visit or not. Only server-side evidence counts.

Is anything blocking the engines from reading you?

The free audit checks AI crawler access along with your content and schema.

Start the free 14-day trial →