17 AI Crawlers: What to Allow, What to Block — A robots.txt and llms.txt Checklist

Why "Allow AI Crawlers" Is Not a One-Line Setting

The first question teams starting AEO and GEO ask is, "If I open GPTBot in robots.txt, are we done?" But AI companies run crawlers not as one bot but split by purpose. One bot collects at scale for model training, another builds a search index used in answers, and another fetches a page at the moment a user asks about it. Different names mean allow-or-block decisions can be made separately, and also that opening just one and feeling safe leaves gaps.

Three Purposes, and One Exception

Training crawlers collect web content in bulk to build or improve language models. GPTBot, ClaudeBot, and CCBot belong here. Allowing them is closer to a policy choice -- "you may use my content for training" -- and does not guarantee citation for any particular query.

Search and indexing crawlers build the index that AI search services draw on for answers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are the main examples. If they are blocked, your site is unlikely to become a candidate in that service's search-based answers. From a GEO standpoint, these matter more directly than training bots.

User-request agents fetch a page at the moment a user shares a link or the AI needs to verify a source. ChatGPT-User, Claude-User, and Perplexity-User fall here. Some vendors describe how robots.txt applies to this type differently, since it acts on a person's behalf, so check each vendor's documentation.

The exception is the control token. Google-Extended and Applebot-Extended are not separate bots that come crawling; they are signals about whether already-collected content may be used for generative AI training or use. Google states that Google-Extended does not affect inclusion or ranking in Google Search. Confusing this leads many teams to misjudge the setting out of fear of "disappearing from Google's AI."

17 Crawlers at a Glance

Operator	User-Agent	Purpose
OpenAI	GPTBot	Training
OpenAI	OAI-SearchBot	Search / indexing
OpenAI	ChatGPT-User	User request
Anthropic	ClaudeBot	Training
Anthropic	Claude-SearchBot	Search / indexing
Anthropic	Claude-User	User request
Anthropic	anthropic-ai	Legacy training identifier
Perplexity	PerplexityBot	Search / indexing
Perplexity	Perplexity-User	User request
Google	Google-Extended	Control token (generative AI use)
Apple	Applebot	Search / indexing
Apple	Applebot-Extended	Control token (AI training use)
Meta	meta-externalagent	Meta AI-related collection
Meta	FacebookBot	Meta service-related collection
Amazon	Amazonbot	Amazon services (Alexa, etc.)
ByteDance	Bytespider	Training
Common Crawl	CCBot	Public dataset (used in training by many models)

The purpose classification above is a general one based on documentation each operator has published, and policies change frequently. Always check each vendor's latest crawler documentation before applying.

What to Open, What to Block

There is no single right answer, but there is an order of judgment. Search/indexing and user-request bots tie directly to the chance of being cited in AI answers, so if you want citations, leaving them open is the default. Training bots depend on your content licensing stance, your view on competitors benefiting from training use, and how much you want accurate brand information reflected inside models. For a marketing site whose goal is brand exposure, allowing them is usually advantageous; for a site with paid content or proprietary data, selectively blocking only training bots is the realistic setup.

robots.txt Audit — Four Places Teams Get It Wrong

What to Check	Common Problem
Wildcard (User-agent: *) block	Everything is Disallow: / with no per-bot exceptions opened
Bot name spelling	Case or hyphen typos mean the rule never matches (e.g. Claude-SearchBot)
CDN / firewall (WAF)	robots.txt is open but the layer in front of the server blocks AI bot requests
Body rendering method	Access works but JavaScript-only content looks like an empty page to crawlers

The last item matters as much as the allow setting. Crawlers are generally understood not to execute JavaScript, so if they walk through the door you opened and find an empty shell, the chance of citation is effectively zero. We will cover that in the next post.

llms.txt: How Much to Expect

llms.txt is a proposed convention: a markdown file at your site root that guides AI to your key pages so it can understand your structure more easily. It costs little to create, so having one is recommended, but no major AI service is publicly known to have confirmed using it to select citations. The honest positioning is a supporting document that organizes your information structure for humans and machines alike, not a lever that raises citations.

Our Own Site Audit

Our robots.txt allows everything in the wildcard block and additionally lists 12 AI bots explicitly as Allow. But the AI crawler diagnostic in AEGIS Insight checks 17. While auditing the tool, we expanded a list that only covered training bots to include live-answer bots, going from 9 to 17, and in the process confirmed that our own robots.txt does not explicitly name Claude-User, Applebot, meta-externalagent, FacebookBot, Amazonbot, or Bytespider.

Fortunately the wildcard already allows everything, so unnamed bots are not blocked. Two lessons follow. First, when the wildcard is open, per-bot Allow lines are more a statement of intent than a functional necessity. Second, the diagnostic should judge "is it actually not blocked," not "is it written in robots.txt." Insight factors in whether the wildcard blocks everything when determining each bot's allowed status.

Self-Check -- Where to Start Right Now

First open yourdomain/robots.txt and check the wildcard block. Then use AEGIS Insight to check the allowed status of all 17 at once, and cross-check actual response codes in your CDN and firewall logs. That structural diagnostics are a leading indicator of citation readiness, and that no tool yet automatically tallies citations, is covered in our AEO KPI design post.

Closing Thoughts

The core of AI crawler configuration is neither "open everything" nor "block everything." It is distinguishing which bot arrives for which purpose, confirming that search and user-request bots, which tie directly to citation, are not blocked, and deciding on training bots deliberately according to your content policy. And once the door is open, you must still check what crawlers actually read inside for the setting to mean anything.