Robots.txt and Crawl Access for AI Answer Engine Bots

Robots.txt turned thirty years old in the recent past, and for most of that life it did one small job: it told search engines which paths to skip. In 2026 the same file is deciding whether a brand appears in AI-generated answers at all, and it is doing so through a set of rules most teams last reviewed when Googlebot was the only reader that mattered. The file has become the gatekeeper of AI visibility, and the policy it most often encodes is an accident.

Why robots.txt has become the gatekeeper of AI visibility, not just search crawling

The file's original job was narrow. The file was standardized as RFC 9309 in September 2022 after 28 years as a gentleman's agreement, but for the better part of its life it governed a single relationship: the site owner and the search engine that wanted to index pages in exchange for sending traffic back. Both parties had an interest in the arrangement working.

That relationship has been replaced by something closer to a multi-party negotiation, and the file was never built to referee it. But the practical consequence is sharper than most site owners recognize: a file that says nothing about AI bots is still making a policy decision. It is applying the wildcard default to every crawler that now arrives, and the default treats the citation bot and the bulk harvester as if they were the same thing.

The split that changed everything: training crawlers versus answering crawlers

Every major AI provider now documents at least two separate user-agents: one for model training and one for search and citation. The choice is no longer "allow AI or block AI" but "which job am I allowing or blocking," and the answer depends on what each crawler does with the content it collects. Provider documentation supports a three-role taxonomy that makes the decision legible.

The first role is the training crawler. Agents like GPTBot, ClaudeBot, CCBot, and Meta-ExternalAgent collect content to build models, and they send nothing back to the sites they visit. Blocking them costs nothing in day-to-day AI visibility, because they were never going to produce a citation or a referral regardless of what they read.

The second role is the answer and search crawler. Agents like OAI-SearchBot, Claude-SearchBot, and PerplexityBot index content so that AI products can cite it in responses to user queries. Anthropic states that blocking Claude-SearchBot may reduce a site's visibility in user search results. These crawlers are the mechanism by which AI products link back, and blocking them removes the site from that channel entirely.

The third role is the user-triggered fetcher. Agents like ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Google-Agent, Meta-ExternalFetcher, and Amzn-User load a specific page because a person asked an assistant to retrieve it. The distinction matters for what robots.txt can and cannot govern.

Google-Extended sits apart from all three roles because it is a control token, not a separate crawler. Google's documentation states that it governs whether content trains Gemini models and grounds answers in Gemini apps and Vertex AI, and that it "does not impact a site's inclusion in Google Search." Blocking it carries a real trade-off for Gemini grounding that is separate from ordinary search visibility, and teams that treat it as a training-crawler block without understanding the grounding consequence make the decision blind.

What the data from real robots.txt files shows publishers are actually doing

The aggregate pattern is now clear enough to read from the files themselves. Cloudflare Radar's snapshot dated 28 September 2026, parsed across several thousand robots.txt files, shows publishers splitting their rules by crawler job rather than by company. Every training crawler appears in more Disallow rules than Allow rules, while the leading answer and search crawlers are allowed at least as often as they are blocked. The distinction the thesis recommends is already the field's emergent consensus.

The block-to-allow ratios make the split concrete, and the direction of the trend matters more than any single number. Training crawlers like GPTBot and CCBot sit heavily on the blocked side of the ledger, while answer-engine crawlers like OAI-SearchBot sit near parity or on the allowed side. What this says is that publishers have largely figured out, in aggregate, that blocking the training crawl carries no visibility cost and blocking the answer crawl carries a direct, verifiable one.

The crawl-purpose data explains why the consensus formed where it did. Roughly 89.4% of AI crawler traffic is training or mixed-purpose. Blocking the training majority loses nothing in visibility, while allowing the narrow answer-engine minority is what preserves the citation channel.

The structural bug that silently breaks the intent of your robots.txt

The most common structural mistake in AI-era robots.txt files is also the quietest. Under RFC 9309, a crawler obeys only the most specific group that matches it. The result is a file that looks correct to a human reader and behaves differently to a crawler.

This happens because the file's grouping rules create an all-or-nothing branch. The fix is to stack every allowed agent above a single shared rule block, so crawl-trap Disallow lines are written once and bind all of them. This is the RFC 9309-compliant pattern that the rewrite of postman.com/robots.txt applied, and it is the structural correction most files in the wild still need.

The CDN layer is a second, separate failure mode, and it is the one most likely to affect teams who think they are already covered. If a managed WAF rule blocks "all AI crawlers" and the robots.txt sets Allow for OAI-SearchBot, the WAF wins, and the site is still excluded from ChatGPT search. The two layers must agree, or the more permissive one is decorative.

User-triggered fetchers introduce a third gray zone. Perplexity-User is documented by its operator as a user-triggered fetcher that generally ignores robots.txt rules, and other providers describe similar behavior for their user agents. The file's authority stops at the boundary of what the provider classifies as a crawl.

The stakes of getting any of this wrong are measurable. Anthropic's crawl-to-refer ratio stands at 20,583 pages crawled for every single referral sent back to publishers. The failure is precisely backward: the block that would have been free didn't happen, and the block that costs visibility did.

What a correctly structured robots.txt for AI crawlers looks like in 2026

A well-formed AI-aware robots.txt in 2026 organizes crawlers into four explicit classes and writes rules for each, rather than relying on wildcard defaults or per-company groups. The structure follows from the taxonomy established earlier, and each class gets a treatment that matches what the crawler does with the content it reads. The first class is answer engines and retrieval fetchers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, Meta-WebIndexer, Amzn-SearchBot, and MistralAI-Index. These cite and link back, making them the closest thing to referral traffic the AI era has produced. Allow on the root path is the default, with exclusions only for specific paths that genuinely need them.

The second class is model-training crawlers: GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, MistralAI-Training, Applebot-Extended, and Google-Extended. Blocking them costs nothing in day-to-day AI visibility. Either answer is defensible, but what is not defensible is not knowing which answer the current file gives, so the file should make the choice explicit and comment it as policy.

The third class is coding agents and agentic browsers: Claude Code, Cursor, Devin, and Operator. These are low-volume, user-initiated, and increasingly how developers actually consume documentation. If developers are a target audience, this class determines whether their tools can read the site's docs on their behalf, and the default posture should reflect that audience more than any generic security concern.

The fourth class is bulk harvesters: Bytespider, img2dataset, and similar dataset-building downloaders. These are high-volume, produce no citations, and send no traffic back. Block them unconditionally, and for Bytespider specifically, enforce the block at the WAF or IP level rather than relying on robots.txt alone, since the file is advisory and the harvester class often ignores it.

# Answer engines and retrieval fetchers (cite and link back)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Coding agents and agentic browsers (user-initiated, low-volume)
User-agent: Claude Code
Allow: /

# Shared rule block: every agent above reads these lines
User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /internal/

Three agents are deliberately omitted from the template. Google provides a separate mechanism in Google-Extended for opting out of Gemini training specifically, and that is the control to use for training decisions. As of 31 August 2026, the opt-out for AI Overviews and AI Mode went worldwide as a Search Console setting rather than a robots.txt directive, which means this is a control layer robots.txt cannot replicate and should not attempt to.

The file itself should be written for two audiences. The file should state its own policy, cite the RFC, and tell the next maintainer exactly how to flip the training decision: which lines to change, what the consequence is, and why the answer-engine rules must stay untouched. This is a starting point requiring review against the site's own logs, not a drop-in file, because the right answer for any given crawler depends on traffic patterns no template can predict.

Why blocking training crawlers doesn't protect content the way most teams assume

The reflexive move is to block GPTBot and claim the content is protected. Robots.txt is a polite request, not a lock, and non-compliant scrapers face no technical barrier from the file itself. A site that blocks GPTBot and CCBot in robots.txt has expressed a preference, not enforced one, and the distinction determines what the block actually accomplishes.

Compliance is opt-in by design. The major labs document their user-agents and generally respect standard directives: Googlebot, GPTBot, ClaudeBot, and PerplexityBot all publish what they crawl and how to control them. Bulk scrapers often do neither, which means the crawlers most likely to respect a Disallow are also the ones most likely to have a legitimate reason to be there in some other role.

The real cost of getting training-crawler blocking wrong runs in the other direction. Allowing training crawlers means future model versions learn the brand by default, and for a brand still building ground in AI answers, that exposure to training data may matter more than the protection a non-enforced Disallow provides. The question is not whether blocking is philosophically right but whether the decision was made deliberately, with clear eyes about what the block does and does not accomplish. The pixelmojo.io case shows how easy it is to misread cause and effect when training and answering bots are conflated. The claim was withdrawn on 2 October 2026.

The same site later reversed course on its training posture. On 16 May 2026 it moved CCBot, Google-Extended, anthropic-ai, and Applebot-Extended from blocked to allowed, with the commit message stating that "being known to future model generations mattered more to us than blanket protection, for a brand still earning ground in AI answers." Blocking training crawlers is a legitimate choice, but it should be made deliberately and without the assumption that the block is doing protective work it was never designed to do.

How crawl access connects to whether AI answer engines actually cite you

Allowing answer-engine crawlers is the prerequisite for AI citation, not the guarantee of it. A site that allows it is merely eligible. What actually gets cited depends on content structure, authority signals, and third-party corroboration, and no robots.txt directive can supply those.

Each major AI platform applies a distinct sourcing philosophy to the pool of content it can access. Gemini leans heavily on brand-owned websites for its citations, which means a brand's own domain carries disproportionate weight in that engine's answer set relative to third-party coverage. Other engines weight different signals differently, and the variance means a single crawl-access decision produces different outcomes across platforms. The content signals that increase citation probability once crawl access exists are concrete and mostly unglamorous. Notably, llms.txt is not among these signals at scale, which the next section addresses directly.

The measurement consequence follows from the two-condition structure. The file opens the channel. It does not guarantee traffic flows through it.

Where llms.txt and CDN-layer controls fit in the access stack — and where they don't

llms.txt has been widely promoted as the next step in AI visibility, and the evidence does not support it as a broad tactic. Analysis of roughly 300,000 domains found no statistically significant correlation between having an llms.txt file and AI citation frequency, and removing the file as a variable from the predictive model actually improved its accuracy. The file has a role in specific contexts, but it is not a citation lever, and teams should not invest in it as though it were one.

CDN controls are a different case: they are real and additive, but they cut both ways. The same CDN layer can also enforce rules against bots that ignore robots.txt, with Bytespider as the clearest example. The layer can undo a good policy or complete a partial one, depending on how it is configured.

The access stack in 2026 therefore has three layers that must agree. Robots.txt declares intent to cooperative bots, and it is advisory. A brand's AI visibility policy is only as coherent as the agreement between all three layers, and the consistency check across them is the actionable takeaway, not any single technology recommendation.

The industry's trajectory points toward enforcing access at the point of consumption rather than the point of crawl. The pay-per-crawl model, with its site-controlled payment-required status signal, is already a concrete fact in the ecosystem, and it represents the logical end of a progression that robots.txt began when it was still a gentleman's agreement. The file that once told a single crawler which paths to skip is now the declaration layer of a three-part access stack, and the sites that treat it that way are the ones that will be visible in the answers that increasingly substitute for search results.