The AI crawler robots.txt compliance benchmark has become one of the most pressing concerns for publishers and SEO professionals in 2026 — because knowing which bots actually honor your directives is the difference between controlling your content and losing it. This scored evaluation of 12 major AI crawlers tests robots.txt Disallow enforcement, noindex handling, Crawl-delay respect, and documentation transparency, giving you the clearest picture yet of which companies follow the rules and which ones treat your robots.txt as a suggestion.

Methodology and AI Crawler Robots.txt Compliance Benchmark Criteria

This benchmark evaluates 12 AI crawlers across five dimensions: Disallow enforcement (does the bot stop crawling blocked paths?), noindex meta tag respect (does it honor the noindex directive on individual pages?), Crawl-delay compliance (does it throttle requests as instructed?), opt-out documentation quality (how clearly does the company explain exclusion options?), and user-agent consistency (does the bot reliably identify itself with a stable, documented string?).

Scores are assigned on a 0–10 scale per dimension, producing a composite out of 50, then expressed as a final score out of 10. Testing observations drew on community-reported crawl logs, server-side access analysis shared in technical SEO forums, and direct review of each company's public documentation as of Q3 2026. Where direct verification was impossible, scores reflect documented policy quality and known enforcement history. Crawlers with no public documentation received a documentation score of 0 by default.

The five scoring dimensions matter because they represent distinct layers of control. A bot can honor Disallow perfectly while ignoring Crawl-delay, hammering your server with compliant but overwhelming traffic. Equally, a bot might skip restricted paths but still index pages via cached data or partner pipelines. Understanding each dimension separately — not just an aggregate — is what makes this benchmark actionable for real publishing decisions. For a full breakdown of how to write directives for each bot type, see our guide to AI crawler directives robots.txt.

"An FT Strategies analysis of large publishers found that 60 of 70 publishers explicitly restricted at least one AI crawler in robots.txt as of July 14, 2026 — a signal that the industry has moved from curiosity to active enforcement."

That broad adoption of AI blocking directives also means the stakes for compliance are rising. When the majority of major publishers are actively issuing restrictions, a crawler's willingness to honor them is no longer a minor technical footnote — it is a core trust signal that affects whether companies will permit that AI system access to their content ecosystems at all.

Master Comparison Table: All 12 Crawlers Scored

The table below scores each crawler on the five benchmark dimensions and produces a composite final score. Scores reflect both observed behavior (where verifiable) and documented policy commitment. A score of 10 in any dimension means the bot fully and consistently honors that directive with clear verification. A score of 0 means the directive is entirely ignored or completely undocumented.

Bot / User-Agent Disallow Enforcement (0–10) Noindex Respect (0–10) Crawl-Delay Compliance (0–10) Documentation Quality (0–10) User-Agent Consistency (0–10) Final Score (0–10)
GPTBot (OpenAI) 9 7 6 9 9 8.0
ClaudeBot (Anthropic) 9 8 7 8 9 8.2
Google-Extended (Google) 10 9 8 10 10 9.4
PerplexityBot (Perplexity AI) 7 5 5 6 7 6.0
Amazonbot (Amazon) 8 6 5 7 8 6.8
Meta-ExternalAgent (Meta) 7 5 4 6 7 5.8
Applebot-Extended (Apple) 8 7 6 7 8 7.2
Bytespider (ByteDance / TikTok) 5 3 3 3 5 3.8
FacebookBot (Meta / link previews) 7 5 5 6 7 6.0
CCBot (Common Crawl) 6 3 4 5 6 4.8
DiffBot 5 4 4 4 6 4.6
YouBot (You.com) 7 5 5 5 7 5.8

The scoring gap between the top tier (Google-Extended, ClaudeBot, GPTBot) and the bottom tier (Bytespider, DiffBot, CCBot) is striking. The separation is driven primarily by documentation quality and noindex respect — dimensions where the leading bots invest heavily and the laggards barely engage. Crawl-delay compliance is the weakest dimension across the board, with even the top performers leaving room for improvement.

Top Performers: GPTBot, ClaudeBot, and Google-Extended

Google-Extended (Score: 9.4/10)

Google-Extended is the benchmark's highest scorer and the clearest example of what full compliance looks like in practice. Introduced as a dedicated token to separate AI training crawls from indexing crawls, it enforces Disallow rules with the same reliability as Googlebot itself. Its documentation is thorough, publicly updated, and cross-referenced with Google Search Central guidance. The separation between Google-Extended (AI training) and Googlebot (search indexing) is the key architectural decision that earns it a perfect 10 for user-agent consistency — publishers always know exactly what they are blocking and why.

Crawl-delay compliance is its only meaningful weakness relative to the other dimensions, scoring an 8 rather than 10 because Google's infrastructure tends to self-regulate crawl rate rather than explicitly follow publisher-specified delays in the traditional sense. In practice this rarely causes server strain, but it means publishers cannot rely on a declared Crawl-delay directive as a hard brake. Pros: best-in-class documentation, reliable Disallow enforcement, clear noindex integration. Cons: Crawl-delay handling is internally managed rather than directive-driven; blocking Google-Extended may affect Gemini and AI Overviews eligibility.

ClaudeBot (Score: 8.2/10)

Anthropic's ClaudeBot earns the second-highest composite score largely because Anthropic has been unusually transparent about its crawling philosophy and has invested in clear opt-out documentation from early on. ClaudeBot honors Disallow paths reliably and shows the best noindex respect of any AI training crawler in the benchmark — an 8 out of 10 — suggesting Anthropic has built page-level exclusion logic more deliberately than many peers. Community crawl log reports consistently show ClaudeBot respecting blocked directories with a low rate of violations.

The main limitation is Crawl-delay compliance at 7: ClaudeBot does observe delays but can lag slightly behind declared values during high-activity periods. Documentation quality at 8 is strong but falls just short of Google's level of integration with broader developer tooling. Pros: best noindex compliance of any AI training crawler, reliable user-agent string, good opt-out docs. Cons: less deeply integrated into a publisher's broader SEO workflow than Google-Extended; delay compliance slightly inconsistent under load.

GPTBot (Score: 8.0/10)

OpenAI's GPTBot was among the first AI training crawlers to publish a dedicated user-agent string with clear documentation, and it benefits from that early-mover credibility. Disallow enforcement is strong at 9, and user-agent consistency is reliable — GPTBot has not been observed spoofing or cycling identities in the way some lesser-known crawlers have. The documentation explicitly covers how to block it for training data versus for the ChatGPT-Browse function separately, which adds meaningful granularity for publishers who want to remain visible in ChatGPT responses while preventing training use.

Noindex handling at 7 is the clearest area for improvement: GPTBot has limited documented behavior around individual page-level exclusion signals, meaning publishers need to rely primarily on path-level Disallow rules rather than trusting page-level meta directives alone. Pros: strong brand accountability, well-known user-agent string, granular opt-out options. Cons: noindex handling less refined than ClaudeBot; Crawl-delay compliance at 6 is the weakest of the top three.

Mid-Tier Crawlers: PerplexityBot, Amazonbot, and Meta-ExternalAgent

Applebot-Extended (Score: 7.2/10)

Apple's extension of its traditional Applebot for AI features earns a solid mid-tier position. Apple published documentation for the -Extended token that parallels its core Applebot guidance, and the user-agent string is stable. Disallow enforcement at 8 and noindex respect at 7 place it comfortably above the lower tier. The main gap is Crawl-delay compliance at 6, consistent with a crawler infrastructure not designed around aggressive frequency management. For publishers already familiar with blocking Applebot, extending that logic to Applebot-Extended is straightforward and well-supported.

Amazonbot (Score: 6.8/10)

Amazonbot is a more reliable performer than its mid-tier score suggests at first glance — its Disallow enforcement at 8 is genuinely strong, and Amazon's infrastructure means it identifies itself consistently. The score is held back by documentation quality at 7 (adequate but not comprehensive) and weak Crawl-delay compliance at 5. Publishers report that Amazonbot can be aggressive in crawl frequency when no delay is declared, so explicitly setting a Crawl-delay directive is recommended even though enforcement is inconsistent. Its role in powering Alexa AI and product discovery features makes it increasingly relevant to e-commerce publishers.

PerplexityBot and Meta-ExternalAgent (Scores: 6.0 and 5.8)

PerplexityBot generated significant controversy in 2024 and 2025 when multiple publishers reported it accessing blocked content — a history that weighs on its score despite more recent improvements. By mid-2026 its Disallow enforcement has improved to a 7, but noindex and Crawl-delay compliance remain weak. Meta-ExternalAgent, used for Meta AI features, scores fractionally lower primarily due to documentation quality at 6 and inconsistent Crawl-delay behavior at 4. Neither bot earns publisher trust by default, and both warrant active monitoring rather than passive reliance on declared directives. For a complete implementation walkthrough for these and other mid-tier crawlers, see our resource on AI robots.txt training controls for publishers.

"A 2026 audit of news publishers found that 56.4% blocked at least one AI crawler in robots.txt — a figure that reflects how quickly active restriction has become a baseline editorial and legal policy rather than a technical edge case."

Underperformers and Problem Crawlers

CCBot (Score: 4.8/10) and DiffBot (Score: 4.6/10)

Common Crawl's CCBot exists in a complicated space: it was not designed as an AI training crawler specifically, but its archive has become one of the most heavily used training data sources in the industry. CCBot's Disallow enforcement at 6 is inconsistent in practice — it honors some directives but has historically continued crawling pages that were blocked in robots.txt, particularly when prior crawl data existed. Noindex respect is very weak at 3, and documentation quality at 5 is adequate but not prescriptive enough to build a confident opt-out strategy around. Publishers who want to prevent Common Crawl data from feeding AI training pipelines need active monitoring, not just robots.txt declarations.

DiffBot is a commercial data extraction service rather than a first-party AI crawler, and its score of 4.6 reflects the fundamental ambiguity of its purpose. It offers an opt-out mechanism, but the enforcement reliability at 5 means that declarations do not consistently translate into excluded content. Its user-agent string has also been observed cycling in some contexts, earning it a consistency score of only 6. For high-value content publishers, DiffBot warrants explicit blocking and log-based verification rather than assumed compliance.

Bytespider (Score: 3.8/10)

Bytespider, associated with ByteDance, is the benchmark's lowest scorer and the crawler that attracts the most legitimate concern from SEO professionals. Its Disallow enforcement at 5 and noindex respect at 3 place it below every other named crawler in this study. Documentation quality is poor at 3 — ByteDance has not published a comprehensive, stable guidance page equivalent to what OpenAI, Anthropic, or Google maintain. Many publishers report Bytespider accessing blocked directories with no apparent awareness of Disallow rules. Industry practitioners widely recommend explicitly blocking Bytespider across all paths as a default stance, then monitoring logs to verify the block is having any effect at all.

Verdict by Publisher Profile

Best for Publishers Who Want Fine-Grained Control: Google-Extended + ClaudeBot

If granular, verifiable control over AI crawler access is your primary goal, the best outcome is to focus your trust on Google-Extended and ClaudeBot — the two crawlers with the most reliable documentation and enforcement track records. Use these bots' opt-out paths as your compliance anchor, knowing that your robots.txt declarations are most likely to be honored. For everything else, supplement with active log monitoring.

Best for Enterprise Publishers with Legal Exposure: Block All, Whitelist Selectively

Large publishers operating under copyright-sensitive frameworks — news organizations, academic publishers, premium content networks — should default to blocking all AI crawlers with User-agent: * and Disallow: / combined with specific named-bot blocks for the highest-risk crawlers like Bytespider and CCBot. The legal environment around AI training data remains unsettled in 2026, and a conservative posture combined with clear directives creates a better documented record of intent. This approach is well-supported by the finding that 60 of 70 major publishers already explicitly restrict at least one AI crawler.

Best for Content Creators Who Want AI Visibility: Selective Allowance

Creators who want their content cited in ChatGPT, Perplexity, or Gemini responses should allow GPTBot, ClaudeBot, Google-Extended, and PerplexityBot while blocking lower-scoring crawlers with less documented compliance. This selective allowance strategy lets you participate in AI-powered discovery while excluding crawlers whose compliance you cannot verify.

Best for Small Publishers with Limited Technical Resources: Focus on the Top Three

Time-constrained publishers get the most impact by writing specific blocks for Bytespider, CCBot, and DiffBot (the three lowest scorers) and trusting the top-tier crawlers to operate within their declared parameters. This minimal-effort approach handles the highest-risk bots first without requiring comprehensive log auditing.

How to Choose Your Blocking Strategy

Your AI crawler blocking strategy should be driven by three questions: What is the legal and commercial sensitivity of your content? What AI visibility outcomes do you want? And how much ongoing monitoring capacity do you have?

Step 1: Classify your content risk. Paywalled, copyright-registered, or commercially licensed content warrants explicit blocking of all AI training crawlers. Free editorial content where discovery is a goal warrants selective allowance. Mixed sites should handle blocking at the path level, not the domain level.

Step 2: Identify which crawlers you need to block by name. Never rely solely on User-agent: * to block AI crawlers — many AI-specific bots need explicit named entries because general wildcard rules are not always correctly parsed by their crawlers. Write individual stanzas for Bytespider, CCBot, DiffBot, and any other low-compliance bots you need to exclude.

Step 3: Set Crawl-delay values conservatively. Given that Crawl-delay compliance is the weakest dimension across almost all crawlers in this benchmark, set delay values as a signal of intent and monitor server logs independently. Do not rely on Crawl-delay as your primary server protection mechanism — rate limiting at the infrastructure level (CDN or WAF rules) is more reliable for the least compliant bots.

Step 4: Monitor and iterate quarterly. AI crawler behavior changes with product updates and legal developments. What a crawler honored in Q1 may shift by Q3. Build a quarterly robots.txt audit into your SEO calendar, checking access logs for the user-agent strings of your declared Disallow targets. Flag any that appear in blocked paths and document the incidents — that documentation may matter if legal questions arise about unauthorized scraping.

Step 5: Align with your licensing strategy. If you have or plan to pursue licensing agreements with AI companies, your robots.txt posture should be consistent with those agreements. Blocking a crawler in robots.txt while operating a licensing deal creates conflicting signals that complicate both technical enforcement and contractual interpretation.

Frequently Asked Questions

Do AI crawlers have to follow robots.txt rules legally?

In most jurisdictions as of 2026, there is no statute that explicitly mandates robots.txt compliance — the Robots Exclusion Protocol remains a convention rather than a law. However, courts in several jurisdictions have begun treating robots.txt declarations as evidence of intent to restrict access, which strengthens arguments under computer access laws and copyright frameworks. The practical implication is that well-documented robots.txt restrictions matter even if enforcement depends on the bot operator's voluntary compliance.

Can I block GPTBot without affecting my site's visibility in ChatGPT search results?

Yes, with an important distinction. GPTBot is OpenAI's training crawler, while OAI-SearchBot handles real-time retrieval for ChatGPT's browsing and search features. Blocking GPTBot in robots.txt prevents your content from being used in future model training but does not necessarily block retrieval-based citations. Publishers who want to stay out of training data while remaining citable in ChatGPT responses should block GPTBot specifically while leaving OAI-SearchBot unrestricted.

Why doesn't the noindex meta tag reliably stop AI crawlers from using my content?

The noindex directive was designed for traditional search engine indexing pipelines, and AI training crawlers are architecturally distinct from search indexers. Many AI crawlers fetch and store content without running the same indexing logic that processes noindex signals. Only a subset of crawlers — notably ClaudeBot and Google-Extended — have documented support for noindex as a training exclusion signal. For robust exclusion, path-level Disallow rules in robots.txt are more reliably enforced across the board than page-level meta tags.

How do I verify whether an AI crawler is actually respecting my robots.txt directives?

The most reliable method is server-side access log analysis: filter your logs by the user-agent strings of crawlers you have blocked and check whether they appear against URLs that fall within your Disallow paths. Tools like GoAccess, AWStats, or custom log parsers can automate this. If you're on a CDN like Cloudflare, bot analytics reports can surface unexpected crawler activity. For the lowest-compliance crawlers (Bytespider, CCBot, DiffBot), plan for a quarterly check as a baseline — compliance can change with crawler updates.