AI Crawler User Agents: The Full List and What Each One Actually Does

AI crawlers do three different jobs — collecting training data, indexing pages so an assistant can cite them, and fetching one page because a user just asked about it — and each job usually has its own user-agent token with its own robots.txt rule. That is the part most "block AI bots" advice gets wrong: a single blanket rule can cut you out of the answers you want to appear in while leaving the training crawl untouched.

This is the reference table, built only from what each operator publishes on its own domain, with the tokens that are not crawlers at all flagged as such. Verified October 2026 — crawler docs change, so treat the vendor links as the source of truth.

Key numbers

• GPTBot was 28.1% of AI-crawler traffic on Cloudflare's network in July 2025, up from 11.9% a year earlier; measured against all crawling including search, its share went from 4.7% to 11.7% (Cloudflare, From Googlebot to GPTBot).
• Anthropic crawled roughly 38,000 pages per referred visitor in July 2025, down from about 286,000:1 in January; OpenAI's ratio was about 1,091:1, and Google crawled about 14 times per referral as of June 2025 (Cloudflare, the crawl-to-click gap).
• Only about 14% of top-10,000 domains that have a robots.txt file target AI bots in it — 546 of 3,816 in a June 2025 snapshot — and GPTBot was the most-blocked token, disallowed by 312 domains. Only around 37% of those top domains have a robots.txt at all (Cloudflare).
• Between July 2025 and January 2026, about seven times as many sites blocked AI crawlers like GPTBot and ClaudeBot as blocked Googlebot or Bingbot (Cloudflare Radar 2025 year in review).
• Two of the most-copied tokens are not crawlers. Google-Extended and Applebot-Extended issue no requests of their own; they are robots.txt switches governing how already-crawled content may be used (Google; Apple).

Cloudflare's figures are one network's view and are dated where stated, not current-month numbers.

The three jobs, and why they need separate rules

Before the table, the mental model. Almost every token below does exactly one of these:

1. Training — collects public pages that may be used to train a model. Blocking it is a copyright/consent decision. It has no effect on whether an assistant can cite you today.
2. Search indexing — builds the index an assistant retrieves from when it answers with citations. Blocking it removes you from those answers. This is the one to leave open if you want AI visibility.
3. User-triggered fetch — grabs a single page because someone asked the assistant about it. Operators generally state that robots.txt may not apply to these, because the request came from a person, not a crawl.

Conflating jobs 1 and 2 is the expensive mistake. Disallowing GPTBot opts you out of training; it does not hide you from ChatGPT search, which uses OAI-SearchBot. Disallow both and you lose the citation, not just the training.

The full table

Sources, per operator: OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Common Crawl, Amazon.

Three notes on precision. Amazon publishes the Amzn-SearchBot token and its IP list but does not describe its job beyond the name, and Google documents who initiates a Google-CloudVertexBot crawl rather than how it treats robots.txt — both cells say so rather than filling the gap with the obvious guess. Anthropic documents all three agents and tells site owners to opt out per agent in robots.txt, but the published example covers ClaudeBot, so the table says what the docs say rather than inferring the other two. Meta's user agents appear in logs with a version suffix — meta-externalagent/1.1 and meta-externalfetcher/1.1 — while the robots.txt token is the bare name.

The full user-agent strings

For WAF rules and log filters, two operators publish complete strings rather than bare tokens:

• Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
• Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Match on the token substring rather than the whole string; operators change the surrounding boilerplate without notice. Note also that robots.txt user-agent matching is case-insensitive, so meta-externalagent and Meta-ExternalAgent behave identically.

The tokens that are not crawlers

This is the most common error in AI-crawler advice, and it runs in both directions.

Google-Extended has no HTTP user agent of its own. Google crawls with its existing agents; the token exists purely as a robots.txt control over whether crawled content can be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google states it has no effect on inclusion in Google Search and is not a Search ranking signal. For Search itself — and therefore for AI Overviews and AI Mode, which draw on the Search index — the documented control is Googlebot directives, not Google-Extended.

Applebot-Extended works the same way: Apple's support page says explicitly that it does not crawl webpages. It governs whether content Applebot already collected may train Apple's foundation models. Disallow it and your pages still appear in search, and remain discoverable through Spotlight, Siri and Safari, because that is Applebot's job, not Applebot-Extended's.

The practical consequence: adding Disallow: / for either token costs you nothing in search visibility, which is why they are the two safest entries on any opt-out list. It also means neither one does anything to reduce crawl load.

Microsoft is the exception: no token, use meta tags

Microsoft publishes no separate AI-crawler token. Bingbot does the crawling, and the controls for Copilot are page-level robots meta tags rather than robots.txt lines:

• NOARCHIVE — the content is not used in Copilot responses or grounding results, is not linked to in answers, and is not used to train Microsoft's generative AI foundation models.
• NOCACHE — no cached copy is stored, and use is limited to the URL, title and snippet, which reduces citation depth.
• Set both and Bing treats the page as NOCACHE.

Source: Bing Webmaster Tools on supported robots meta tags and the 2023 announcement of the Bing Chat controls. Because these are meta tags, they are per-page decisions — which is more precise than robots.txt, but means a CMS that cannot set them has no control at all.

What the vendors do not publish

Being honest about the gaps matters more here than a longer table, because a row you cannot source is a row that gets someone blocking the wrong thing.

• Cohere says it operates no crawler. Its web-crawlers page states that it does not use Cohere bots or user agents to crawl or scrape web content for training generative AI foundation models at this time, and its bot table is empty. If that changes it will name the bots there. Its example block uses the token Coherebot — so a cohere-ai line, which circulates widely in robots.txt templates, matches nothing Cohere documents (Cohere).
• ByteDance publishes nothing about Bytespider on its own domains. We searched ByteDance's domains and found no crawler documentation and no Bytespider rule in its own robots.txt. The token is widely reported in server logs, but there is no operator statement about its purpose or its robots.txt behaviour to cite, so it is not in the table above.
• xAI, Mistral and DeepSeek have no crawler documentation we can source on their own domains. Absence of a published crawler is not evidence of no crawling — it means you cannot write a rule with any confidence about what honours it.

How to verify a crawler is genuine

User-agent strings are trivially spoofed, and both OpenAI's and Common Crawl's own channels carry reports of floods from agents impersonating their bots. Every major operator therefore publishes IP data:

Two cautions from the operators themselves. Anthropic and Cohere both warn that blocking by IP is not a reliable way to opt out — partly because an IP block stops the crawler reading your robots.txt in the first place, which is the signal you actually wanted it to see. And published lists lag: OpenAI's own developer forum carries reports of GPTBot requests from IPs missing from the list, and OpenAI's guidance is to validate on several signals rather than IP logs alone.

Use IP lists to decide whether to trust a request. Use robots.txt to express what you want. They are different tools.

A starting robots.txt

For most B2B sites that want to be cited by assistants, the shape below is the sensible default: let the search-indexing and user-fetch agents through, and make the training decision separately and deliberately.

# Search indexing — leave open if you want to be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

# Training — your call; these do not affect citations
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: CCBot
Allow: /

# Control tokens, not crawlers — blocking these costs no search visibility
User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

Two things this file cannot do. Repeat the rules on every subdomain, because robots.txt is per host — Anthropic and Amazon both say so explicitly. And remember robots.txt is a stated preference, not enforcement: Cloudflare's own framing is that it allows the expression of crawling preferences and depends on good bots complying. If you need enforcement, that is a WAF rule against the verified IP lists above.

Changes are not instant either. OpenAI says robots.txt edits take about 24 hours to register for search, and Meta says its crawlers may cache the file for up to 24 hours.

Where Sightivo fits

Allowing the right crawlers is the floor, not the win — it makes you eligible to be cited, and nothing more. What decides whether an assistant actually names you is which pages and brands it already reaches for in your category.

Sightivo runs your buyer prompts, records which brands and which sources each answer used, and turns the gap into a ranked list of pages to get mentioned on. The free AI visibility checker shows in a minute whether ChatGPT names you for five prompts in your category.

Common Questions

What are the AI crawler user agents I should know about?

The documented ones are OpenAI's GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot; Anthropic's ClaudeBot, Claude-SearchBot and Claude-User; Perplexity's PerplexityBot and Perplexity-User; Meta's meta-externalagent and meta-externalfetcher; Apple's Applebot; Amazon's Amazonbot, Amzn-SearchBot and Amzn-User; and CCBot for Common Crawl. Google-Extended and Applebot-Extended are robots.txt control tokens rather than crawlers.

Does blocking GPTBot stop ChatGPT citing my site?

No. GPTBot collects training data. Live citation in ChatGPT search runs through OAI-SearchBot, a separate token with its own rule, and ChatGPT-User handles pages fetched because a user asked. Blocking GPTBot alone opts you out of training while leaving citation intact — which is what most publishers actually want.

Do AI crawlers obey robots.txt?

The training and search-indexing agents are documented as respecting it. The user-triggered fetchers are the exception by design: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, Meta says meta-externalfetcher may bypass it, and Amazon says Amzn-User may not follow all directives. In every case the stated reason is that a person requested the page. And robots.txt is a preference, not enforcement — only cooperating bots comply.

Is Google-Extended a crawler?

No. It has no HTTP user agent of its own. It is a robots.txt token that controls whether content Google has already crawled can be used to train Gemini models and ground their answers. Google states it does not affect inclusion in Google Search and is not a ranking signal. For Search — and so for AI Overviews and AI Mode — the control is Googlebot directives.

How do I block AI crawlers from training on my content without losing AI search visibility?

Disallow the training agents and the two control tokens — GPTBot, ClaudeBot, meta-externalagent, CCBot, Google-Extended, Applebot-Extended — and leave the search-indexing agents open: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot. Add Bing's NOARCHIVE meta tag if you also want to exclude Microsoft training use, since Microsoft has no robots.txt token for it. Repeat the rules on every subdomain.

How do I verify a crawler is really GPTBot and not a spoof?

Check the request IP against the operator's published list — openai.com/gptbot.json for GPTBot, openai.com/searchbot.json for OAI-SearchBot, and the equivalents Perplexity, Amazon and Common Crawl publish. Google and Apple use reverse-DNS verification instead. Treat the lists as advisory: they lag, so OpenAI recommends validating on several signals rather than IP alone. Do not use IP blocks as your opt-out — Anthropic and Cohere both warn it is unreliable, partly because a blocked crawler never reads your robots.txt.

Does Cohere crawl my site?

By its own documentation, no. Cohere's web-crawlers page states it does not use Cohere bots or user agents to crawl or scrape content for training generative AI foundation models at this time, and its bot table is empty. If it starts, it will name the agent there, and its example token is Coherebot — not the cohere-ai line that circulates in robots.txt templates.

How often should I review this file?

Twice a year is enough for the tokens, which change rarely, but check before you act on any rule you copied from a template — several widely-shared robots.txt blocks still list agents no operator documents. The IP lists change far more often, so pull them from the JSON endpoints rather than hard-coding ranges.

Key Takeaways

• AI crawlers do three jobs — training, search indexing, user-triggered fetch — and each has its own token. Blocking by job, not by vendor, is what keeps you citable.
• Blocking GPTBot opts you out of training, not out of ChatGPT search; that runs on OAI-SearchBot.
• Google-Extended and Applebot-Extended are not crawlers. They issue no requests, cost nothing in search visibility to disallow, and do nothing for crawl load.
• User-triggered fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User) are documented as possibly ignoring robots.txt, because a person asked for the page.
• Microsoft has no AI-crawler token: use the NOARCHIVE and NOCACHE meta tags.
• Verify with the operators' published IP lists, not the user-agent string — and never rely on IP blocks as your opt-out.
• Cohere documents no crawler at all, and ByteDance documents nothing about Bytespider, so any rule you have written for either is a guess.

Start with the free AI visibility checker to see whether the assistants name you today, then read where ChatGPT gets its data for how the retrieval pipeline behind these crawlers actually works. If you are picking a tool to monitor it, compare the options in the best AI visibility tools, and work through the fixes with the generative engine optimization guide.