GUIDE

AI crawler list and user agents by company

AI bots index for search, collect training data or fetch pages for users. Rules take ~24 hours for ChatGPT search and up to 30 days for Amazonbot.

Updated October 30, 20265 min read

24 hrsTime for ChatGPT and Perplexity search to adjust to robots.txt changes
30 daysMaximum time Amazonbot caches your robots.txt rules
11.50%Share of ChatGPT crawler requests for JS files (Vercel and MERJ, 2024)

In short

AI bots can index for search, collect data that may be used for training, fetch pages for users, or check ad safety.
OpenAI says GPTBot and OAI-SearchBot settings are independent. Google says Google-Extended is a robots.txt token, not a crawler, and does not affect Google Search inclusion or ranking. Anthropic says ClaudeBot collects content that may be used for training.
User fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) act for human users and may ignore robots.txt.
A December 2024 Vercel and MERJ study found dedicated AI crawlers from OpenAI, Anthropic, Meta, ByteDance, and Perplexity do not render JavaScript, though 11.50% of ChatGPT's and 23.84% of Claude's requests were for JS files.

Review robots.txt rules for the relevant search and training tokens separately.

How AI crawlers split into search, training, and user bots

The table below lists documented AI crawlers and tokens as of October 2026.

Company	Token or Bot	Job	Follows robots.txt	Notes and User Agent Details
OpenAI	`OAI-SearchBot`	Search	Yes (about 24h delay)	UA: `OAI-SearchBot/1.4; +https://openai.com/searchbot`
OpenAI	`GPTBot`	Training	Yes	UA: `GPTBot/1.4; +https://openai.com/gptbot`
OpenAI	`ChatGPT-User`	User fetch	May not apply	UA: `ChatGPT-User/1.0; +https://openai.com/bot`
OpenAI	`OAI-AdsBot`	Ads safety	Required for ads	UA: `OAI-AdsBot/1.0`
Anthropic	`Claude-SearchBot`	Search	Yes (honors Crawl-delay)	Full UA string not published
Anthropic	`ClaudeBot`	Training	Yes (honors Crawl-delay)	Full UA string not published
Anthropic	`Claude-User`	User fetch	Yes (honors Crawl-delay)	Blocking may reduce user-directed search visibility
Perplexity	`PerplexityBot`	Search	Yes (up to 24h delay)	Not used for foundation model training
Perplexity	`Perplexity-User`	User fetch	Generally ignores	Fetches pages a user asked about
Google	`Googlebot`	Search	Yes	Controls Search, AI Overviews, and AI Mode
Google	`Google-Extended`	Training and grounding control	Robots.txt token only	Controls Gemini training and Gemini Apps grounding
Apple	`Applebot`	Search	Yes	Falls back to Googlebot rules if Applebot is omitted
Apple	`Applebot-Extended`	Training control	Robots.txt token only	Opts out of Apple model training; keeps pages eligible for search
Microsoft	`Bingbot`	Search	Not specified in the fact sheet	Powers Bing and Copilot; control Copilot via NOARCHIVE
Meta	`Meta-ExternalAgent`	Training	Not specified	Crawls for uses such as training AI models
Meta	`Meta-ExternalFetcher`	User fetch	May bypass	Fetches links at a user's request
DuckDuckGo	`DuckAssistBot`	Search	Not specified in the fact sheet	Real-time crawl for AI answers; no model training; opt-out takes effect after 72 hours
Mistral	`MistralAI-Index`	Not specified in the fact sheet	Not specified in the fact sheet	Documented at docs.mistral.ai/robots
Mistral	`MistralAI-Training`	Not specified in the fact sheet	Not specified in the fact sheet	Documented at docs.mistral.ai/robots
Mistral	`MistralAI-User`	User fetch	Not specified in the fact sheet	Handles user actions; no automatic crawl or training
Amazon	`Amazonbot`	Training	Yes (cached 30 days)	May be used to train Amazon AI models
Common Crawl	`CCBot`	Training	Yes	Builds open corpus widely used for LLM training
ByteDance	`Bytespider`	Unspecified	Not published	No official docs; listed as AI crawler by Cloudflare

Step 1: Allow the bots that control AI search visibility

Review the robots.txt rules for the search crawlers relevant to your site. See our guide on What is GEO.

OpenAI's October 2026 documentation states that OAI-SearchBot surfaces websites in ChatGPT search features; sites that opt out will not appear in ChatGPT search answers, though they can still appear as navigational links. See our OpenAI crawlers guide. Perplexity (October 2026) documents that PerplexityBot indexes pages for search without training foundation models. ChatGPT search can take about 24 hours to adjust after a robots.txt change; PerplexityBot robots.txt changes can take up to 24 hours. Anthropic (April 2026) uses Claude-SearchBot for search quality, and DuckDuckGo (October 2026) uses DuckAssistBot for real-time AI answers (72-hour opt-out delay).

Google, Microsoft, and Apple tie AI search to classic crawlers. Google (October 2026) uses Googlebot for Search, AI Overviews, and AI Mode. Microsoft (October 2026) shares Bingbot between Bing and Copilot; use NOARCHIVE to keep content out of Copilot answers, or NOCACHE to limit Copilot to URL, title, and snippet. Apple (September 2026) uses Applebot for Spotlight, Siri, and Safari search, falling back to Googlebot rules if Applebot is omitted.

Step 2: Block model training crawlers separately

Review training-related tokens separately from search crawler settings:

OpenAI (GPTBot): Disallowing GPTBot tells OpenAI the content should not be used for training. OAI-SearchBot and GPTBot settings are independent, though if you allow both, OpenAI may use one crawl for both purposes.
Google (Google-Extended) and Apple (Applebot-Extended): Both are robots.txt tokens, not separate crawlers. Google-Extended opts out of Gemini training and Gemini Apps grounding without affecting Google Search inclusion or ranking. Applebot-Extended opts out of Apple model training while keeping pages eligible for search.
Other training bots: ClaudeBot collects content that may be used for training; Meta-ExternalAgent crawls for uses including training; CCBot builds a corpus widely used for LLM training; and Amazonbot may be used for training and caches robots.txt for up to 30 days. Mistral also documents MistralAI-Training.

Step 3: Understand user-triggered fetchers

OpenAI states (October 2026; covered by Search Engine Journal in August 2026) that ChatGPT-User handles user actions in ChatGPT and Custom GPTs, does not crawl automatically or decide Search inclusion, and robots.txt rules may not apply. Similarly, Perplexity-User generally ignores robots.txt, and Meta-ExternalFetcher may bypass it. By contrast, Anthropic states (April 2026) that Claude-User honors robots.txt, supports Crawl-delay, and does not bypass CAPTCHAs, noting blocking it may reduce user-directed search visibility.

Common mistakes when managing AI crawlers

Relying on client-side JavaScript: A December 17, 2024 Vercel and MERJ study found dedicated AI crawlers from OpenAI, Anthropic, Meta, ByteDance, and Perplexity do not render JavaScript; Googlebot's infrastructure renders JavaScript, and Applebot renders through a browser. If text requires JavaScript, OAI-SearchBot and PerplexityBot will miss it. See our JavaScript SEO and llms.txt guides.
Expecting robots.txt to hide bare links in ChatGPT: OpenAI's publisher FAQ notes a URL blocked in robots.txt but learned from a third-party search provider can still show as a bare link and page title. A noindex meta tag readable by the crawler prevents that.
Blocking OAI-AdsBot while running ChatGPT ads: OpenAI's October 19, 2026 advertiser guidance requires allowing OAI-AdsBot (which checks ad safety without training models) and recommends OAI-SearchBot.

How to check if OpenAI's search crawler can read your site

Test your setup with the free GEO audit (no account required, 5 audits a day). It fetches your page with the OAI-SearchBot user agent, checks robots.txt rules for OAI-SearchBot, GPTBot, and ChatGPT-User, verifies HTML text without JavaScript, checks bot protection and noindex, rebuilds the ~200-character snippet ChatGPT keeps, and scores the page out of 100.

To track pages over time, FoundByChat is free to start and checks ChatGPT search every 2 hours for your customers' questions, recording whether pages are Found or Cited and emailing you when pages enter the index or snippets change.

FOUNDBYCHAT

Know the moment ChatGPT picks up your pages

We ask ChatGPT your questions every 2 hours and mail you what its index picked up or rewrote. Free to start: sign in with Google or an email link.

Start free Or run the free GEO audit

Questions

What is the difference between GPTBot and OAI-SearchBot?

Per OpenAI's October 2026 documentation, GPTBot crawls content that may be used to train OpenAI's generative AI foundation models, while OAI-SearchBot surfaces websites in ChatGPT search features. You can disallow GPTBot in robots.txt and still allow OAI-SearchBot without blocking OAI-SearchBot.

Why do ChatGPT-User and Perplexity-User ignore robots.txt?

OpenAI says ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs; Perplexity says Perplexity-User fetches pages a user asked about. Because a person initiates the request rather than an automated crawl, OpenAI says robots.txt rules may not apply, and Perplexity says its user bot generally ignores robots.txt.

Does Google-Extended affect Google AI Overviews?

No. Google's October 2026 crawler documentation states Google-Extended only controls whether content is used to train Gemini models and ground answers in Gemini Apps. Inclusion in Google Search, AI Overviews, and AI Mode is controlled by Googlebot.

How do I block Microsoft Copilot from using my pages?

Microsoft's October 2026 webmaster guidelines state Bing and Copilot share Bingbot. Microsoft says a NOARCHIVE meta tag keeps content out of Copilot answers, while NOCACHE limits Copilot to URL, title, and snippet.

Do AI crawlers execute JavaScript?

According to a December 2024 analysis by Vercel and MERJ, dedicated AI crawlers from OpenAI, Anthropic, Perplexity, Meta, and ByteDance download JavaScript files without executing them. Googlebot's infrastructure renders JavaScript, and Applebot renders through a browser.