AI crawlers, search bots, and training agents
A compact reference of the crawler identities Reignat checks: what they are for, how they identify themselves, and where their verification sources live.
58 crawlers found
AI answers
Assistants that fetch a page live to answer someone's prompt.
ChatGPT-User
IP verifiedUser-triggered fetches when ChatGPT opens a page to answer a person.
Claude-User
IP verifiedUser-triggered fetches when Claude opens a page to answer a person.
Perplexity-User
IP verifiedUser-triggered fetches for Perplexity answers and source citations.
Google-Agent
IP verifiedGoogle user-triggered fetcher used by AI and product experiences.
Google-NotebookLM
IP verifiedGoogle fetcher associated with NotebookLM answer workflows.
Google-Read-Aloud
IP verifiedGoogle fetcher for read-aloud and assistant experiences.
GoogleAgent
IP verifiedGoogle user-triggered AI fetcher.
MistralAI-User
IP verifiedUser-triggered fetcher for Mistral assistant answers.
Copilot
UA matchUser-triggered Microsoft Copilot fetches for AI answers.
Amzn-User
UA matchUser-triggered Amazon fetches for fresh answers in products such as Alexa.
DuckAssistBot
IP verifiedDuckDuckGo real-time crawler for AI-assisted answers with citations.
xAI-SearchBot
UA matchxAI user-triggered fetcher for Grok answers.
Grok-DeepSearch
UA matchxAI deep-search fetcher used when Grok researches a prompt.
meta-externalfetcher
UA matchMeta fetcher used when a person requests or shares a specific URL.
Kimi-User
IP verifiedUser-triggered fetcher for Kimi assistant answers.
Qwen-User
UA matchUser-triggered fetcher for Qwen assistant answers.
Search indexes
Crawlers that discover and refresh content for search and answers.
OAI-SearchBot
IP verifiedOpenAI crawler used to surface sites in ChatGPT search results.
Claude-SearchBot
IP verifiedAnthropic crawler used to discover and refresh content for Claude search.
PerplexityBot
IP verifiedPerplexity crawler used to keep its answer and search index fresh.
Google-InspectionTool
IP verifiedGoogle crawler used by Search testing and inspection tools.
GoogleOther
IP verifiedGeneric Google crawler for research and product discovery.
Googlebot
IP verifiedClassic Google crawler used for Search indexing and discovery.
MistralAI-Index
IP verifiedMistral crawler that keeps its answer index fresh.
Bingbot
IP verifiedMicrosoft crawler used for Bing Search indexing.
msnbot
IP verifiedLegacy Microsoft crawler used for search indexing.
Applebot
IP verifiedApple crawler used for Apple search and assistant surfaces.
Amzn-SearchBot
UA matchAmazon crawler used to make content eligible for Amazon search experiences.
Kimi-SearchBot
IP verifiedMoonshot crawler that discovers content for Kimi search.
TikTokSpider
UA matchByteDance crawler used to surface content across TikTok experiences.
Baiduspider
UA matchBaidu crawler used for Baidu Search indexing.
YouBot
UA matchYou.com crawler that keeps its search and answer index fresh.
Training
Scrapers collecting public content that may train future models.
GPTBot
IP verifiedOpenAI crawler for collecting public content that may improve future models.
ClaudeBot
IP verifiedAnthropic crawler for public content that may be used to improve Claude.
Applebot-Extended
IP verifiedRobots token controlling whether Apple may use crawled content for AI training.
Amazonbot
UA matchAmazon crawler used to improve Amazon products and services.
meta-externalagent
UA matchMeta crawler for indexing or improving Meta products and AI systems.
KimiBot
IP verifiedMoonshot crawler collecting public content that may improve Kimi.
Bytespider
UA matchByteDance crawler collecting public content for model training.
ERNIEBot
UA matchBaidu crawler collecting public content that may improve ERNIE.
QwenBot
UA matchAlibaba crawler collecting public content that may improve Qwen.
ChatGLM-Spider
UA matchZhipu crawler collecting public content that may improve ChatGLM.
DeepSeekBot
UA matchDeepSeek crawler collecting public content for model training.
cohere-ai
UA matchCohere crawler used to fetch public content.
cohere-training-data-crawler
UA matchCohere crawler collecting public content for model training.
AI2Bot
UA matchAllen Institute crawler used to find documents for AI research systems.
CCBot
IP verifiedCommon Crawl crawler that builds open web crawl datasets.
Other AI bots
Recognised AI bots that don't fit the categories above.
OAI-AdsBot
UA matchOpenAI crawler associated with ad and landing-page fetch workflows.
Google-CloudVertexBot
IP verifiedGoogle Cloud Vertex AI fetcher triggered by customer agents.
GrokBot
UA matchxAI Grok bot recognised across xAI experiences.
xAI-Bot
UA matchGeneric xAI bot token.
xAI-Grok
UA matchxAI Grok bot token.
xAI-Web-Crawler
UA matchxAI web crawler token.
Grok
UA matchxAI Grok user-agent token.
FacebookBot
UA matchMeta crawler historically used to improve products and language models.
Doubaobot
UA matchByteDance bot associated with Doubao experiences.
YiyanBot
UA matchBaidu bot associated with Ernie Bot (Yiyan) experiences.
TongyiBot
UA matchAlibaba bot associated with Tongyi experiences.
AliyunBot
UA matchAlibaba Cloud bot token.
See which AI bots crawl your site
Reignat tracks every crawl server-side, verifies it against the provider's official IP ranges, and groups it by AI answers, indexing, and training.