BotBase — Bots Directory (vNext)

Bot Activity

Bot Policy

Show all types Show all operators Show all countries Show all content use Show all Last 24 hours

Anomaly Insights i What is this?Bots with anomalous activity in the selected time period — significant traffic spikes or newly-seen bots you aren't yet mitigating. Select a card to investigate.

High Trust (Verified) 6 i High Trust — Verified BotsCloudflare confirms each bot's identity through cryptographic Web Bot Auth signatures, published IP lists, or reverse DNS. They operate within a narrow scope, are publicly documented, and obey robots.txt and Content-Signals directives.Cloudflare default: Allow

All allowed

Name	Operator	Country	Content Use	Volume	Policy
Googlebot Google search crawler	Google	United States	Training	{{spark}}987	{{switch:on}}
Bingbot Microsoft Bing web crawler	Microsoft	United States	User-Agent	{{spark}}748	{{switch:on}}
DuckDuckBot DuckDuckGo crawler	DuckDuckGo	United States	Indexing	{{spark}}439	{{switch:on}}
Applebot Customized Apple Spotlight crawler	Apple	United States	Reproduction	{{spark}}678	{{switch:on}}
SlackBot Customized Slack link previews	Slack Technologies	United States	Training	{{spark}}439	{{switch:on}}
Discordbot Customized Discord link previews	Discord	United States	User-Agent	{{spark}}304	{{switch:on}}

Moderate Trust (Signed Agents & Corporate Proxies) 6 i Moderate Trust — Signed Agents & Corporate ProxiesIdentity is confirmed, but access is broad (many users can present as the agent) or it's an enterprise tool. Cloudflare tracks them and lets you allow or block them as a group, but does not allow by default.Cloudflare default: Block (easy to allow)

Partial

Name	Operator	Country	Content Use	Volume	Policy
GPTBot OpenAI's crawler for foundation model training	OpenAI	United States	Indexing	{{spark}}665	{{switch:on}}
ClaudeBot Anthropic's crawler for Claude training	Anthropic	United States	Reproduction	{{spark}}530	{{switch:off}}
PerplexityBot Perplexity AI search and answer engine	Perplexity AI	United States	User-Agent	{{spark}}498	{{switch:off}}
Cohere-ai Cohere's crawler for model training and RAG	Cohere	Canada	User-Agent	{{spark}}410	{{switch:on}}
Palo Alto Networks RBI Customized Remote browser isolation for enterprise users	Palo Alto Networks	United States	Reproduction	{{spark}}389	{{switch:off}}
Netskope Customized Netskope SASE/SWG proxy traffic	Netskope Inc.	United States	Training	{{spark}}639	{{switch:off}}

Low Trust (Benign Bots) 12 i Low Trust — Benign BotsWell-behaved bots that self-identify, but for which Cloudflare has incomplete verification metadata. They are tracked and continuously monitored. Cloudflare manages these on your behalf using its defaults.Cloudflare default: Allow & monitor

Name	Operator	Country	Content Use	Volume
CCBot Common Crawl's open web archive crawler	Common Crawl	United States	Indexing	{{spark}}865
YandexBot Yandex search engine crawler	Yandex	Russia	User-Agent	{{spark}}530
Baidu Spider Baidu search engine crawler	Baidu	China	Training	{{spark}}995
Internet Archive Customized Wayback Machine crawler	Internet Archive	United States	User-Agent	{{spark}}756
AhrefsBot Ahrefs SEO & backlink analysis crawler	Ahrefs	Singapore	User-Agent	{{spark}}748
UptimeRobot Website uptime monitoring service	UptimeRobot	United States	Reproduction	{{spark}}878

Showing 1–6 of 12

Page of 2

Untrusted 7 i Untrusted BotsBots that actively evade detection — ignoring robots.txt, rotating user-agents, or switching to residential proxies when blocked. Cloudflare blocks these by default as part of managed protection.Cloudflare default: Block

Name	Operator	Country	Content Use	Volume
MJ12bot Unknown operator	Unknown	—	User-Agent	{{spark}}756
DotBot Unknown operator	Unknown	—	Indexing	{{spark}}621
PetalBot Unknown operator	Unknown	—	Reproduction	{{spark}}410
Bytespider Unknown operator	Unknown	—	Reproduction	{{spark}}878
DataForSeoBot Unknown operator	Unknown	—	Training	{{spark}}439
SemBot Unknown operator	Unknown	—	User-Agent	{{spark}}304
Scrapy Customized Unknown operator	Unknown	—	Indexing	{{spark}}865