AI Crawlers Explained — GPTBot, ClaudeBot, PerplexityBot, and Which to Allow

Your server logs are full of bots with names like GPTBot, ClaudeBot, and PerplexityBot. Some collect training data. Some power AI search results that can send you visitors. Some only fetch a page when a user asks an assistant to read it. Blocking all of them—or allowing all of them—without understanding the difference can cost you visibility or control.

Table of Contents

1. Three Types of AI Crawlers
2. Common AI User Agents
3. How to See Which AI Crawlers Visit You
4. Allow or Block? A Decision Guide
5. robots.txt Examples
6. Watch Out for Your CDN
7. Key Takeaways

Three Types of AI Crawlers

Common AI User Agents

Vendors add and rename agents—check each operator's documentation for the current list.

How to See Which AI Crawlers Visit You

Crawlers don't run JavaScript the way browsers do, so most client-side analytics never see them. Options:

• Server or CDN logs filtered by user agent.
• A tracking pixel or server-side beacon that records bot visits.
• DeepSync [AI visibility](/docs/ai-visibility) (Business and Enterprise) shows which AI crawlers read your site and how often, next to the AI referral traffic you receive.

Allow or Block? A Decision Guide

• Marketing sites, SaaS, e-commerce: usually allow search and user-triggered agents—you want visibility. Many also allow training to be well represented in future models.
• Publishers monetizing content: often block training crawlers, allow search agents selectively.
• Private or paywalled content: block, and enforce at the server—robots.txt is a request, not access control.

robots.txt Examples

Allow AI search and user fetchers, block training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Watch Out for Your CDN

Some CDNs and security products block AI crawlers by default or inject a managed robots.txt. Your own file may say "allow" while the edge says "block." After changing settings, verify that crawler hits actually appear in your logs or analytics.

Key Takeaways

• Training, search, and user-triggered crawlers serve different purposes.
• Blocking search agents can remove you from AI citations.
• Client-side analytics miss crawlers—use logs or a dedicated tool.
• Check your CDN settings, not just robots.txt.

Conclusion

AI crawlers are the new search spiders. Decide deliberately which ones you let in—and verify that your infrastructure agrees.

See AI visibility · Is ChatGPT recommending your brand?