Is Your CDN Blocking AI Crawlers? How to Check

Short answer: fetch your live robots.txt from outside your own network and compare it with the file in your repository. CDNs and security products can prepend their own rules or block AI crawlers at the network edge, so the file you wrote may not be the file crawlers receive. An AI system cannot cite content it is not allowed to fetch.

Table of Contents

1. How We Found It
2. Two Ways an Edge Blocks Crawlers
3. How to Check Your Own Site
4. Reading the Result
5. Deciding What to Allow
6. After You Change Anything
7. Key Takeaways

How We Found It

We were investigating why our content did not seem to appear in AI assistants' answers. We had comparison pages, guides, and documentation, all written to answer specific questions.

The robots.txt in our repository was as permissive as it gets:

User-agent: *
Allow: /

That was not what the site was serving. Fetching the live file showed a block of rules above ours that we had never written, disallowing a list of AI crawlers by name, including GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, and the Google-Extended and Applebot-Extended tokens. It also carried a content signal line stating that search was permitted and AI training was not.

The rules were being added by our CDN's managed robots.txt feature, which was switched on in the dashboard. Nothing in our code or build mentioned it. Our own file was still there, underneath.

Two details were worth noting. Some AI agents were not in the blocked list, including PerplexityBot and OpenAI's search agent, so the effect was partial. And the setting was a legitimate choice, not a fault: it exists to give site owners control over AI training.

The lesson was about where to look. We had been reviewing content and markup, when the first question should have been whether the crawlers could read the pages at all.

Two Ways an Edge Blocks Crawlers

The second is harder to spot, because robots.txt may look fine while requests are rejected before reaching your site. Bot-fighting and "block AI scrapers" settings work this way.

How to Check Your Own Site

1. Fetch the live robots.txt

curl -s https://yourdomain.com/robots.txt

Run it from a machine outside your office network, or use an online fetch tool. Read the whole file from the top.

2. Compare it with your source

Open the robots.txt in your repository or CMS. Any line in the served version that is not in your source was added somewhere between your server and the internet.

3. Test a specific crawler's access

Request a page while identifying as an AI crawler and look at the status code:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yourdomain.com/

A 200 means the request was served. A 403 or a challenge means it was blocked at the edge. This is an indication, not proof: some services verify crawlers by IP address and treat an unverified user agent differently from the real bot.

4. Check your CDN and security settings

Look for anything named along the lines of AI crawl control, AI bots, scrape protection, bot management, or managed robots.txt. Note whether each is on.

5. Look at real traffic

Search your server or CDN logs for AI crawler user agents and see what responses they received. This is the definitive check. DeepSync's AI visibility feature, on Business and Enterprise, reports which AI crawlers read your site and how often.

Reading the Result

Remember that crawlers differ in purpose. Training crawlers, search crawlers, and user-triggered fetchers are separate agents (guide to AI crawlers). Blocking one kind does not block the others.

Deciding What to Allow

This is a business decision, not a technical one, and there is no universally right answer.

Reasons to allow AI crawlers

• You want to be cited and recommended in AI answers.
• Your content is marketing: its purpose is to be read as widely as possible.
• Being absent from future models' knowledge has a cost.

Reasons to block training crawlers

• Your content is the product, as with publishing or paid research.
• You do not want your work used for model training.
• Crawler load on your servers is a concern.

A common middle position

Allow search and user-triggered agents, so you can be cited and linked, and decide separately on training crawlers.

Whatever you choose, make it a decision someone actually took. Ours had been made by a default.

After You Change Anything

1. Fetch the live file again and confirm it reads as intended.
2. Repeat the user agent test.
3. Purge the CDN cache for robots.txt if it is cached.
4. Wait, then check logs for AI crawler visits over the following weeks.
5. Record the decision and who made it, so it is not reversed by accident.
6. Recheck after CDN plan or settings changes. Defaults can change.

Key Takeaways

• The robots.txt you wrote may not be the one being served.
• Edge products can add rules or block requests outright.
• Fetch the live file and test with a crawler's user agent.
• Decide deliberately which kinds of AI crawler to allow.

Conclusion

Before you optimise a single page for AI visibility, confirm the crawlers can get in. For us, the answer was hiding in a dashboard setting, not in our code.

More on AI crawlers.