Bots vs Humans — How Analytics Tells Them Apart

Short answer: no single test identifies a bot. Classification combines what the client says it is (the user agent), where the request comes from (the network), whether it behaves like a full browser, and whether its interaction looks human. Well-behaved crawlers announce themselves and are easy. Bots that imitate people are caught only by behavior, and never perfectly.

Table of Contents

1. Four Kinds of Evidence
2. Declared Identity
3. Network Origin
4. Browser Environment
5. Behavior
6. The Crawler Blind Spot
7. How We Classify AI Crawlers
8. What No Method Solves
9. Key Takeaways

Four Kinds of Evidence

Each catches a different class of bot. Good classification uses all four.

Declared Identity

Every request carries a user agent string. Reputable crawlers identify themselves in it: search engines, AI crawlers, monitoring services.

Matching is a lookup against a maintained list. Two practical details:

• Order matters. Some agents share a prefix with others. A more specific name must be checked before its shorter relative, or everything is filed under the general one.
• Lists go stale. Operators add and rename agents regularly.

The limit is obvious: anyone can send any user agent. A scraper claiming to be a search engine is common. Major operators publish ways to verify their crawlers, usually by checking the requesting IP address against published ranges or with a reverse DNS lookup. An identity that fails that check is an impostor.

Network Origin

• Data centre addresses. Ordinary visitors use home, office, and mobile networks. Traffic from cloud hosting ranges is usually automated.
• Published crawler ranges confirm real search and AI crawlers.
• Request rate. Hundreds of requests a minute from one address is not a person.
• Geography. A burst from a region you do not serve is a signal (spotting bot traffic).

The weaknesses: corporate VPNs and privacy relays route real people through data centres, and sophisticated bots rent residential addresses.

Browser Environment

Simple bots do not run a full browser. They fetch HTML and never execute scripts. That is why most crawlers never appear in script-based analytics at all.

Bots that do run a browser used to give themselves away through automation flags, missing features, or inconsistencies, for example a user agent claiming a phone alongside a desktop-sized screen. Current automation tools drive real browsers and hide most of these.

This layer still catches unsophisticated automation, and inconsistencies remain a useful hint.

Behavior

This is the layer that is hardest to imitate, and it is where session data helps.

In a session recording, the difference is plain within seconds. People wander. Automation either does nothing or does everything instantly.

Behavioral classification needs a few seconds of activity, so it cannot block a request before it is served. It is for cleaning data and flagging segments, not for defence at the edge.

The Crawler Blind Spot

Here is a problem particular to analytics. Suppose you want to know which AI crawlers read your site. Your analytics runs as JavaScript in the page. Most crawlers do not execute JavaScript. So your analytics cannot see the very thing you are trying to measure.

There are three ways around it.

1. Server and CDN logs. Every request is recorded there, whatever the client does. This is the complete source.
2. A beacon that needs no script. A tiny image requested from the page is fetched by some clients that never run JavaScript.
3. Classification at the first server contact. When any client starts a session, check its declared identity then.

Methods two and three undercount, because many crawlers fetch only the HTML and none of its sub-resources. Log-based measurement is the only complete answer. Anyone offering crawler analytics from a script alone should say so.

How We Classify AI Crawlers

DeepSync's AI visibility feature reports which AI crawlers read a site. The approach:

• A registry of AI agents, matched against the user agent, each labelled with its operator and its purpose: training, search indexing, or fetching on behalf of a user. The purpose matters, because blocking one kind does not block the others (crawler guide).
• Specific names are matched before general ones, for the prefix problem described above.
• Bot hits are logged separately from visitor sessions, so they never inflate human analytics.
• Two collection points: the start of a session, and a public image beacon for clients that do not run scripts.

We are open about the limit. Client-side collection undercounts crawlers that do not load sub-resources. Ingesting server or CDN logs is the fuller solution.

The same registry idea handles our own automation. Synthetic test traffic carries a signed marker in its user agent, verified by the server, so it is labelled as test traffic and not mistaken for either a visitor or an unknown bot. Any automation you run against your own site should be identifiable in the same way.

What No Method Solves

• Determined imitation. A real browser on a residential address moving the mouse along recorded human paths will pass.
• AI agents acting for people. An assistant browsing on a user's behalf is automated and also represents a real customer. Whether to count it as a visitor is a question without a settled answer.
• False positives. People using privacy tools, VPNs, and assistive technology can look unusual. Over-aggressive filtering removes real customers.
• The moving target. Every published detection method is something bot authors work around.

So the realistic aim is not a perfect split. It is keeping automated traffic from changing your decisions, and keeping a "likely bot" segment visible so you can check the filter is not catching people.

Key Takeaways

• Bot classification combines identity, network, environment, and behavior.
• Honest crawlers are easy; imitators are caught by behavior, imperfectly.
• Script-based analytics cannot see most crawlers; logs can.
• Keep automated traffic separate and keep a suspected-bot segment visible.

Conclusion

Telling bots from people is a matter of weighing evidence, not running one test. Be clear about what your tools can see, and treat every count as an estimate with a known blind spot.

Read about AI visibility.