83% of our AI crawler traffic was fake
We logged every request carrying an AI crawler user agent for eight days, then checked what each one actually asked for. Most of them wanted files that have never existed on our site.
If you run a website in 2026 you have probably looked at your logs and noticed a lot of AI crawlers. GPTBot, ClaudeBot, PerplexityBot, plus a growing list of others. The obvious conclusion is that AI systems are reading your site far more than search engines are.
We checked that assumption on our own site and it did not survive contact with the data.
What we measured
Between 19 and 26 July 2026 we pulled every HTTP request to vantaweb.io from the Cloudflare GraphQL Analytics API, using httpRequestsAdaptiveGroups with two dimensions: userAgent and clientRequestPath. The free tier caps each query to a single day and retains roughly eight days, so the study is a per-day loop over that window.
Then we did the part that changed the answer: we compared every requested path against the 105 URLs in our own published sitemap, plus the infrastructure files a legitimate crawler would reasonably ask for — robots.txt, sitemap.xml, llms.txt, llms-full.txt, rss.xml and our .well-known files.
The logic is simple. A crawler reading your site requests pages you publish. A request for a path you have never published is not a crawler reading your site, whatever its user agent claims.
What the fake traffic was asking for
The spoofed requests were not random. They were looking for credentials:
/id_ed25519and/.ssh/config— SSH private keys/service-account.json,/google-credentials.json,/firebase-adminsdk.json— cloud service account keys/.env.development,/config.env,/dev/.env— environment files/config/master.key,/key.pem,/privatekey.key,/.htpasswd/.s3cfg,/rclone.conf,/appsettings.json,/application.yml
ChatGPT does not need your SSH private key. This is ordinary credential scanning wearing an AI crawler's name, and the reason to wear that name is that plenty of sites now allow AI crawlers through filters that would otherwise block an unknown bot.
The numbers after cleaning
| Crawler | Operator | Legitimate fetches/day |
|---|---|---|
| ClaudeBot | Anthropic | 43 |
| OAI-SearchBot | OpenAI | 13 |
| ChatGPT-User | OpenAI | 8 |
| Googlebot | 13 |
Across the window, legitimate AI crawler requests ran at about 64 per day against about 13 per day for traditional search crawlers — a ratio of roughly 4.9 to 1.
That is still a striking number, and it is the honest version. The raw, unfiltered figure was 5.14 to 1 across a much larger volume, which overstated real AI crawler activity by roughly three times.
The list of "AI crawlers" gets much shorter
The bigger surprise was which crawlers survived filtering. In the raw logs we saw meaningful volume from Applebot, Amazonbot, PerplexityBot, Perplexity-User, MistralAI, meta-externalagent and Bytespider. After filtering to paths that actually exist, those had close to zero legitimate content fetches in this window. Their apparent presence was almost entirely spoofed traffic.
Only three AI crawlers were genuinely reading pages at volume, and one of them — ClaudeBot — accounted for more than half of all legitimate AI crawler activity.
Why a blacklist does not work
Our first attempt filtered with a regular expression for suspicious paths: anything containing .env, .git, credentials and so on. It was not enough. Requests for /api/v1/settings, /config/master.key, /application.yml and /firebase-admin.json went straight through, and every one of them is a probe.
You cannot enumerate the probe list because it keeps changing. You can enumerate your own sitemap, because you wrote it. Whitelisting is the only filter that stays correct.
What the real crawlers actually read
Once cleaned, the pages AI crawlers fetched most were our homepage, then our statistics page — a page whose entire purpose is to be quotable, with every figure carrying a named source and a date.
That is a small sample and we are not going to over-claim from it. But it is consistent with the advice that gets repeated in this space: if you want AI systems to cite you, publish things that are easy to cite.
How to check your own site
- Pull request logs with both the user agent and the requested path. User agent alone tells you nothing.
- Build a whitelist from your sitemap plus your known infrastructure files.
- Count requests to paths on that list separately from requests to everything else.
- Report only the first number as crawler traffic.
If you have published AI crawler statistics based on user agent counts alone, they are probably too high.
Limitations, stated plainly. This is one site over eight days. The window is eight days because that is roughly what Cloudflare's free analytics tier retains, not because we chose it. Our traffic is modest, so a single scanning campaign moves our percentages more than it would on a large site — the 83.4% figure is specific to us and we would expect it to vary a lot elsewhere.
We also make no claim about who is behind the spoofed requests. We logged a user agent string and a requested path. That is all we know, and inferring an actor from it would be guessing.
Everything above is reproducible with the Cloudflare Analytics API and your own sitemap. If you run the same check we would genuinely like to hear what you get.
Frequently asked questions
- Why is raw AI crawler traffic data unreliable?
- Because a user agent is a self-declared string that anyone can set. On our site, 83.4% of requests carrying an AI crawler user agent asked for paths that do not exist, such as SSH key files and cloud credential files. Counting user agents alone overstated real AI crawler volume by roughly three times.
- How can I tell real AI crawler traffic from spoofed traffic?
- Whitelist against your own sitemap. Compare each requested path to the URLs you actually publish, plus known infrastructure files like robots.txt and llms.txt. A request for a path you do not publish is not a crawler reading your site. A blacklist of suspicious patterns is not sufficient, because the probe list keeps changing.
- Which AI crawlers actually fetched content?
- In our 8-day window only three did so at meaningful volume: ClaudeBot at about 43 requests per day, OAI-SearchBot at about 13, and ChatGPT-User at about 8. Several other well-known AI user agents appeared in the raw logs but had almost no legitimate content fetches once spoofed requests were removed.