Last updated
83% of our AI crawler traffic was fake
We logged every request carrying an AI crawler user agent for eight days, then checked what each one actually asked for. Most of them wanted files that have never existed on our site.
If you run a website in 2026 you have probably looked at your logs and noticed a lot of AI crawlers. GPTBot, ClaudeBot, PerplexityBot, plus a growing list of others. The obvious conclusion is that AI systems are reading your site far more than search engines are.
We checked that assumption on our own site and it did not survive contact with the data.
What did we measure?
We pulled every request to vantaweb.io between 19 and 26 July 2026 from Cloudflare's GraphQL Analytics API, grouped by user agent and requested path, and compared each path with the 105 URLs in our sitemap plus standard infrastructure files. A request for a path we have never published was counted as spoofed, whatever its user agent claimed.
Between 19 and 26 July 2026 we pulled every HTTP request to vantaweb.io from the Cloudflare GraphQL Analytics API, using httpRequestsAdaptiveGroups with two dimensions: userAgent and clientRequestPath. The free tier caps each query to a single day and retains roughly eight days, so the study is a per-day loop over that window.
Then we did the part that changed the answer: we compared every requested path against the 105 URLs in our own published sitemap, plus the infrastructure files a legitimate crawler would reasonably ask for — robots.txt, sitemap.xml, llms.txt, llms-full.txt, rss.xml and our .well-known files.
The logic is simple. A crawler reading your site requests pages you publish. A request for a path you have never published is not a crawler reading your site, whatever its user agent claims.
What was the fake traffic asking for?
Mostly credentials. The spoofed requests asked for SSH private keys, cloud service-account files, environment files and other configuration secrets. None of those files exist on our site, and no search engine or AI crawler needs them. It is ordinary credential scanning that borrows an AI crawler's name to slip past bot filters.
The spoofed requests were not random. They were looking for credentials:
/id_ed25519and/.ssh/config— SSH private keys/service-account.json,/google-credentials.json,/firebase-adminsdk.json— cloud service account keys/.env.development,/config.env,/dev/.env— environment files/config/master.key,/key.pem,/privatekey.key,/.htpasswd/.s3cfg,/rclone.conf,/appsettings.json,/application.yml
ChatGPT does not need your SSH private key. This is ordinary credential scanning wearing an AI crawler's name, and the reason to wear that name is that plenty of sites now allow AI crawlers through filters that would otherwise block an unknown bot.
How much real AI crawler traffic was left after cleaning?
About 64 legitimate AI crawler fetches a day, against about 13 from traditional search crawlers: a ratio of roughly 4.9 to 1. ClaudeBot accounted for more than half of the legitimate AI activity, followed by OAI-SearchBot and ChatGPT-User. Several well-known AI user agents had almost no legitimate fetches at all once spoofed requests were removed.
| Crawler | Operator | Legitimate fetches/day |
|---|---|---|
| ClaudeBot | Anthropic | 43 |
| OAI-SearchBot | OpenAI | 13 |
| ChatGPT-User | OpenAI | 8 |
| Googlebot | 13 |
Across the window, legitimate AI crawler requests ran at about 64 per day against about 13 per day for traditional search crawlers — a ratio of roughly 4.9 to 1.
That is still a striking number, and it is the honest version. The raw, unfiltered ratio was 5.14 to 1 across a much larger volume. Of the 3,727 requests that carried a crawler user agent, 3,109 asked for paths we have never published, which leaves 618 real ones. Counting user agents alone therefore overstated real crawler activity (AI and search combined) by about six times, since 3,727 divided by 618 is 6.03. Both counts come from the same Cloudflare GraphQL Analytics API pull, 19 to 26 July 2026.
The list of "AI crawlers" gets much shorter
The bigger surprise was which crawlers survived filtering. In the raw logs we saw meaningful volume from Applebot, Amazonbot, PerplexityBot, Perplexity-User, MistralAI, meta-externalagent and Bytespider. After filtering to paths that actually exist, those had close to zero legitimate content fetches in this window. Their apparent presence was almost entirely spoofed traffic.
Only three AI crawlers were genuinely reading pages at volume, and one of them — ClaudeBot — accounted for more than half of all legitimate AI crawler activity.
Why doesn't a blacklist of suspicious paths work?
Because the list of probe paths keeps changing, so any pattern list misses new ones. Our first filter, a regular expression for paths such as .env and .git, let requests for /config/master.key and /application.yml straight through. Your sitemap is a list you control, which is why whitelisting against it is the only filter that stays correct.
Our first attempt filtered with a regular expression for suspicious paths: anything containing .env, .git, credentials and so on. It was not enough. Requests for /api/v1/settings, /config/master.key, /application.yml and /firebase-admin.json went straight through, and every one of them is a probe.
You cannot enumerate the probe list because it keeps changing. You can enumerate your own sitemap, because you wrote it. Whitelisting is the only filter that stays correct.
What did the real crawlers actually read?
Mostly our homepage and our statistics page. That is a small sample from one site over eight days, so we draw no strong conclusion from it. It does fit common advice for appearing in AI answers: publish pages that are easy to quote, with figures that carry a named source and a date.
Once cleaned, the pages AI crawlers fetched most were our homepage, then our statistics page — a page whose entire purpose is to be quotable, with every figure carrying a named source and a date.
That is a small sample and we are not going to over-claim from it. But it is consistent with the advice that gets repeated in this space: if you want AI systems to cite you, publish things that are easy to cite.
How can you check your own site?
Pull logs that include both the user agent and the requested path, build a whitelist from your sitemap and infrastructure files, and count requests to whitelisted paths separately from everything else. For a stronger check, compare the source IP addresses with the lists that OpenAI, Anthropic and Google publish for their crawlers.
- Pull request logs with both the user agent and the requested path. User agent alone tells you nothing.
- Build a whitelist from your sitemap plus your known infrastructure files.
- Count requests to paths on that list separately from requests to everything else.
- Report only the first number as crawler traffic.
To confirm that a request really came from the crawler it names, check its IP address against the operator's published list: OpenAI links IP files for OAI-SearchBot, GPTBot and ChatGPT-User on its bots page, Anthropic links a list for its crawlers in its crawler help article, and Google explains how to verify Google crawlers by reverse DNS and published IP ranges.
If you have published AI crawler statistics based on user agent counts alone, they are probably too high.
Limitations, stated plainly. This is one site over eight days. The window is eight days because that is roughly what Cloudflare's free analytics tier retains, not because we chose it. Our traffic is modest, so a single scanning campaign moves our percentages more than it would on a large site — the 83.4% figure is specific to us and we would expect it to vary a lot elsewhere.
We also make no claim about who is behind the spoofed requests. We logged a user agent string and a requested path. That is all we know, and inferring an actor from it would be guessing.
Correction, 15 October 2026. An earlier version said user-agent counts overstated real AI crawler activity by roughly three times. That was wrong. The same data gives about six times (3,727 requests against 618 real ones). The ratios, the 83.4% figure and the per-crawler counts are unchanged.
Everything above is reproducible with the Cloudflare Analytics API and your own sitemap. If you run the same check we would genuinely like to hear what you get.
Frequently asked questions
- Why is raw AI crawler traffic data unreliable?
- Because a user agent is a self-declared string that anyone can set. On our site, 83.4% of requests carrying an AI crawler user agent asked for paths that do not exist, such as SSH key files and cloud credential files. Counting user agents alone overstated real crawler volume by about six times: 3,727 requests carried a crawler user agent, and 618 of them were real.
- How can I tell real AI crawler traffic from spoofed traffic?
- Whitelist against your own sitemap. Compare each requested path to the URLs you actually publish, plus known infrastructure files like robots.txt and llms.txt. A request for a path you do not publish is not a crawler reading your site. A blacklist of suspicious patterns is not sufficient, because the probe list keeps changing.
- Which AI crawlers actually fetched content?
- In our 8-day window only three did so at meaningful volume: ClaudeBot at about 43 requests per day, OAI-SearchBot at about 13, and ChatGPT-User at about 8. Several other well-known AI user agents appeared in the raw logs but had almost no legitimate content fetches once spoofed requests were removed.
The counts in this study are published as CSV and JSON on our open data page, free to reuse with credit under CC BY 4.0.