We Verified Every AI Crawler That Visited Us for 15 Days. 45% of Checkable Claims Were Forged.
Short answer: between 9 and 23 October 2026 our web server received 76,848 requests that named an AI or search crawler in their user-agent string. Where the operator publishes IP ranges we could check the claim. 8,324 of those 18,617 checkable claims were false — about 45 percent. They came from 777 distinct addresses, and several single rented servers were presenting nineteen different crawler identities each.
We are publishing this because we could not find anyone else who had measured it on a real site and shown their method.
Why this needed checking
A user-agent string is not an identity. It is a line of text the client chooses, and any program can set it to anything. When your analytics tells you "ClaudeBot visited 110 times", what it actually knows is that 110 requests contained the word ClaudeBot.
That distinction has become commercially interesting. Site owners now watch AI crawler activity as a proxy for whether assistants are reading their content. If that number is inflated by forgeries, people are making decisions on a measurement that does not mean what they think.
Method
Three tests, applied to the connecting IP address rather than the claimed name.
1. Published range lists
Several operators publish the IP ranges their crawlers use, specifically so you can verify them:
- OpenAI: separate lists for
GPTBot,OAI-SearchBotandChatGPT-User - Google:
googlebot.json,special-crawlers.json,user-triggered-fetchers.json - Microsoft:
bingbot.json - Perplexity: a single published list
A request claiming one of these, from an address outside the operator's ranges, is forged.
2. Forward-confirmed reverse DNS
Where no list exists, the method Google and Microsoft both document: take the connecting IP, look up its PTR record, check the hostname belongs to the operator, then resolve that hostname forward and confirm it returns the same IP. The forward step is essential — whoever controls an address also controls its PTR record, so a PTR alone proves nothing.
3. Identity rotation
The strongest test, and it requires no list at all. If one address claims to be crawlers operated by two or more different companies, it is lying. No real crawler is simultaneously Anthropic, OpenAI and Perplexity.
This test matters because it works against the operators nobody publishes ranges for.
Results
| Verdict | Requests | What it means |
|---|---|---|
| Verified genuine | 10,293 | Address matched the operator's published ranges or forward-confirmed DNS |
| Forged | 8,324 | Claimed a crawler we could check, and failed |
| Unverifiable | 55,607 | Operator publishes nothing to check against; no claim made either way |
Among the checkable claims, the false share by crawler:
| Claimed identity | Genuine | Forged |
|---|---|---|
| Googlebot | 7,921 | 5,479 |
| GPTBot (OpenAI) | 1,303 | 694 |
| bingbot | 965 | 698 |
| ChatGPT-User | 160 | 755 |
| OAI-SearchBot | 142 | 729 |
The pattern worth noting: the training crawlers are forged least and the user-facing ones most. Over four in five requests claiming to be ChatGPT fetching a page on a user's behalf were not ChatGPT.
The part that surprised us
The forgeries were not spread thinly. 777 addresses produced all of them, and the busiest were Google Cloud virtual machines — reverse DNS *.bc.googleusercontent.com — each cycling through nineteen crawler identities. One rented server, presenting itself as nearly every AI company in the industry, in rotation.
What were they doing? Mostly probing for credentials: requests for /.env, /.env.local, /proc/self/environ and directory-traversal patterns. They almost certainly wear crawler names because site owners are inclined to let crawlers through.
None of it succeeded, which is a separate matter from whether it distorted our measurements.
Two mistakes we made, and what they cost
Our first run of this analysis produced two false accusations, and they are instructive.
We flagged real Googlebot as an impostor. Addresses in 66.249.77.x, with reverse DNS crawl-66-249-77-x.googlebot.com — genuine Google infrastructure — were caught by our rotation test because they presented both Googlebot and GoogleOther. Those are two crawlers run by the same company from one fleet. The fix: count distinct operators, not distinct crawler names.
We were checking one crawler against the wrong list. GoogleOther was being validated only against Google's special-crawlers list when it also appears in the main Googlebot ranges. Checked properly, all 109 of its requests are genuine. Checked carelessly, all 109 looked forged.
Both errors ran in the same direction: toward accusing legitimate crawlers. We now apply a deliberately conservative rule — forgery is only asserted where the operator's published list is known to be complete. For Anthropic, Apple and Perplexity, where it is not, a failed check records as unknown rather than forged. The rotation test still catches the real impersonators without needing any list.
If you run this analysis on your own logs, expect to make both mistakes.
One more trap: the address you are logging
If your site sits behind a CDN, your web server may be recording the CDN's address rather than the visitor's. Ours was. Every attacker appeared as a Cloudflare IP, which makes attribution impossible and makes IP-based blocking actively dangerous — a naive blocklist built on those logs would have banned the CDN and taken the site offline.
The real address was available the whole time in the CF-Connecting-IP header. Check which one your logs actually contain before you trust any of this.
How to repeat this on your own site
- Confirm your logs record the visitor's IP, not your CDN's.
- Pull the operators' published range files and match each crawler request against them.
- Where no list exists, use forward-confirmed reverse DNS — both directions, not just the PTR.
- Flag any address claiming two or more operators' crawlers.
- Keep "unverifiable" as a separate bucket. Resisting the urge to collapse it into a verdict is most of the discipline.
Our figures cover one mid-sized site over fifteen days. We would be interested to know whether the 45 percent holds elsewhere.
