Verification
A User-Agent header is a claim, not an identity. Anyone can send
GPTBot/1.0, and plenty of scrapers do precisely that, because sites tend to
treat crawlers from big AI companies generously.
So Snow doesn’t take the claim at face value. Every crawler hit carries one of four verification states.
The four states
| State | Badge | What it means |
|---|---|---|
| Signed | green | The request carried a cryptographic signature that the operator’s published key validates. |
| Verified | green | The request came from an IP address the operator publishes as theirs. |
| Unverified | grey | Snow could not confirm the claim either way. |
| Spoof suspected | amber | The claim is contradicted by evidence. |
Signed
The strongest. Web Bot Auth (RFC 9421) lets a crawler sign its request with a private key, publishing the matching public key in a directory. Snow rebuilds the signature base and checks it.
This is proof, not inference: a signature cannot be forged without the operator’s private key, regardless of what IP it came from.
Verified
Most operators publish the IP ranges their crawlers run from. Snow refreshes those ranges from about 15 official sources (OpenAI, Anthropic, Google, Perplexity, Bing, Apple, Mistral, Moonshot, DuckDuckGo, Common Crawl, Meta and others) every six hours, and checks each hit’s source address against them.
Where an operator publishes no ranges but does use identifiable hostnames, Snow falls back to a reverse-DNS lookup, confirmed by a forward lookup back to the same address.
This is inference rather than proof, but strong inference: the request really did come from the operator’s infrastructure.
Unverified is not a bad grade
This is the distinction that matters most, and the one that’s easiest to misread.
Unverified means “no evidence either way.” Spoof suspected means “evidence against.” They are completely different claims, which is why they are separate states rather than a single “not verified” bucket.
A hit is unverified when Snow has nothing to check the claim against:
- The operator publishes no IP ranges and no signing key. Plenty of legitimate, well-behaved crawlers are in this position.
- A reverse-DNS lookup timed out, or the API had no outbound network access at that moment.
- The crawler is new enough that its ranges aren’t in the feeds yet.
None of that is suspicious. A large share of perfectly genuine crawler traffic is unverified, and a low verified percentage usually says more about which crawlers visit you than about anyone lying.
A hit is spoof suspected only when the evidence actively contradicts the
claim. Most often that is a User-Agent naming an operator whose published
ranges Snow does have, arriving from an address outside every one of them.
That is a positive finding.
Even so, the badge is amber, not red. “Suspected” is a warning, not a verdict: a misconfigured corporate proxy or an operator who has just added capacity without updating their published ranges can both produce it.
Using the Verified filter
The Verified only toggle on the AI Traffic section restricts every panel to
signed and verified hits.
- Leave it off to answer “how much AI crawler traffic do I get?”, since the honest total includes unverified hits.
- Turn it on to answer “what can I stand behind?”, for a report to someone else, or when you suspect a scraper is inflating a number.
Filtering out unverified traffic will usually cut the totals substantially. That is expected, and does not mean the rest was fake.
Privacy
Snow never stores raw IP addresses for crawler hits, the same as for human analytics.
Verification runs on the raw address in memory, at the moment the hit is received. Only the resulting verdict, plus a salted hash of the address, is stored. The hash lets Snow group hits from a single source without ever being able to recover the address.
Country and network-operator (ASN) enrichment happens in that same moment, before the address is discarded.
When verification can’t run
Verification needs outbound network access for the range feeds, occasional DNS lookups, and key-directory fetches. If your Snow instance is self-hosted with restricted egress, every hit degrades to unverified.
Reporting still works and every crawler is still recorded and categorised; only the Verified metric loses its meaning. If you’re self-hosting, confirm outbound HTTPS and DNS are permitted before relying on that number.