An engine cannot cite a page it has never fetched. That sentence is the whole case for logging AI crawlers, and it is also the fastest way to find out that the page you most want quoted has never been read by anything.

Here is what thirty days of our own crawler log turned up, what the options cost, and the one difference between them that decides whether the data means anything.

Four things the log tells you that nothing else will

Which of your pages no AI crawler has ever opened.

Our comparison page, the one we most want an assistant to reach for, has been fetched by zero AI crawlers in thirty days. Not GPTBot, not ClaudeBot, not PerplexityBot. It ranks 70th in Google for its target term, and now we know it is not a ranking problem alone: nothing has read it. No amount of rewriting fixes a page that never gets fetched, and the fix is different in kind, being internal links, a sitemap that lists it, and a request for indexing.

You cannot find this in Google Search Console, which reports on Google. You cannot find it in your analytics, which reports on people.

Which content the engines actually pick up.

Page	Requests	Which crawlers
why two AI visibility checks disagree	5	bingbot, googlebot, gptbot, claude-user
what AI assistants actually cite	4	bingbot, googlebot, gptbot, perplexitybot
90 day SEO strategy template	3	bingbot only
choosing a keyword research tool	4	bingbot only
affordable SEO tools for startups	4	bingbot only
best AI visibility tools	0	none

The two at the top carry sourced statistics and cite their sources. The three below them are ordinary SEO articles and have been up for months. Both groups get Bing. Only the top group gets the AI crawlers. That is one site and one month, so treat it as a hint rather than a law, but it is the kind of hint you cannot get any other way.

Which bots ignore your robots.txt.

Our robots.txt disallows /auth/. Over the month, meta-externalagent requested paths under it 21 times. Nothing else did. Whether you care depends on the path, but you cannot decide until you can see it, and a rule you never check is a rule you assume is working.

What your server actually returns them.

Of the requests where our collector could see a status, 71 were 404 and 28 were 410. Crawlers spend a meaningful share of a small budget on pages that no longer exist. Googlebot averaged 5.4 requests a day here, so a fifth of the month's attention went to things we deleted.

What it costs

	Price	What the free level gives you
Cloudflare AI Crawl Control	free on every plan	identification by user agent string
Known Agents (was Dark Visitors)	free for most sites, then $29, then $299	event based, upgrade at higher volumes
LadderFox	free, one site	full history, verification, robots compliance
Your raw server logs	free	whatever you are willing to grep

Nobody charges much for this and Cloudflare charges nothing. If you are already behind Cloudflare, turn AI Crawl Control on today. It sits in front of your origin, so it sees every request including ones your server never answered, which nothing installed on the origin can match.

The difference that decides whether any of it is true

Every one of these tools will tell you ClaudeBot visited. The question nobody asks is how it knows.

A crawler announces itself in a header it writes about itself, and anyone can send that header. So an honest answer has three states rather than two.

Verified. The operator publishes the IP ranges its crawler uses and this request came from inside one. Google, Bing, OpenAI and Anthropic all publish such a list.

Claimed. The header says ClaudeBot and we cannot check, because the operator publishes nothing, or nothing about this address family. Not suspicious, unchecked.

Mismatch. The operator publishes ranges covering this kind of address and the request came from outside all of them. Only this state earns the word.

Cloudflare's own documentation says that on the free plan it identifies AI crawlers by their user agent strings. That is the first two states collapsed into one.

Here is why the distinction is not academic. Thirty days, 1,046 requests, our site:

Crawler	Requests	Verified	Mismatch
meta-externalagent	296	0	0
bingbot	184	166	0
googlebot	168	166	2
claudebot	103	0	102
oai-searchbot	72	61	11
amazonbot	63	0	0
applebot	58	0	0
gptbot	55	48	7
chatgpt-user	27	1	26
perplexitybot	11	3	8

Read the ClaudeBot row against the Googlebot row. Same site, same collector, same way of reading the client address. Googlebot verifies 166 times out of 168. ClaudeBot verifies zero times out of 103, and 102 of those came from addresses outside the list Anthropic publishes at claude.com/crawling/bots.json.

We checked the obvious explanations before writing that down. The range file loads and holds 26 prefixes. The mismatches spread across the whole month rather than clustering on the day the file changed. And the Googlebot column rules out our collector reading the wrong address, since that would fail for everyone.

What we cannot tell you is who sent them. A mismatch means an address sits outside a published list, and a list can be incomplete. It is a question worth having rather than an accusation.

Two other rows repay attention. Meta, Amazon and Apple show zero verified and zero mismatch, which is the third state: they publish nothing to check against, so nobody can verify them, Cloudflare included. And chatgpt-user, which fires when a person asks ChatGPT something right now, verified once in 27 requests.

One more thing about your logs

The single most requested path on our site was /robots.txt, 71 times by 5 different crawlers, more than any page of content. They check the rules constantly, which is worth knowing before you decide a robots change is not urgent.

And Bing crawled us more than Google did, 184 requests against 168. Most people assume the reverse and check only Google.

What we charge and why

Nothing. One site on the free plan, full history, verification and robots compliance included.

That is arithmetic rather than generosity. A site producing 1,046 requests a month stores 321 rows and 424 kB here, because we aggregate per bot per day rather than keeping every hit. A thousand sites would cost fourteen cents a month in storage. There is nothing to meter, and selling a thinner version of something Cloudflare gives away is a bad trade in both directions.

We had a per plan retention window in the code, 14 days on free rising to 365, and deleting it is what prompted this article. It had never been enforced, so every user already had the full history. Implementing it would have meant taking data off people in order to sell it back at a price too small to matter.

What a paid plan buys is the join: whether the pages being crawled are the pages that end up cited in answers. That costs real money, because it means asking the engines questions and reading what they say. Crawler logs cost us nothing, so they cost you nothing.

The free crawler check tells you whether AI crawlers can reach your site at all, with no account.