Why a JavaScript Snippet Cannot Count AI Crawlers

The measurement everyone gets wrong

Almost every AI-crawler tracker on the market works the same way: you paste a small JavaScript snippet into your site, it fires on page load, and it reports the visitor's user-agent. If that user-agent looks like GPTBot, you get a crawler visit in your dashboard.

There is one problem with this, and it is fatal: AI crawlers do not execute JavaScript. They request your HTML and leave. The snippet never runs, so the visit is never reported.

Why this is worse than simply missing data

A tracker that undercounts is annoying. A tracker that undercounts silently is dangerous, because the visits it misses look exactly like visits that never happened. There is no error, no gap in a chart, no warning. You see a low number and reasonably conclude that AI engines are not crawling you — when in fact they may be crawling you constantly.

We ran both methods against the same site over the same period. The JavaScript snippet reported one crawler. The server logs, for the same window, reported six: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider.

What server logs show that a snippet cannot

Reading the access log instead of the browser changes what is knowable:

• Every request, not just rendered pages. A crawler that fetches robots.txt, a JSON-LD file or an image appears in the log and never in a snippet.
• The status code the crawler received. This is the one most people have never seen. In our own logs, PerplexityBot received a 403 and Bytespider a 404. "The bot came and your site turned it away" is a finding in its own right, and a JavaScript beacon can never surface it — by definition it only runs on pages that rendered successfully.
• Crawl frequency and path coverage. Which sections get crawled often, and which never get crawled at all.
• Crawlers you did not know to look for. A snippet only reports agents it was told about; a log contains everything that asked.

How to set it up

Both major hosting platforms already stream logs to an HTTP endpoint on their own schedule, so nothing needs polling and no credentials change hands:

• Vercel — Settings, then Log Drains. Add a drain in NDJSON format pointing at your ingestion URL, with a custom secret. Vercel signs every delivery with it.
• Cloudflare — Analytics & Logs, then Logpush. Create a job for the HTTP requests dataset with an HTTP destination, and include at minimum ClientRequestUserAgent, ClientRequestURI, EdgeResponseStatus and EdgeStartTimestamp.

If you are on neither, any system that can POST newline-delimited JSON works — the fields are read by alias, so most access-log shapes are understood without transformation.

Allowing a crawler is not the same as being cited

One caveat worth stating plainly, because it is where a lot of AEO advice overpromises: letting crawlers in is necessary and nowhere near sufficient. In our measurements, engines frequently crawled and cited a site's pages and then recommended a competitor anyway — Perplexity cited one brand's own site in 22 answers and named that brand in none of them.

Crawler analytics tells you whether you are reachable. It does not tell you whether you are persuasive. You need both numbers, and you need to not confuse them.

Frequently asked questions

Do any AI crawlers execute JavaScript?

Google's rendering infrastructure does for traditional search indexing, and that capability partly informs AI Overviews. The dedicated AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Bytespider, Meta-ExternalAgent — do not. Treat JavaScript-rendered content as invisible to them.

Will server-log tracking slow down my site?

No. Logs are pushed by your CDN after the fact, on its own schedule. Nothing is added to the request path and no code runs on your pages.

What about content rendered by JavaScript?

The same blind spot applies to your content, not just your analytics. If your FAQ, pricing table or product copy is injected by a script, AI crawlers receive an empty container. Render it server-side and let JavaScript enhance it, rather than the other way round.

Does allowing AI crawlers in robots.txt guarantee citations?

No. It makes citation possible. Whether an engine then names you depends on whether other sources corroborate what your page says — see our guide on building authority for AI search.