You see a full page. The agent receives a raw HTML file. If your content is assembled in the browser by JavaScript, there is a real chance that what reached the other side was an empty shell. The only public measurement I could find on this — by Vercel and MERJ, December 2024 — concluded that none of the major AI crawlers execute JavaScript; they do fetch the JS files (11.50% of ChatGPT’s requests, 23.84% of Claude’s) but never run them. Google is the exception: it renders with an evergreen version of Chromium, though it too stops after 2MB. The test itself takes two minutes and one line of curl.
The only question that matters: what is in the initial HTML
There is a difference between what your site displays and what your site sends. The browser receives an HTML document, downloads the JavaScript files, runs them, makes further API calls, and assembles the page you see out of all of that. The entire chain happens on the user’s machine, not on your server. An agent that requests the URL and stops there receives only the first step.
On most classic content management systems this is a non-issue, because the HTML arrives from the server fully formed. The problem begins when the site is built as a client-side application: React, Vue or Angular generating the content after load. In that case the document leaving your server can be three kilobytes of empty tags with a <div id=”root”></div> in the middle, and everything else is constructed only in the browser.
One point deserves emphasis: this is not a permissions question. You can allow every bot full access, return 200 on every request, and still appear in no answer at all — simply because there was no content to read. I took apart the permissions layer separately in the 2026 bot map; this is the layer that comes after it.
What has actually been measured — and what has not
This subject needs care, because it is full of numbers circulating without a source. What does exist: a joint study by Vercel and MERJ published on 17 December 2024, analysing real logs from large sites. Its central finding is that none of the major AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others — execute JavaScript. They download the JS files but do not run them. The exception in their list is AppleBot, which renders through a browser.
The same study also gives a sense of volume: roughly 569 million monthly requests from GPTBot and around 370 million from Claude, together about a fifth of Googlebot’s request volume in that period.
A caveat I am obliged to state: that measurement is from December 2024, on a limited sample of sites. It is not an internet-wide measurement and not a 2026 figure. I checked the official documentation of OpenAI, Anthropic and Perplexity — none of the three documents whether its bots render JavaScript at all. That absence is a finding in itself: you cannot rely on a promise that was never made, so the only safe assumption is that the content has to be in the HTML.
Google is the exception, not the rule
Google Search Central states explicitly, on a page last updated 4 March 2026, that Google Search runs JavaScript with an evergreen version of Chromium, and that Googlebot queues all pages returning a 200 status code for rendering. In the same document Google adds the caveat most relevant here: not all bots can run JavaScript.
Two practical Google limits are worth knowing. First, per the Googlebot documentation, the crawler fetches the first 2MB of a supported file type and stops there. If your HTML is heavier than that, everything past the cutoff simply is not read. Second, on AI-based features, Google states that there are no additional technical requirements beyond the page being indexed and eligible to appear with a snippet. There is no special file and no dedicated markup; whatever reads well in Search reads well there too.
Three patterns that empty out the page
Client-side rendering. The classic case. The document from the server contains a shell, and the content loads afterwards. Easy to spot: the page source is far shorter than what the eye sees.
Content that only opens on interaction. Accordions, tabs, “load more”, infinite scroll. If the content is injected into the DOM only after a click, it does not exist in the document. If it exists in the HTML and is merely hidden with CSS, it is there, and that is perfectly fine.
Blocking layers in front of the content. A cookie banner that requires consent before the content loads, a paywall, a login modal, or a bot-management layer returning a challenge. In those cases the bot does get a page — just not yours.
How to check it in two minutes
No external tool is needed. Three lines are enough to know exactly what leaves your server:
curl -sA “GPTBot” https://example.com/page/ -o raw.html
wc -c raw.html
# 2. How many words of visible text does it actually contain
curl -sA “GPTBot” https://example.com/page/ \
| sed -e ‘s/<script[^<]*<\/script>//g’ -e ‘s/<[^>]*>/ /g’ \
| tr -s ‘ \n’ ‘ ‘ | wc -w
# 3. Is a key sentence from the page really in the document
curl -sA “GPTBot” https://example.com/page/ | grep -c “your key sentence”
Interpretation is simple. If line 2 returns a number close to the word count you see on the page, everything is fine. If it returns 40 words for a 1,200-word page, the content is being built in the browser. If line 3 returns 0, your central sentence exists in no answer that will ever be generated from that page. It is also worth running the same check with an ordinary browser user-agent and comparing: a large gap between the two indicates a server serving different content to bots.
How to confirm the bot is genuine
Before drawing conclusions from your log, make sure whoever claimed to be GPTBot actually is GPTBot. All the major vendors publish official IP address lists, and Google also publishes a reverse-DNS verification method:
| Vendor | Official verification source | Note |
|---|---|---|
| OpenAI | openai.com/gptbot.json openai.com/searchbot.json openai.com/chatgpt-user.json | A separate file per bot |
| Anthropic | claude.com/crawling/bots.json | One file covering all bots |
| Perplexity | perplexity.com/perplexitybot.json perplexity.com/perplexity-user.json | A separate file per bot |
| developers.google.com/crawling/docs/ crawlers-fetchers/verify-google-requests | Both IP ranges and reverse-DNS verification against googlebot.com |
Google is the only one of the four offering reverse-DNS verification. With the others, comparing the IP against the official list is the only check available. It is worth pulling these files automatically rather than copying ranges by hand — they change.
What to do when the answer is “the page is empty”
The fix depends on the architecture, but the direction is the same in every case: move content assembly from the browser to the server.
- Server-side rendering or static generation. In modern frameworks this is a per-page setting, not a rewrite. The result is complete HTML in the very first response, which also improves load time for human users.
- Make sure core content does not depend on interaction. An accordion whose content sits in the HTML and is hidden with CSS is a perfectly valid solution. An accordion that fetches its content on click is not.
- Check what happens before consent. If the cookie banner blocks the content until a click, the bot sees nothing. The content needs to be in the document even when nothing has been clicked.
- Measure again; do not assume. After every change, the same curl line. The check takes seconds, and assuming it worked is exactly what created the problem in the first place.
And if the broader question interests you — why the web was built around an assumption that no longer holds — I wrote about that in your site was built around a gravity that no longer exists.
A practical order of priorities
If you have an hour, this is the order I recommend. First, run the curl check on the three most important pages on the site — the homepage, the main service page, and the article with the most traffic. Then compare the word count to what is actually visible. Only if there is a gap should you open the architecture question. On most WordPress and marketing sites the answer will be that everything is fine, and that is a perfectly good outcome — better to know than to spend a month solving a problem you do not have.
Frequently asked questions
Do AI crawlers execute JavaScript?
None of the major companies documents this officially. The only public measurement that exists, by Vercel and MERJ in December 2024, found that none of the major AI crawlers execute JavaScript — they download the files but do not run them. Absent an official commitment, the safe assumption is that content must be present in the initial HTML.
Does Googlebot render?
Yes. Google states that Search runs JavaScript with an evergreen version of Chromium, and that every page returning a 200 status code is queued for rendering. There is a technical limit: Googlebot crawls the first 2MB of a supported file type and stops there.
How do I know whether my content is in the HTML?
Fetch the page with curl, with no browser involved, strip the tags and count words. If the number is close to what you see on the page, the content is server-rendered. If it is significantly lower, the content is built in the browser and never reaches the agent.
Do I need a special file to appear in Google’s AI features?
No. Google states that there are no additional technical requirements, and that the condition is for the page to be indexed and eligible to appear in results with a snippet.
How do I confirm a bot in my log is genuine?
OpenAI, Anthropic, Perplexity and Google all publish official IP lists as JSON files. Google additionally publishes a reverse-DNS verification method against the googlebot.com domain. With the other vendors, cross-checking against the IP list is the only verification available.