A page can rank on Google, load perfectly in a browser, and still be functionally invisible to ChatGPT, Claude and Perplexity at the same time. Not because the content is wrong, and not because of a penalty. It's because of how the page gets built, and most sites built in the last several years were built the way that causes exactly this problem.
What Vercel and MERJ actually found
Vercel and MERJ analyzed real crawler traffic across nextjs.org and several other production sites, tracking hundreds of millions of requests from the major AI crawlers over a full month. GPTBot alone generated 569 million requests across Vercel's network in that window, and Claude's crawler added another 370 million — a combined volume equal to roughly a fifth of Googlebot's traffic over the same period. The finding that matters most: Google is the only major crawler with full JavaScript rendering support. Every other AI crawler in the study fetched the raw HTML a server returned and nothing more.
The gap between "fetched" and "rendered"
The detail that removes any ambiguity here is that these crawlers aren't ignoring JavaScript out of laziness. GPTBot downloaded the actual JavaScript files on the page roughly 11.5% of the time, and Claude's crawler did the same closer to 24% of the time — likely pulling the code in as raw text for training data. Neither one ever executed it. If a page's real content only appears after that script runs in a browser, a client-rendered React or Vue app with no server-side rendering serves these crawlers an empty shell: a root div, some script tags, nothing to read. The one meaningful exception is Google's Gemini, which inherits Googlebot's existing rendering infrastructure rather than running its own.
Why AI crawlers hit so many dead ends
The same research surfaced a second, less-discussed gap: GPTBot and Claude's crawler returned 404 errors on roughly 35% of their fetches, compared to about 8% for Googlebot over the same window. Part of that is AI crawlers working from older or incomplete link data rather than a live, current sitemap. Part of it is these systems being newer and less forgiving of redirect chains, broken internal links, and URL structures that Google has spent two decades learning to work around quietly. Either way, a site with genuinely clean internal linking and an accurate sitemap is handing these crawlers a meaningfully easier job than most of the web currently does.
What this actually means for how a site gets built
None of this is a reason to abandon JavaScript-heavy frameworks — it's a reason to be deliberate about where content lives. Server-side rendering, static site generation, or a hybrid approach that renders the core content server-side while keeping interactive elements client-side all solve this the same way: the words a person or a crawler needs to read are already sitting in the HTML the server sends back, before any script runs. A useful gut check: view a page's actual source, not the rendered DOM in browser dev tools, and check whether the real content is there. If it only shows up after the page finishes loading, that's what GPTBot and Claude's crawler see instead of your content: nothing.
This is also, incidentally, one of the more mundane reasons a simple, server-rendered site has an advantage it doesn't always get credit for. There's no rendering step to fail because there's no client-side rendering step at all — the HTML a crawler fetches is the actual page, the same one a person sees. That's not a clever optimization. It's closer to just not having built the problem in the first place.
Technical SEO work has always included crawlability and indexation basics — robots.txt, sitemaps, redirect chains, canonical tags. What's changed is the audience for getting those basics right. It used to be mostly about Googlebot. Now it's a growing list of AI crawlers with less mature infrastructure, less patience for a broken link, and in most cases, no ability to wait for a page to finish rendering at all.