AI Crawlers Saw a Blank Page on Our Site — Adopting Static Prerendering for a SPA

The Door Was Open, but the Room Was Empty

In the previous post we covered AI crawler allow settings. But however perfectly you open robots.txt, it means nothing if there is no body for crawlers to read. Our own site was in exactly that state. marketing-pivot.com is a React client-side rendered (CSR) SPA. A person's browser runs JavaScript and draws the screen, but the HTML a non-JavaScript crawler received was nothing more than <div id="root"></div>.

We diagnosed this twice, on August 30 and September 8, 2026. For crawlers like GPTBot, ClaudeBot, and PerplexityBot that do not execute JavaScript, a site that talks about AEO and GEO was, to AI, a blank page.

Why This Happens

A CSR SPA has the server send only a shell HTML and scripts; the actual body is built after JavaScript runs in the browser. For a person there is no difference, but for a collector that does not render, the body does not exist. It is generally understood that many AI training and search crawlers do not execute JavaScript, though this varies by service and can change. So instead of assuming they will, the safer design makes your content visible even when they do not.

How to Check It Yourself

It is simple. Look at view-source or a curl response, not the browser element inspector. The inspector shows the result after JavaScript has run, so it is not a valid test. Search the raw response for a key sentence from your body; if you cannot find it, assume crawlers that do not execute JavaScript cannot see it either.

Comparing the Options

Approach	Strength	Burden
Move to SSR (Next.js, etc.)	Server returns completed HTML on every request	High migration cost and server operations
Build-time static prerendering	Keeps the SPA structure; generates static HTML at build	Poor fit for frequently changing data-driven pages
Dynamic rendering (detect crawlers, serve separately)	Rendered output only for crawlers	Complex detection logic and risk of content mismatch

We chose the second. Most of our pages are static marketing content such as blog posts, guides, and product introductions, and we wanted to avoid the cost of restructuring the app.

What We Did

After the build finishes, a script opens every route registered in sitemap.xml in a headless browser against a local preview server, saves the rendered result as static HTML, and overwrites it in the output folder. Because Vercel serves static files before routing rules, crawlers receive completed body content without changing the SPA structure at all. We started with 56 routes and now regenerate 68 on every build.

We hit a snag along the way. A build that worked locally failed in Vercel's build container: the bundled Chromium needed shared libraries the container did not have, and without root access we could not install them. So only on Vercel builds we branch to a separately packaged, statically built Chromium meant for serverless environments.

Problems We Found Along the Way

Prerendering let us see the site through a crawler's eyes for the first time, and several defects that normally stay hidden came out with it.

Problem Found	Cause and Fix
Blog post title was an empty string	React 19's title handling only supports a single text child, but the title and a fixed string split it into two. Merged into one string
Default meta tags duplicated on every page	index.html's default title, description, and canonical stayed in the head next to per-page values. Cleaned up to keep one per tag type
New posts missing from prerendering	Posts missing from the sitemap were missing from prerendering, so crawlers got the home shell. Connected automatic sitemap generation to the build
llms.txt actually returned the home HTML	With no file present, the SPA catch-all rule served index.html. Added the real file
FAQ answers absent from the DOM	Answers were not mounted while collapsed, so crawlers saw only questions. Now always rendered, with collapsing handled by style only

What this list shows is that rendering problems rarely come alone. Auditing once from a crawler's point of view surfaces neighboring defects in meta tags, sitemaps, and static file handling.

What We Still Don't Know

There is something we should say plainly. We have not yet measured how citations in AI answers changed before and after adopting prerendering. It is true that crawlers can now read the body, but there is no evidence that this translates into more citations. Prerendering is not a device that creates citations; it is work to meet the minimum condition for being citable.

Self-Check -- Where to Start Right Now

First, curl your key pages and check whether the body is visible. Then run the structural diagnostic in AEGIS Insight to check AI crawler allow status. That structural diagnostics are only a leading indicator of citation readiness and do not tally citations themselves is covered in our AEO KPI design post.

Closing Thoughts

AI crawler readiness does not end with allow settings. You have to confirm there is actually body content inside the door you opened for the setting to mean anything. If you run a SPA, open view-source once right now. If all you see is an empty shell, that is the first item on your AEO to-do list.