JEEV TRIKA

Digital marketing and AI · 5 min read

What a Crawler Actually Receives

Your browser shows the page after the scripts have run. Most AI crawlers never see that page. They see the HTML your server sent, and nothing else.

By

Open your website in a browser and you see the finished page: text, menus, prices, reviews, all assembled. A crawler may see something quite different. What it receives is the HTML your server sends in response to a request. If the content is added afterwards by JavaScript, some crawlers will wait for it and many will not.

That gap is the most common reason a company’s site says less to an answer engine than its owners think it does.

There is no single “AI crawler”

The companies behind the major assistants now publish separate agents for separate jobs, and they behave differently. As of their current documentation:

OpenAI runs GPTBot to “crawl content that may be used in training our generative AI foundation models”; OAI-SearchBot, “used to surface websites in search results in ChatGPT’s search features”; and ChatGPT-User, for “certain user actions in ChatGPT and Custom GPTs,” where OpenAI says “robots.txt rules may not apply” because a person asked for the page. Blocking OAI-SearchBot means a site will not appear in ChatGPT’s search answers, though it may still show as a navigational link.

Anthropic runs ClaudeBot, which collects web content “that could potentially contribute to” model training; Claude-User, used when “individuals ask questions to Claude”; and Claude-SearchBot, which “navigates the web to improve search result quality for users.” Anthropic says its bots honour robots.txt.

Perplexity runs PerplexityBot, “designed to surface and link websites in search results on Perplexity” and, it says, not used for foundation-model training; and Perplexity-User, which supports user requests and “generally ignores robots.txt rules.”

Google is the odd one out. AI Overviews and AI Mode are built on Google Search, so the crawler that matters for them is ordinary Googlebot. Google-Extended is a separate control over whether content can be used to train and ground Gemini models outside Search, and Google states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

So “should we block AI crawlers?” is really several questions. Blocking training crawlers and blocking search crawlers have very different consequences, and blocking Google-Extended does not take you out of AI Overviews.

Many of them do not run your JavaScript

Googlebot renders pages. Google describes crawling, rendering and indexing as separate phases, with pages queued for rendering, and still recommends server-side or pre-rendering because “not all bots can run JavaScript.”

Most AI crawlers are among those bots. When Vercel analysed AI crawler traffic across its network in December 2024, it found that OpenAI’s and Anthropic’s crawlers, along with Meta’s, ByteDance’s and Perplexity’s, did not execute JavaScript. They fetched script files, in Claude’s case nearly a quarter of its requests, but did not run them. The crawlers that did render were Google’s, through Googlebot’s infrastructure, and Apple’s.

Crawler behaviour changes, and that is a 2024 measurement. The safe design assumption has not changed: if the words only exist after a script runs, assume some of the systems you care about will never see them.

Why company websites go quiet

Put those two facts together and the common failures are predictable:

  • The content is assembled in the browser. A site built as a JavaScript application can send a nearly empty HTML shell, with the services, team and case studies arriving later.
  • The important text is not text. Prices in images, credentials in a logo strip, a capabilities deck as a PDF, a video with no transcript.
  • The facts are scattered. Who the company is, what it does, where, for whom and since when is spread across sliders, tabs and footers rather than said plainly on one page.
  • The pages are missing. Vercel also found AI crawlers hitting dead URLs far more often than Googlebot: about a third of ChatGPT’s and Claude’s requests ended in 404s, against about 8% for Googlebot. Redesigns that break old addresses without redirects make that worse.

None of this is new to anyone who did technical SEO for Google ten years ago. What changed is that there are now more readers with less patience.

What this site sends

When I rebuilt jeevtrika.com in 2026 I made the server’s response the whole page. Every page is pre-rendered at build time, so the article you are reading was complete HTML before anyone requested it. The raw HTML of a typical article on this site carries all of its words. The entire site ships about 1.6 KB of JavaScript, and none of it is needed to read anything.

Each page also carries structured data describing the person, the organisations, the places and the article, linked together, and it matches what is visible on the page. The robots file names the AI agents and allows them. The seventeen articles that moved from old addresses all have permanent redirects.

The site also publishes an llms.txt file, a plain-text summary for language models. That one deserves a caveat: it is a proposal, not a standard, and Google says you do not need it to appear in its AI features. It is cheap to provide. It is not a lever.

How to check your own

Fetch a page the way a crawler does, without a browser: view the page source, or request it from the command line, and read what comes back. If your services, prices and proof are not in that response, a large share of answer engines cannot see them either. Then look at your robots file and make sure you are blocking what you meant to block, and only that.

Sources

Auditing what your own pages hand back to a crawler is usually the first thing I do when a client brings me their search visibility.

Related: Built for Google, Rebuilt for Answers and A Website From Scratch, in 2026.