Why Some Websites Are Invisible to ChatGPT Without Their Owners Knowing

A website can disappear from ChatGPT answers without its owner choosing to block the service. Hosting defaults, security settings and old technical rules can stop AI crawlers from reading a site, leaving businesses unaware that assistants cannot find or cite their pages.

Cloudflare changed a default on July 1, 2025, so new domains on its network block known AI crawlers unless the owner turns the setting off. Cloudflare sits in front of roughly a fifth of the public web. Separately, 6sense research found that 94% of B2B buyers use large language models at some point in the purchasing process. That makes crawl access a visibility issue for companies whose prospective customers use AI assistants to research products and form shortlists.

Three ways crawlers get blocked

A CDN or security service may challenge or block AI crawlers while letting human visitors browse normally. A crawler that receives a challenge page or a 403 error may record the failure and move on. The site owner may see no obvious problem.

A robots.txt file can also preserve a decision made for a different purpose. Some sites blocked GPTBot in 2023 over training-data concerns. OpenAI now has separate crawlers for training, search indexing and fetching a page in response to a user request: GPTBot, OAI-SearchBot and ChatGPT-User. Blocking one does not necessarily mean blocking all three, but broad rules can keep a site out of ChatGPT search.

The page itself may be difficult for a crawler to read. A Vercel and MERJ crawl-log study published in December 2024 found that none of the major AI crawlers it examined executed JavaScript. On single-page applications, product details, pricing or documentation that appear only after a script runs may therefore be missing from what a crawler sees.

Training access is not search access

The choice is not simply whether to let AI companies use a site’s content. Training crawlers collect material that may be used in a model, while search and retrieval crawlers fetch pages to answer live questions and cite their sources. A business may want to block training while allowing search crawlers to reach its pages.

For site owners, the practical issue is to check which crawlers their CDN, security tools and robots.txt rules allow, then consider whether key pages are readable without JavaScript. The article does not establish how many sites are affected or how often blocked pages change an assistant’s answer. But when a crawler cannot fetch a page, it cannot cite that page in its response.

This report draws on information from Yonkers Times.