Is your site blocking AI? How to find and fix AI visibility blockers
Your site may be blocking AI without anyone having decided to. ChatGPT, Claude and Perplexity can be turned away in five places: your robots.txt, your firewall or CDN, tags and headers on each page, pages that only fill in with JavaScript, and walls such as logins.
Most of these blocks are invisible in a browser, so the site looks fine to everyone who checks it. This guide shows how to test each one, what the results mean, and how to fix what you find.
Key takeaways
- A site can turn AI crawlers away in five places: robots.txt, the firewall or CDN, page tags and headers, JavaScript, and walls such as logins and pay-per-crawl.
- In robots.txt, a group that names a crawler replaces the general rules for it. One misplaced group can block or open far more than intended.
- Firewalls are the blocker we meet most often: 6 of 32 well-known sites we tested turned our crawler away.
- Since 15 September 2026, Cloudflare's Block settings for AI also block Googlebot and Bingbot. To keep out training only, use its Disallow AI Training option.
- Blocking training crawlers is a fair choice. To be cited, keep the search crawlers and on-request fetchers allowed.
How can you tell if ChatGPT can see your website?
You can tell with three quick tests, each catching something the others miss:
- Ask it. Give ChatGPT the address of a specific page and ask it to summarise that page. If it says it cannot open the page, or describes something that is not on it, something is in the way.
- Pretend to be it. Request your home page with an AI crawler's name, then as a browser, and compare. The commands are below.
- Look for its visits. Your server or CDN logs record each crawler by name. No visits from OAI-SearchBot in a month usually means it is being stopped.
The first test only proves that ChatGPT-User, which fetches pages when a person asks, can get in. The search crawler, OAI-SearchBot, decides whether you appear in ChatGPT search answers, and can be blocked separately.
OpenAI publishes the IP address ranges each of its crawlers uses, so you can confirm that a visit in your logs is really from OpenAI.
Blocker 1: Is robots.txt turning AI crawlers away?
robots.txt is the first thing a crawler reads, and the most common mistakes are simple ones. A Disallow: / left over from a staging site. A plugin or security tool that added AI crawlers to the file without asking.
A rule written for one crawler that caught others with similar names.
The subtle one is how groups work. Under the robots.txt standard, RFC 9309, a crawler that finds a group with its own name follows that group only, and ignores the * group completely.
So a single named group can quietly cancel every general rule you wrote.
If your site is on Cloudflare, check whether it is serving a managed robots.txt. Cloudflare can add its own rules above yours, disallowing training crawlers such as GPTBot, ClaudeBot and Google-Extended.
Those rules leave search crawlers alone, but you should know they are there, because they will not appear in the file in your own codebase.
After changing robots.txt, give crawlers time to read it again. OpenAI says on its crawler page that an update can take about 24 hours to affect its search results. Our AI crawlers list has the name of every crawler and what each one is for.
Blocker 2: Is a firewall or CDN blocking AI crawlers?
A firewall can refuse AI crawlers even when robots.txt invites them in, and it is the blocker we meet most often. When we tested our checker on 32 well-known websites in October 2026, we found that 6 of them (19%) turned our crawler away at the firewall.
Test yours by requesting your home page twice, once with a crawler's name and once as a browser, and comparing the status codes:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.3)" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 (Macintosh)" https://example.com/
Each line prints a status code and the size of the page. Repeat with PerplexityBot, Claude-SearchBot and ChatGPT-User. On Windows, type curl.exe rather than curl.
Some firewalls check where a request really comes from, so treat a block here as a strong warning, then confirm it in your firewall's settings.
If your site is on Cloudflare
Cloudflare now sorts AI bots into three kinds: Search, which indexes pages to answer questions later; Agent, which fetches pages for a person in real time; and Training.
Each can be blocked on all pages, blocked only on pages with ads, or allowed. You will find them under Security Settings, in Configure AI bot policies.
One change matters more than the rest. Since 15 September 2026, Cloudflare's Block settings also apply to mixed-use crawlers such as Googlebot, Bingbot and Applebot, which crawl for both search and training. Blocking training that way can take you out of Google and Bing as well.
If you want to keep your content out of AI training but stay in search, choose Disallow AI Training for the Training category instead. Allow Search, so you can be cited, and allow Agent, so ChatGPT and the others can open your pages when a person asks about them.
Other firewalls and hosts
Elsewhere the setting goes by other names: bot protection, bot management, bot fight mode, or a rule that blocks unusual user agents.
Look for anything that challenges or refuses automated visitors, and allow the search crawlers by name. Strict rate limits can do the same job by accident, refusing a crawler that asks for several pages in a row.
Blocker 3: Do your pages tell crawlers to stay away?
A page can carry its own instructions. In the HTML, a robots meta tag with noindex keeps a page out of search, and nosnippet or max-snippet:0 forbids quoting it.
Search for those words in View Page Source on your key pages.
The same instructions can also arrive in an HTTP header called X-Robots-Tag, which never appears in the page source. Google confirms that any rule that works in a robots meta tag also works as an X-Robots-Tag.
To see the headers, run curl -I https://example.com/your-page and look for that line.
Blocker 4: Is your content only there after JavaScript runs?
Most AI crawlers read the HTML your server sends and do not run JavaScript. If your pages are built in the browser, those crawlers receive an almost empty page. In our test of 32 sites, 2 delivered pages that were empty until JavaScript ran.
To check, open a key page, choose View Page Source rather than Inspect, and search for a sentence from the middle of the page. If it is missing, the fix is server-side rendering, pre-rendering or static generation. Google recommends the same, noting in its JavaScript guidance that "not all bots can run JavaScript".
Look out for smaller versions of the same problem too: text inside tabs or "read more" sections that only loads when clicked, and prices or reviews pulled in by a script after the page arrives.
Blocker 5: Is your content behind a wall?
Anything a crawler has to get past is a wall. Logins and member areas, hard paywalls, cookie banners that hide the page until someone clicks, and blocks on visitors from other countries all stop AI crawlers the same way they stop people.
Pay-per-crawl is a newer wall. With Cloudflare's version, a crawler that has not paid receives a 402 Payment Required response instead of the page. That is a fair way to charge for training data, but a search crawler that is refused will not cite you either.
Should you block AI crawlers at all?
You can block AI crawlers selectively, and that is often the right call. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for future models; blocking them does not remove you from AI search.
Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you can be cited, and on-request fetchers such as ChatGPT-User open your pages when someone asks about them. If you want to appear in AI answers, keep those two kinds allowed, in robots.txt and in your firewall.
How do you check that the fix worked?
Run the same tests again: the curl commands, View Page Source and a request to ChatGPT. Give it a day or two, then look in your logs for visits from the search crawlers by name.
Our free checker runs the robots.txt, firewall and JavaScript tests in about a minute, and the step-by-step audit guide covers everything that comes after access.