If ChatGPT, Perplexity, Claude or Google’s AI never mention your brand, the first thing to rule out is the simplest: your site might be blocking their crawlers — often without anyone realising. Here’s how to check in seconds, the accidental causes most people miss, and how to fix each.
There are three places your site can block AI crawlers: (1) robots.txt — a Disallow that names them or a blanket block; (2) your WAF/CDN — a firewall rule or a “block AI bots” toggle that returns 403 or a challenge; and (3) your host’s bot-protection returning interstitials. Start with a free AI crawler check of your robots.txt, then confirm real crawlers aren’t being challenged at the edge. Fix the block, and remember that being allowed only matters if your content is in the static HTML.
The fastest way is to read your live robots.txt the way an AI crawler does. Run a free AI crawler check (it fetches https://yourdomain.com/robots.txt and reports each AI bot as allowed or blocked), or open the file yourself and look for “Disallow” rules under “User-agent: *” or under specific bot names like GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended.
Most AI-crawler blocks aren’t deliberate. Common culprits: a security plugin or “SEO” plugin that added AI-bot Disallow rules; a Cloudflare/CDN “Block AI Bots” or “AI Scrapers” toggle switched on by default or by a teammate; a staging “Disallow: /” that got shipped to production; or an over-broad firewall rule that treats unfamiliar user-agents as bad actors. Any of these can make you invisible to ChatGPT while your Google ranking looks fine.
robots.txt is a polite request; a WAF or CDN is a hard wall. Even with a perfect robots.txt, a firewall can return 403 or a JavaScript “Just a moment…” challenge to GPTBot and PerplexityBot, and those crawlers simply give up. Check your CDN’s bot-management settings and logs for AI user-agents receiving 403/429/challenge responses, and allow-list the legitimate AI crawlers you want to read you.
Once you’ve found the block: in robots.txt, remove the Disallow rules that hit AI bots and explicitly allow them (see our copy-paste robots.txt template for AI crawlers). In your CDN/WAF, turn off the “block AI bots” toggle and allow-list GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended. Then re-check — and give the engines a few days to a few weeks to re-crawl before judging the result.
Run a free AI crawler check — it reads your live robots.txt and reports which AI bots (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) are allowed or blocked. Also check your CDN/WAF for a “block AI bots” setting and look for 403s to AI user-agents in your logs.
Yes. A WAF or CDN (like Cloudflare) can return a 403 or a bot challenge to AI crawlers at the firewall, regardless of robots.txt. That’s a hard block robots.txt can’t override — check your bot-management settings.
Common accidental causes: a security/SEO plugin that added AI-bot Disallow rules, a CDN “block AI scrapers” toggle enabled by default, or a staging “Disallow: /” shipped to production. None of these are obvious unless you check.
Crawl-access fixes are usually reflected within days to a few weeks as the engines re-read your site. Being consistently recommended also depends on your content and citations, which take longer.
Instant robots.txt check for GPTBot, PerplexityBot, ClaudeBot & more. No signup.