If your robots.txt disallows GPTBot, PerplexityBot, ClaudeBot or Google-Extended, those engines can’t read your site — and won’t recommend it. Paste your URL for a real read of your live robots.txt and see exactly which AI crawlers are allowed or blocked. No signup.
AI engines can only recommend what they can read. A single overly-broad Disallow line in robots.txt can quietly remove you from ChatGPT, Perplexity, Claude and Gemini — one of the most common and most invisible visibility mistakes.
Some sites deliberately block AI crawlers to protect content; most want the visibility. Either way, you should know your current state. This tool simply reports what your robots.txt says today, so the decision is informed rather than accidental.
If a crawler is blocked, remove or narrow the relevant Disallow rule for that user-agent in robots.txt, then re-check. After access is fixed, make sure your content is in static HTML (not JavaScript-only) so crawlers actually see it — a free Beacon account measures that render gap across your site.
Paste your URL above. Beacon reads your live robots.txt and reports whether GPTBot, PerplexityBot, ClaudeBot and Google-Extended are allowed or disallowed.
It depends on your goals. Blocking protects content from being used for training and answers, but it also removes you from AI recommendations. Most brands seeking visibility should allow them.
Yes — it’s a real fetch of your /robots.txt at check time, parsed for each AI user-agent. Nothing is fabricated.
Allowing crawlers is step one; they also need readable HTML. If your content only appears after JavaScript runs, crawlers see an empty shell. A free Beacon account measures this render gap.
Free, real, no signup — see who can read your site.