If ChatGPT, Perplexity, Claude or Google’s AI can’t crawl your site, they can’t cite or recommend you. The most common reason is a robots.txt that blocks their crawlers — often by accident. Here’s the exact robots.txt to allow every major AI crawler, what each user-agent does, and how to check it’s working.
Add these blocks to the robots.txt at the root of your domain (e.g. https://yourdomain.com/robots.txt). Each block explicitly allows one AI crawler; the final block keeps the site open to everyone else. Replace the sitemap URL with your own.
# Allow the major AI crawlers to read your site
# ChatGPT (OpenAI)
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
# Claude (Anthropic)
User-agent: ClaudeBot
Allow: /
# Google AI (Gemini & AI Overviews)
User-agent: Google-Extended
Allow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThese are the crawlers worth allowing in 2026, and why:
robots.txt is allow-by-default: a crawler may fetch any URL that isn’t explicitly disallowed for its user-agent. So the real question is usually whether you are accidentally blocking AI bots. Two things to check: a “Disallow: /” under “User-agent: *” (which blocks everyone, including AI), and any bot-specific “Disallow” blocks that name GPTBot, PerplexityBot and the rest. The template above makes the intent explicit and safe.
After you publish robots.txt, confirm it: open https://yourdomain.com/robots.txt in a browser to see the live file, and run a free AI crawler check that reads it and reports which AI bots are allowed or blocked. Remember that robots.txt only controls crawl access — your content still has to exist in the static HTML (not only after JavaScript) for AI crawlers to actually read it.
For most sites that want AI visibility: yes. Being readable is the price of being recommended. The exceptions are content you don’t want used for training (some publishers block GPTBot but still allow OAI-SearchBot so they appear in answers without feeding training), and paywalled or private areas you should disallow for all bots. Decide per-bot — GPTBot (training) and OAI-SearchBot (live answers) can be treated differently.
Add an “Allow: /” block for each AI user-agent (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended) in the robots.txt at your domain root, and make sure no “Disallow: /” rule blocks them. Copy the template above and replace the sitemap URL.
The main ones in 2026 are GPTBot and OAI-SearchBot (ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Claude) and Google-Extended (Gemini & AI Overviews). Allow all of them to be eligible for recommendations across the major AI engines.
GPTBot is the training crawler; OAI-SearchBot is the live-answer crawler. If you want to appear in ChatGPT’s answers but not feed training, you can allow OAI-SearchBot and disallow GPTBot specifically.
A separate control token: allowing it lets Google use your content for Gemini and AI Overviews, while blocking it does not affect your normal Google Search ranking. They are independent.
Open your live robots.txt and look for “Disallow” rules under “User-agent: *” or under specific AI bot names — or run a free AI crawler checker that reads your robots.txt and lists exactly which AI bots are allowed or blocked.
Free instant robots.txt check for GPTBot, PerplexityBot, ClaudeBot & more — no signup.