SEO-024
robots.txt blocks AI search crawlers
Last reviewed
What it checks
Whether robots.txt disallows the homepage for the crawlers AI assistants use to find and cite pages (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User), while normal search engines are allowed.
Why it matters
ChatGPT search, Perplexity and Claude cannot read or cite the site, so it is missing from AI answers even though it ranks in normal search. Sometimes this is intended; often it was copied from a template that meant to block AI training only.
Recommended fix
If you want to appear in AI answers, remove the Disallow for these search crawlers. You can still block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) separately.
How to verify
Open /robots.txt and confirm none of the listed AI search crawlers is disallowed from /.
Example fix prompt
Please fix this production issue in my app.
Issue SEO-024: robots.txt blocks AI search crawlers
Site: https://your-app.example
First confirm the issue in the code or live config. If it is not present, say so and change nothing.
Task: Our robots.txt blocks AI search crawlers. Keep any blocks for AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) if they exist, but allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User to crawl public pages. Do not change rules for private paths such as /api or /dashboard.
Constraints: make the smallest change that fixes this, keep the current design and behaviour, never paste secret values into code or chat, and tell me which files you changed.
Done when: Open /robots.txt and confirm none of the listed AI search crawlers is disallowed from /.