SEO-002
robots.txt blocks all search engines
HighDefault severity · SEO / Quality · SEO, quality and performance
Last reviewed
What it checks
Whether the `User-agent: *` group disallows the whole site.
Why it matters
Search engines are told not to crawl anything, so the site will not appear in results.
Recommended fix
Remove `Disallow: /` from the `*` group unless the site is meant to be private.
How to verify
Open /robots.txt and confirm `User-agent: *` no longer has `Disallow: /`.
Example fix prompt
Generic prompt (your report adds the evidence)
Please fix this production issue in my app.
Issue SEO-002: robots.txt blocks all search engines
Site: https://your-app.example
First confirm the issue in the code or live config. If it is not present, say so and change nothing.
Task: robots.txt currently blocks all crawlers with `Disallow: /`. Change it so public pages are crawlable (keep disallows only for private paths like /dashboard or /api) and add a Sitemap line. This is often left over from a staging config.
Constraints: make the smallest change that fixes this, keep the current design and behaviour, never paste secret values into code or chat, and tell me which files you changed.
Done when: Open /robots.txt and confirm `User-agent: *` no longer has `Disallow: /`.