REL-006
robots.txt returns a server error
HighDefault severity · Reliability · SEO, quality and performance
Last reviewed
What it checks
Status code of /robots.txt.
Why it matters
When robots.txt answers with a 5xx error, Google treats the whole site as off-limits and stops crawling until it recovers.
Recommended fix
Serve a valid robots.txt (or a plain 404 if you do not need one).
How to verify
Request /robots.txt and confirm 200 or 404.
Example fix prompt
Generic prompt (your report adds the evidence)
Please fix this production issue in my app.
Issue REL-006: robots.txt returns a server error
Site: https://your-app.example
First confirm the issue in the code or live config. If it is not present, say so and change nothing.
Task: Fix /robots.txt so it returns 200 with valid content (or 404 if the site has no crawl rules). Find out why it currently errors and fix that route or file.
Constraints: make the smallest change that fixes this, keep the current design and behaviour, never paste secret values into code or chat, and tell me which files you changed.
Done when: Request /robots.txt and confirm 200 or 404.