CRAWLER ACCESS
Free Robots.txt Tester for Search & AI Crawlers
This robots.txt tester checks whether pasted Allow and Disallow rules permit a crawler to request a specific URL path. Compare search and training crawlers, see the matching rule and line number, and diagnose a blocked path without making a network request.
- 100% free
- No signup
- Runs in your browser
- No input uploaded
Crawler results
Run a test to compare search and training crawlers. No requests will be sent to your website.
FROM INPUT TO NEXT STEP
How to use this tool
Copy the robots.txt text for the same origin as the URL you want to test.
Enter the full page URL, including its path and any query string.
Run the test to compare crawler results and inspect the winning rule. Edit your text and test again before publishing.
Understanding your results
The tester selects the most specific matching user-agent groups, combines groups with the same specificity, and checks path rules. Longer matching paths take precedence; Allow wins a tie. Matching is case-sensitive for paths and supports * wildcards and an ending $ anchor. The results explain the rule used instead of giving an unexplained pass or fail.
PRACTICAL ANSWERS
Frequently asked questions
Can I allow AI search while blocking training crawlers?
Some providers use different agents for search and model training. For example, OAI-SearchBot serves OpenAI search and GPTBot serves training. Review each provider’s documentation and test the relevant user-agent separately.
Does a robots.txt block remove a page from Google?
A crawl restriction is not a reliable removal instruction. A URL may still be known through links. Review indexing controls separately; this tester only evaluates the supplied crawl rules.
Why does an Allow rule override a Disallow rule?
A more specific matching path takes precedence over a broader path. If matching Allow and Disallow rules have equal specificity, this tester chooses Allow.
Why do I paste the file instead of entering a website to scan?
Pasting lets the tool run entirely in your browser without a proxy, crawler service, or account. Make sure the text belongs to the origin you are testing.
Documentation & further reading
- Google: robots.txt matching rules
- OpenAI: crawler roles
- Anthropic: crawler roles
- Perplexity: crawlers
Continue with our AI search readiness guide to put these checks in context.
KEEP IMPROVING
More free website tools
READY FOR THE BIGGER PICTURE?
Turn individual checks into a website action plan.
SiteMagnetIQ audits your public pages and organizes findings across SEO, AI readiness, trust, and more.