robots.txt tester: check a URL against RFC 9309 rules

This tool tests one URL against a robots.txt file using the RFC 9309 matching rules. It runs entirely in your browser: the file you paste is not uploaded, logged, or sent to a server.

How to use it

1. Paste a robots.txt file, or keep the example.
2. Enter the URL path you want to test. A full URL is fine; the tool uses the path and query.
3. Choose a known user agent or enter a custom one.
4. Read the result: allowed or blocked, the matching rule, the Crawl-delay, and the sitemaps.

The tool selects the most specific matching user-agent group, falls back to *, matches * and $, uses the longest match, and lets Allow win a tie. These are the rules a correct crawler should implement.

What the result means

• Allowed means no matching Disallow rule is longer than a matching Allow rule. It does not mean the crawl is lawful; the terms, copyright, and privacy rules still apply.
• Blocked means a Disallow rule matched. The tool shows the rule so you can see why.
• No rule matched means the default is allow.

Common mistakes this tool catches

• A Disallow: / with no Allow blocks the whole site.
• Allow: /private/public/ does not override a longer Disallow: /private/ unless it is more specific and longer.
• Disallow: /search blocks /search and /search?q=…, but not /search-about.
• A User-agent line must be followed by rules; a stray rule before any user agent is ignored.
• Crawl-delay is not part of RFC 9309; it is shown as a hint, not enforced by the matching result.

Limits

• The tool tests the file you paste, not the live file at a URL. Fetch the live file yourself and paste it.
• It tests the path you enter, not every path.
• It does not verify that the server returns the file with the right content type or status.
• It does not enforce a rate limit; it only reports a Crawl-delay if one is present.

For the protocol, read robots.txt and RFC 9309. For AI-specific policy examples, read AI crawler policies.

Sources

• RFC 9309