Skip to content

Legal & ethics

robots.txt and RFC 9309: what the standard actually says

How robots.txt matching works under RFC 9309: groups, Allow and Disallow, wildcards, longest match, crawl-delay, sitemaps, and the limits of the protocol.

Last updated 4 min readBy CrawlerBros Editorial

robots.txt is a text file at the root of a site that tells crawlers which paths they may request. The Robots Exclusion Protocol is specified in RFC 9309, published in September 2022. It is a voluntary protocol: it tells a well-behaved crawler what the site wants, not what the law requires.

This page is research material, not legal advice. It explains the protocol; it does not tell you whether a particular crawl is lawful.

Where the file lives

A crawler fetches https://<host>/robots.txt before crawling that host. The file applies to the host and scheme it is served from. A robots.txt on https://example.com does not automatically apply to https://www.example.com or http://example.com; they are different origins for this purpose.

If the file is missing (404), the crawler may proceed. If it is unreachable because of a 5xx, RFC 9309 says the crawler should assume complete disallowance for a reasonable period. If it is unreachable because of a 4xx other than 404/410, the crawler may proceed but should be cautious.

Groups and matching

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines, followed by rules.

User-agent: *
Disallow: /private/
Allow: /private/public/

User-agent: GPTBot
Disallow: /

A crawler selects the group whose user-agent token matches its own most specifically. If no specific group matches, it uses the * group. The match is on the product token, not the full user-agent string; the token is case-insensitive.

Rules are path prefixes. Disallow: /private/ blocks /private/ and everything under it. Allow: /private/public/ re-allows a subtree. When both an Allow and a Disallow match, the longest match wins; if they are the same length, Allow wins.

* matches any sequence of characters, and $ anchors the end of the path. Thus Disallow: /*.pdf$ blocks PDFs anywhere on the host. A pattern is matched against the path and query string, not the full URL.

Directives

Directive Standard? Meaning
User-agent Yes Which crawler the group applies to
Allow Yes A path the crawler may fetch
Disallow Yes A path the crawler may not fetch
Sitemap Widely supported, not RFC 9309 Absolute URL of a sitemap
Crawl-delay No Requested seconds between requests
Host No Preferred host (Yandex-specific)

Crawl-delay is not part of RFC 9309. A crawler that ignores it is not violating the standard, but it may still be ignoring the site’s wishes. Treat it as a request and use your own rate limit.

What robots.txt does not do

  • It does not grant permission. A path that is not disallowed may still be prohibited by the terms of service, copyright, privacy law, or an access control.
  • It does not bind every bot. It is a protocol, not an enforcement mechanism. Malicious crawlers ignore it.
  • It does not protect a page from being linked or indexed. A disallowed URL can still appear in search results if other pages link to it, though the crawler should not fetch the content.
  • It does not replace authentication. A Disallow is not a login.

Practical parsing

A correct implementation:

  1. Fetches the file for the exact scheme and host.
  2. Groups rules by user-agent.
  3. Selects the most specific matching group, falling back to *.
  4. Matches paths with * and $, longest match, Allow wins ties.
  5. Applies the result before every request, not just at the start.
  6. Honours Sitemap as a discovery hint.

The robots.txt tester implements this matching in your browser. It is useful for checking a rule before you ship a crawler, and for explaining to a site owner why a path is blocked.

robots.txt can be relevant evidence in a dispute about whether access was authorised, but it is not the legal test by itself. Courts look at the terms of service, technical access controls, cease-and-desist letters, and the nature of the data. The CFAA guide and the terms of service guide cover that side.

If a site disallows a path you need, the defensible options are: use an official API, ask for permission, use a public dataset, or stop. Editing your user agent to match a group you are not is not a solution; it is misrepresentation.

Sources

Sources checked