Tools

Free, client-side crawling tools

No signup, no server, no tracking. Test robots.txt rules, validate a sitemap, try CSS or XPath selectors, parse a user agent, inspect headers, and convert cURL.

Small, focused, client-side tools. Nothing is uploaded to a server, nothing requires an account, and every tool page explains the underlying concept.

Last updated 2 November 2026

TOOLrobots.txt tester: check a URL against RFC 9309 rulesPaste a robots.txt file, choose a user agent, and test a URL. The tester applies group selection, wildcard matching, longest match, and Allow-wins-ties.Updated 2 November 2026 · 2 min read

TOOLSitemap validator: check XML structure, duplicates, and datesPaste a sitemap.xml or sitemap index and validate the structure: root element, loc values, duplicates, lastmod formats, entry count, and size limits.Updated 2 November 2026 · 2 min read

TOOLCSS and XPath selector tester: paste HTML, see every matchTest a CSS selector or XPath expression against pasted HTML in your browser. Shows match counts and the matched markup, with no server round trip.Updated 2 November 2026 · 2 min read

TOOLUser-agent parser: see what a server can infer from a UA stringPaste a user-agent string and see the product token, browser, engine, operating system, device class, and known bot. Runs locally; no lookup service.Updated 2 November 2026 · 2 min read

TOOLCrawl-delay calculator: turn a politeness budget into a delayConvert a target requests-per-second rate and worker count into a per-worker delay, and see the estimated crawl time. Includes common Crawl-delay values.Updated 2 November 2026 · 2 min read

TOOLHTTP header inspector: parse response headers and spot the important onesPaste raw HTTP response headers to see them parsed, with checks for caching, content type, HSTS, X-Robots-Tag, Retry-After, rate-limit headers, and cookies.Updated 2 November 2026 · 1 min read

TOOLcURL to Python and JavaScript converterPaste a cURL command and get equivalent Python requests and JavaScript fetch code. Handles headers, methods, data, basic auth, redirects, and common flags locally.Updated 2 November 2026 · 2 min read

Keep going

Learn the mechanicsThe six-lesson course that makes the guides readable.Use a toolRobots tester, selector tester, sitemap validator, and more.Check a definitionThirty terms, defined first and linked into the guides.