Search
Find a page.
Search runs in your browser against the site index. The query is not sent to a server, logged, or used for anything else.
Learn
A short, ordered course on how the web works for crawlers: HTTP, HTML and the DOM, selectors, rendering, sessions, and the boundary between crawling and scraping.
Guides
Tested, vendor-neutral how-tos for Python, JavaScript, headless browsers, pagination, retries, sitemaps, incremental crawls, storage, and scheduling.
Compare
Measured, dated, vendor-neutral comparisons of scraping libraries, browser automation stacks, parsing libraries, HTTP clients, proxy types, and hosted platforms.
Legal & ethics
What robots.txt, terms of service, the CFAA, GDPR, copyright, and AI-crawler policies actually say, with primary sources and a practical respectful-crawling checklist.
Site guides
A small set of high-demand site guides that start with official APIs and public data, state the terms plainly, and never teach bypassing a login or a paywall.
Research
Original measurements with published methodology, datasets, and code. Every number is reproducible; every study carries a visible data-as-of date.
Blog
Short, timely notes on crawler policy, library releases, and the work behind the site. Evergreen material lives in Learn, Guides, Compare, and Legal.
Tools
No signup, no server, no tracking. Test robots.txt rules, validate a sitemap, try CSS or XPath selectors, parse a user agent, inspect headers, and convert cURL.