Editorial policy

CrawlerBros publishes practical material about crawling and scraping. This policy describes the standard a page has to meet before it ships and what happens when it does not.

What “tested” means

A code sample is tested when we have run it, as written, against a fixture or a real endpoint that permits the request, on the date shown on the page. The page states the language and library versions used. If a sample depends on a site that has changed since the test, we update the sample or mark the limitation; we do not bump the date without changing the page.

For a tutorial, “tested” means the full path works: install, run, and produce the output shown. For a comparison, it means the criteria and measurements are stated and, where the measurement is ours, reproducible from the published method. For a legal page, it means every case, statute, and policy link resolves to the primary source we actually read.

We do not claim to have tested something we have not. A page can be useful without a test, but it cannot carry the tested stamp.

Dates and updates

Every page shows a visible last updated date. The date changes only when the page changes materially: a corrected fact, a new library version, a re-run sample, a revised conclusion. Cosmetic edits do not move it. The sitemap uses the same date, so a crawler sees the same signal a reader does.

When a library or policy changes enough to invalidate a page, we either update the page or add a prominent note at the top explaining what is stale and linking to the replacement. We do not silently leave a stale page ranking.

Sources and evidence

We prefer primary sources: RFCs, W3C specifications, statutes, court opinions, regulator guidance, official API documentation, and the vendor’s own dated pricing page. Secondary sources are used for context, not as the sole basis for a factual claim.

Statistics must be one of two things:

1. Sourced — the page links to the document that published the number, and the date of that document is visible.
2. Ours — the page states the method, publishes the dataset, and links to the code that produced it.

We do not invent statistics, studies, quotes, reviews, ratings, user counts, or case studies. If we cannot verify a number, it does not go on the page.

Code and measurements

Code is written to be read and run, not to fill space. We prefer the standard library and the official client where they are sufficient. We state the version, the platform, and the date. When a benchmark is ours, we publish the machine class, the corpus, the number of runs, and the source. When a benchmark is not ours, we say whose it is and link it.

We do not benchmark third-party services by sending them traffic they did not agree to. For hosted platforms, we use published documentation and dated prices, and we say so.

AI assistance

Drafting, outlining, and copy-editing are assisted by software, including large language models. The editorial responsibility is human: the page is reviewed, the code is run, the sources are checked, and the decision to publish is made by the editor. We do not use AI to generate fake authors, fake reviews, fake interviews, or fake data.

If a page contains a passage that was generated and not substantively edited, we do not publish it. The standard is the same whether a sentence was typed or drafted with assistance.

Conflicts of interest

There are no affiliate links, sponsored posts, paid placements, or referral fees on this site. We do not accept payment to review a tool, rank a service, or remove a criticism. If the funding model changes, the change will be disclosed here and on the affected pages before it takes effect.

Comparisons may include tools made by companies we have no relationship with. That is the point. A comparison that only includes vendors who pay is not a comparison.

Corrections

Send corrections to corrections@crawlerbros.com. Include the URL, the sentence or code, and the source that shows the correct version. We aim to acknowledge substantive corrections within five working days.

When a correction changes what a reader would conclude, we log it in the corrections log with the date and a short description. Typo fixes and broken-link replacements are made without a log entry. If a correction is disputed, we say so on the page rather than pretend the disagreement does not exist.

Legal and ethical content

Legal pages are research material, not legal advice. They cite the statute, regulation, case, or policy they discuss and state the jurisdiction. They do not tell a reader that a particular crawl is lawful; that depends on facts a website cannot know. Where the law is unsettled, we say so.

We do not publish instructions for bypassing logins or paywalls, defeating CAPTCHAs on behalf of others, credential stuffing, scraping personal data at scale, or evading a site’s access controls in breach of its terms.

Accessibility and performance

Pages target WCAG 2.2 AA. Tools are keyboard-operable, focus states are visible, colour is not the only carrier of meaning, and motion respects prefers-reduced-motion. Pages are prerendered so the content is present without JavaScript; client-side code is an enhancement, not a requirement for reading.

How to cite us

Cite the page URL and the last updated date. For a research dataset, cite the study page and the dataset version. A short citation form is:

CrawlerBros Editorial, “Page title,” CrawlerBros, last updated 2 November 2026, https://www.crawlerbros.com/….

Text is licensed CC BY 4.0; code samples are MIT. Attribution is required for the text, not for the facts themselves.