SEO · 17th of 19 by usage
Common Crawl Scraper
Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.
- Price
- $0.005 per result
- Runs to date
- 102
- Teams using it
- 2
- Last updated
- 2026-07-02
Run it from the API
POST · run and wait for the datasetcURL
curl -X POST "https://api.apify.com/v2/acts/crawlerbros~common-crawl-scraper/run-sync-get-dataset-items" \
-H "Content-Type: application/json" \
-d '{ "maxItems": 50 }' \
-G --data-urlencode "token=$APIFY_TOKEN"Billed $0.005 per result. The call returns the dataset directly, so there is nothing to poll for a run of this size.
Common Crawl Scraper vs the other SEO scrapers
All 19 are ours and all are maintained, so the choice is about fit rather than quality. 13 of them cost less per result.
| Scraper | Price | Runs | Rating |
|---|---|---|---|
| Common Crawl Scraperthis page | $0.005 | 102 | not yet rated |
| Website Screenshot Generator | $0.002 | 7,428 | not yet rated |
| Similarweb Scraper | $0.002 | 1,405 | not yet rated |
| Traffic Generator | $0.002 | 1,215 | 5.0 (1) |
| SEMrush Free Website Stats Scraper | $0.002 | 1,007 | not yet rated |
| Ahrefs Free Website Stats Scraper | $0.002 | 909 | not yet rated |
| Website Content Crawler | $0.002 | 506 | not yet rated |
| Website Links Graph Generator | $0.002 | 391 | 5.0 (2) |
| URL to BibTeX Converter | $0.002 | 323 | 5.0 (2) |
| Wikipedia Article Scraper | $0.001 | 246 | not yet rated |
| Ubersuggest Keyword Scraper | $0.002 | 172 | not yet rated |
| Canva Template Scraper | $0.005 | 170 | not yet rated |
| PubMed Search Scraper | $0.002 | 162 | not yet rated |
| Sitemap Sniffer | $0.002 | 143 | not yet rated |
| Find Broken Links | $0.002 | 131 | not yet rated |