Website Content Crawler: 12 Data Fields per Record (2026)

A thousand results cost $2.00 on the free plan, delivering clean content from any public domain without navigation menus, scripts, or footers. Each record carries 12 fields, including page titles, meta descriptions, and the full body content in your choice of markdown, plain text, or cleaned HTML. This setup is built for developers who need structured data for LLM training or RAG pipelines and want to avoid the noise of standard web scraping. It is not for users who need to bypass login screens or extract contact details, as those features are not supported.

Try it before you read further. Apify's free plan includes $5.00 of usage every month with no credit card, enough for up to 2,500 results at $0.002 each before platform usage. Open Website Content Crawler on Apify and run the prefilled example.

How reliable is Website Content Crawler in production?

Across the last 30 days of public runs on the Apify platform, Website Content Crawler recorded 57 runs with the following outcomes.

Expect about 7 in a hundred runs to fail or time out based on recent performance metrics. For unattended schedules, you should implement automatic retries and alert on failed runs to ensure consistent data delivery. Since the success rate is high, small batches are typically stable, but monitoring is advised for larger crawls.

What does it cost to run Website Content Crawler?

Each result costs $0.002 on Apify's free plan, which is $2.00 per 1,000 results. Starting a run is charged separately at $0.005 per GB of Actor memory. Apify also bills the platform usage each run consumes, at the rates of your Apify plan, on top of these charges.

Worked example: collecting 10,000 results costs $20.00 in result charges before run-start fees and platform usage. With 7.0% of runs failing or timing out in the last 30 days, budget for re-running a portion of those batches rather than assuming every run completes.

The maxCrawlPages control has the largest effect on your bill because it directly limits the number of results charged. The cheapest way to test your configuration is to set maxCrawlDepth to 1 and maxCrawlPages to 5, which costs approximately $0.01 in result charges.

How do you run Website Content Crawler from the API?

The schema marks 1 of its 9 controls as required: startUrls. Nothing in the payload below is illustrative. Those are the schema's prefilled defaults for Website Content Crawler, so the request works once your token is in place.

Call the synchronous endpoint to start a run and receive dataset items in one request:

curl -X POST "https://api.apify.com/v2/acts/crawlerbros~website-content-crawler/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com"}],"crawlerType":"playwright:chromium","maxCrawlDepth":10,"maxCrawlPages":20,"maxConcurrency":5,"includeUrlGlobs":[],"excludeUrlGlobs":[],"outputFormat":"markdown"}'

The same run from Python, using the official client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run_input = {
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "crawlerType": "playwright:chromium",
  "maxCrawlDepth": 10,
  "maxCrawlPages": 20,
  "maxConcurrency": 5,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "outputFormat": "markdown"
}

run = client.actor("crawlerbros~website-content-crawler").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

And from Node.js:

import { ApifyClient } from 'apify-client'

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' })

const input = {
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "crawlerType": "playwright:chromium",
  "maxCrawlDepth": 10,
  "maxCrawlPages": 20,
  "maxConcurrency": 5,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "outputFormat": "markdown"
}

const run = await client.actor('crawlerbros~website-content-crawler').call(input)
const { items } = await client.dataset(run.defaultDatasetId).listItems()
console.log(items)

Because the call is synchronous, your client waits for the whole run. Keep it for exploration. For scheduled work, start the run without waiting and collect the dataset afterwards, so network trouble costs you a retry rather than the results.

Which Website Content Crawler inputs matter, and which can you skip?

The startUrls and includeUrlGlobs controls are the most important for defining your dataset scope. Most people should leave proxyConfiguration alone on a first run as the default settings handle most public sites without additional proxy overhead.

• startUrls (array): One or more URLs to start crawling from. The crawler will follow links from these pages.
• crawlerType (string): Select the browser engine for crawling. Chromium handles most sites well. Firefox may work better on some sites. HTTP mode is fastest but only works with static (server-rendered) pages. Default: "playwright:chromium".
• maxCrawlDepth (integer): Maximum number of links to follow from the start URL. 0 means only the start URLs will be crawled. Default: 10.
• maxCrawlPages (integer): Maximum total number of pages to crawl. The crawler stops after reaching this limit. Default: 20.
• maxConcurrency (integer): Maximum number of pages to load in parallel. Higher values are faster but use more memory. Default: 5.
• includeUrlGlobs (array): Only crawl URLs matching these glob patterns. Leave empty to crawl all URLs on the same domain. Default: [].
• excludeUrlGlobs (array): Skip URLs matching these glob patterns. Useful for excluding login pages, media files, etc. Default: [].
• outputFormat (string): Format of the extracted content. Markdown is ideal for LLMs and RAG. Text gives plain text. HTML preserves the original markup. Default: "markdown".
• proxyConfiguration (object): Optional proxy settings. Not required for most websites.

Fixed-choice controls: crawlerType accepts playwright:chromium (Headless browser (Chromium)), playwright:firefox (Headless browser (Firefox)), http (Raw HTTP (static sites only)); outputFormat accepts markdown, text (Plain text), html.

What does Website Content Crawler return?

The returned records are perfect for building searchable knowledge bases because they provide clean markdown that preserves document structure. They conspicuously do not contain images, social media handles, or buyer contact information.

• url: String - Original URL that was requested
• loadedUrl: String - Final URL after any redirects
• title: String - Page title
• description: String - Meta description of the page
• languageCode: String - Language code from the HTML lang attribute
• text: String - Clean plain text extracted from the page
• markdown: String - Page content converted to Markdown
• html: String - Cleaned HTML content (when output format is HTML)
• depth: Integer - Crawl depth (0 = start URL)
• httpStatusCode: Integer - HTTP response status code
• loadedTime: String - ISO 8601 timestamp when the page was loaded
• referrerUrl: String - URL of the page that linked to this one

These are the documented fields. Optional ones can be empty on a given record, so measure how often each field your deliverable depends on is populated across a real sample before automating the handoff.

How do you build the workflow end to end?

Open Website Content Crawler and work through these in order. Each step ends with something to check, so a bad configuration surfaces on a small run rather than a scheduled one.

1. Input a domain URL into the startUrls array to define the crawl scope.
2. Set the outputFormat to markdown if you intend to feed the data into an LLM or RAG pipeline.
3. Adjust maxCrawlPages to a small number like 5 for an initial test to verify the content extraction quality.
4. Select playwright:chromium as the crawlerType to ensure dynamic JavaScript content is fully rendered.
5. Specify includeUrlGlobs if you only want to extract content from specific directories, such as blog posts or documentation subfolders.
6. Review the resulting dataset to confirm the text field contains the clean, boilerplate-free content you expect.
7. Scale up the maxCrawlPages and maxConcurrency once you have verified the filtering and extraction logic is correct.

How do you apply it? Three worked playbooks

These are Website Content Crawler's own documented use cases, each worked through as an operating pattern rather than a description.

Use case 1: LLM Training Data

Outcome: Crawl documentation sites, blogs, or knowledge bases to build training datasets

Configure: Set startUrls to a documentation root, maxCrawlDepth to 10, and outputFormat to markdown.

Working method: Start with a low maxCrawlPages to inspect how the extraction handles specific documentation components like code blocks. Once verified, remove the page limit and use includeUrlGlobs to target only the relevant technical sections. Check the languageCode field to ensure you are only collecting data in your target language.

Deliverable: A dataset of markdown files structured for fine-tuning or training large language models.

Stop condition: The crawler begins following links to irrelevant legal or footer pages not excluded by globs.

Use case 2: Competitive Analysis

Outcome: Extract and compare content across competitor websites

Configure: Add multiple competitor homepages to startUrls and set maxCrawlDepth to 2.

Working method: Execute the crawl using playwright:chromium to capture dynamic landing page text. Use the description and title fields to quickly aggregate how competitors are positioning their products in search metadata. Compare the text output across different loadedUrl entries to identify unique selling points.

Deliverable: A comparative spreadsheet containing clean text, meta descriptions, and page titles from multiple competitor domains.

Stop condition: The result count exceeds the expected number of unique landing pages for the target competitors.

Use case 3: RAG Pipelines

Outcome: Extract and index website content for retrieval-augmented generation

Configure: Set startUrls to the knowledge base URL and outputFormat to markdown with a maxConcurrency of 5.

Working method: Configure excludeUrlGlobs to skip login or signup pages that do not contain useful context for a RAG system. Run the Actor and pipe the resulting markdown and url fields directly into your vector database. Use the loadedTime timestamp to manage incremental updates and avoid re-indexing old content.

Deliverable: A structured dataset ready for vector embedding and retrieval in an AI-driven search interface.

Stop condition: The extracted markdown contains navigation menus or footer text that would pollute vector search results.

What breaks, and how do you design around it?

• Over the last 30 days, 7.0% of public runs failed and 0.0% timed out. Build retries and alerting around those rates rather than assuming every run completes.

When you hit the 100,000 page limit, split your crawl into multiple runs using different includeUrlGlobs patterns for each sub-directory. If a site fails to render in Chromium, switch the crawlerType to Firefox or use the faster HTTP mode for static pages.

When should you not use Website Content Crawler?

Do not use this Actor if you specifically need to extract emails or social media links for lead generation; use Website Contact Finder instead as it is optimized for that specific metadata. If your goal is only to extract image assets without the surrounding text, Website Image Scraper is a more efficient choice. For scenarios where you only need to process a single known URL into markdown rather than crawling an entire domain, Markdownify MCP Server provides a narrower, more focused toolset. Finally, avoid this Actor for password-protected documentation or private internal wikis, as it cannot authenticate or bypass login walls.

What should you check before trusting the output?

• Verify that the text field is not empty for pages where content extraction is expected.
• Check that httpStatusCode is 200 for your primary target pages to ensure no blocks are occurring.
• Monitor the markdown field for structural elements like headers and lists to ensure formatting is preserved for AI consumption.
• Ensure the depth field matches your expectations relative to the startUrls to avoid crawling too deep into a site.
• Watch for a high number of skipped pages in the log which may indicate the need for a different crawlerType like Firefox or Chromium.

None of this proves a record is correct. It gives a scheduled Website Content Crawler run defined points where it should stop instead of quietly passing bad data downstream.

Frequently asked questions

What is the expected reliability for large-scale crawls?

The Actor has a success rate of 89.5%, meaning about 7 in a hundred runs may fail or time out. For large crawls, it is safer to run multiple smaller batches using the maxCrawlPages control to minimize the impact of a single failed run on your total data collection.

How much will it cost to crawl 5,000 pages?

On the free plan, 5,000 results will cost $10.00 in result charges ($2.00 per 1,000 results). Note that Apify also bills for platform usage on top of these result charges. You can use the free trial's $5.00 monthly credit to cover the first 2,500 results.

Does this crawler work on React or Vue applications?

Yes. By using the default playwright:chromium crawlerType, the Actor renders the page in a headless browser. This ensures that any content generated by JavaScript frameworks is fully loaded before the extraction process begins.

Can I target only specific sections of a website?

Yes, you should use the includeUrlGlobs input to define patterns like https://example.com/blog/**. This restricts the crawler to only those paths, preventing it from wasting your budget on irrelevant pages like terms of service or contact forms.

Which format is better for training a language model?

Markdown is generally superior for LLM training because it retains the semantic structure of the page, such as headers and lists, without the overhead of HTML tags. This helps the model understand the hierarchy and context of the extracted text.

Where to go next

When you are ready to run it, open Website Content Crawler on Apify; the free plan covers up to 2,500 results a month.

Start with the Website Content Crawler Actor page for the current input schema, pricing tier, and run history.

Readers running Website Content Crawler commonly pair it with:

• RAG Web Browser: Search the web or fetch direct URLs and return clean markdown for LLM/RAG pipelines.
• Markdownify MCP Server: Convert any webpage to clean, formatted Markdown perfect for AI consumption.
• Website Contact Finder: Crawl any website and extract emails, phone numbers, and social media profiles.

Related guides:

• RAG Web Browser: 3 Practical Use Cases for Clean Markdown Extraction
• Website Image Scraper: 13 Data Fields, Up to 2,500 Free Results/Month
• LinkedIn Top Content Scraper Playbooks: 3 Ways to Build Workflows

Resources

• Actor documentation, input schema, and pricing: verified against the published Actor on 2026-09-24.

• Actor last updated by its maintainers on 2026-03-14.

• Run outcome figures cover the 30 day public window ending 2026-09-24.

• Website Content Crawler on Apify