Skip to content

Site guides

Job data for crawlers: use official ATS APIs before scraping job boards

Collect public job listings from Greenhouse, Lever, Ashby, and Workable APIs, respect board terms, normalise fields, and avoid the login walls and duplicates of job boards.

Last updated 3 min readBy CrawlerBros Editorial

Job data is often available from the applicant-tracking system (ATS) that hosts the company’s careers page. Greenhouse, Lever, Ashby, and Workable publish board APIs for public listings. Those endpoints are more stable than a job board’s search page and avoid the login walls and duplicate listings that make boards hard to crawl.

This page is research material, not legal advice. Each ATS and each employer has its own terms.

Start with the company’s ATS

A careers page usually embeds or proxies one of these:

ATS Public board shape (illustrative)
Greenhouse https://boards-api.greenhouse.io/v1/boards/<board>/jobs
Lever https://api.lever.co/v0/postings/<company>?mode=json
Ashby https://api.ashbyhq.com/posting-api/job-board/<board>
Workable A per-account careers API; check the current docs

Read the current documentation before coding; paths and auth change. Some boards require a token, some are public, and some expose only a subset of fields.

What to collect

  • stable job ID;
  • title;
  • company;
  • location(s) and remote policy;
  • department/team;
  • employment type;
  • posted/updated date;
  • canonical application URL;
  • description (if the board provides it and the terms allow storage);
  • salary range, when published.

A job without a canonical URL and a posted date is not useful for a tracker. Do not infer a salary or a remote policy from the description unless the text states it.

Deduplicate across boards

The same job can appear on the company ATS, a job board, and an aggregator. Deduplicate by:

  • ATS job ID + company;
  • canonical application URL;
  • title + company + location + description hash.

Prefer the company ATS as the source of truth. Aggregators add stale copies and affiliate redirects.

Respect the terms

  • Read the ATS’s API terms and the employer’s careers-page terms.
  • Do not bypass a login, an application form, or a CAPTCHA.
  • Do not collect applicant or recruiter personal data.
  • Do not republish full descriptions if the terms do not allow it; link to the application.
  • Honour rate limits and caching rules.
  • Remove listings when they close; a stale job is a bad user experience and a legal risk.

A safe pipeline

company careers page -> identify ATS -> board API
  -> validate fields
  -> normalise location/remote/date
  -> deduplicate by ATS ID + company
  -> store source URL + fetched time
  -> expire closed listings

Run the crawl on a schedule matched to the board’s update rate. Daily is usually enough; hourly is usually unnecessary.

When the ATS has no public API

Use the public careers page only if the terms permit it and the data is public. Do not scrape LinkedIn, Indeed, or Glassdoor by evading their login walls or anti-bot controls. If you need those datasets at scale, use their official APIs or a licensed data provider.

  • Contract: the ATS API terms and the employer’s terms govern the use.
  • Personal data: job descriptions can contain contact names and emails; minimise and set retention.
  • Copyright: descriptions are employer content; do not republish beyond the licence.
  • Employment law: some jurisdictions regulate job advertising and salary transparency; do not alter the listing’s meaning.

Sources

Sources checked