Skip to content

Guides

Scheduling and monitoring a crawler: cron, locks, metrics, and alerts

Run crawls on a schedule without overlapping jobs, duplicate writes, or silent failures. Covers cron and systemd timers, locks, checkpoints, metrics, and alert thresholds.

Last updated 3 min readBy CrawlerBros Editorial

Scheduling decides when a crawl runs; monitoring decides whether it worked. A crawler that runs on time but silently writes zero records is worse than one that fails loudly, because nobody notices until a report is wrong.

Choose the scheduler

Scheduler Good for Watch out for
cron Simple recurring jobs on one machine No overlap protection, limited logging, timezone confusion
systemd timer Linux services, journal logs, dependencies More setup; still no distributed locking
Application queue Multiple workers, retries, priorities More infrastructure
Managed scheduler Cloud jobs, audit trail Vendor lock-in, cost

For a single VM, a systemd timer is usually better than cron: it logs to the journal, can run in a specific working directory, and can be inspected with systemctl status. Cron is fine for a small job if you add a lock.

Cron expressions

Cron fields, in order:

Field Allowed values
Minute 0–59
Hour 0–23
Day of month 1–31
Month 1–12
Day of week 0–6 (Sunday = 0)

A five-field expression is minute hour day-of-month month day-of-week, followed by the command. For example, */15 * * * * runs every 15 minutes.

Expression Meaning
0 * * * * Every hour at minute 0
*/15 * * * * Every 15 minutes
30 2 * * * Daily at 02:30
0 9 * * 1 Mondays at 09:00
0 6 1 * * First day of the month at 06:00

Cron uses the system timezone unless CRON_TZ is set. A job that must run at 02:00 UTC should set the timezone explicitly, or the server’s locale will move it twice a year.

Prevent overlapping runs

A long crawl can still be running when the next schedule fires. Two runs writing to the same table can duplicate records or fight over the frontier.

Use a lock:

#!/usr/bin/env bash
set -euo pipefail

LOCK=/tmp/crawlerbros-crawl.lock
exec 9>"$LOCK"
if ! flock -n 9; then
  echo "previous run still active; exiting" >&2
  exit 0
fi

exec /opt/crawler/bin/run-crawl --config /etc/crawler/config.toml

flock -n fails immediately instead of waiting. Exiting 0 is deliberate: an overlapping run is not an error, it is a skipped run. Log it and alert if it happens repeatedly.

Make runs idempotent

A scheduled run should be safe to repeat. That means:

  • stable record keys and UPSERT/ON CONFLICT;
  • a checkpoint for the frontier;
  • a run ID on every record;
  • a retention rule for old runs;
  • no “increment a counter twice” side effects.

If a run crashes and the scheduler retries it, the second run should produce the same result, not a duplicate.

Monitor the things that fail silently

Metric Why it matters
Pages fetched A sudden drop means a block, a selector change, or a broken seed
Records written The most direct measure of usefulness
Records per page A drop is usually a layout change
HTTP status counts 403/429 spikes mean the site is pushing back
Retry count Rising retries precede failures
Run duration A slow crawl may overlap the next schedule
Last successful run The simplest freshness alarm
Database size Unbounded growth is a cost and retention problem

Alert on trends, not only on crashes. “Records per page fell 60% for two consecutive runs” is a better alert than “the process exited 1”.

A minimal run report

Write one JSON line per run:

{"run_id":"2026-11-02T02:00:00Z","started":"2026-11-02T02:00:01Z","finished":"2026-11-02T02:14:33Z","pages":1840,"records":22110,"status_2xx":1795,"status_4xx":38,"status_5xx":7,"retries":21,"errors":2}

Keep the reports for 90 days. They answer “was yesterday normal?” without reading application logs.

Scheduling cadence

Match the cadence to the change rate and the site’s tolerance:

  • News/feeds: every 15–60 minutes.
  • Prices: every 6–24 hours.
  • Documentation: weekly.
  • Reference pages: monthly.

A faster schedule is not free. It increases load on the target, your bandwidth, and the chance of overlapping runs. Measure the change rate before increasing frequency.

Failure handling

  • Transient failure: retry with backoff inside the run.
  • Repeated failure: stop the host and alert.
  • Schema change: quarantine the records and alert; do not write nulls.
  • Auth expiry: fail the run with a clear message; do not retry a 401 forever.
  • Disk full: stop before corrupting the database; alert.

A run that cannot write should not keep fetching. Check the destination before the crawl starts.

Sources

Sources checked