News data for crawlers: RSS, Atom, and official APIs first

A news feed is a dated list of items with titles, links, and summaries. RSS and Atom are the standard formats; most publishers also offer a sitemap or an API. For a crawler, a feed is cheaper and more stable than parsing the homepage, and it gives you a timestamp to reason about.

Use the feed or API first. Crawl the article page only when you need the full text, the publisher permits it, and the feed does not carry enough.

Parse an Atom or RSS feed

import feedparser

feed = feedparser.parse(open("fixture-feed.xml", encoding="utf-8").read())
print(feed.feed.title, len(feed.entries))
for entry in feed.entries:
    print(
        entry.title,
        entry.link,
        entry.get("published") or entry.get("updated"),
    )

Tested on 2 November 2026 with feedparser 6.0.14 against a local Atom fixture:

Example News 2
First story https://example.com/first 2026-11-02T07:00:00Z
Second story https://example.com/second 2026-11-02T06:00:00Z

An Atom entry often has updated and no published; an RSS item often has pubDate. Read both, and normalise to UTC. A missing date is a data-quality problem, not a reason to invent one.

What a feed gives you

A feed is not a full-text source. It may contain only a summary or a truncated first paragraph. Do not call a summary “the article”.

Official APIs to check first

• Publisher APIs — some news organisations expose a licensed content API.
• GDELT DOC 2.0 — a public index of global news coverage with search and time windows.
• NewsAPI — a commercial aggregator with a developer tier; check its terms and attribution rules.
• Government/open-data APIs — for official releases, use the source rather than a news rewrite.

Read the terms for storage, display, attribution, and redistribution before you build a product on the data.

Deduplicate and track provenance

News is republished, syndicated, and updated. Deduplicate by:

• canonical URL (after redirects);
• a stable item ID from the feed;
• a content hash of the title + body;
• a near-duplicate check for syndicated copies.

Store the source URL, the feed URL, the fetch time, and the published time. A story without a source and a timestamp is not usable evidence.

Paywalls, robots, and copyright

• Paywalls: do not bypass them. A feed summary is not a licence to fetch the full article behind a paywall.
• robots.txt: a feed URL is still a request; respect the host’s rules.
• Copyright: headlines and short summaries may be used under fair dealing/fair use in some contexts; republishing the article is a different act. Get a licence for full text.
• Personal data: a news article can contain personal data; minimise and set retention.
• AI training: check the publisher’s AI-crawler policy and terms; a public feed is not a training licence.

Refresh cadence

Faster is not free. Use conditional requests where the feed supports them, and stop when the feed’s updated timestamp has not changed.

A safe pipeline

1. Fetch the feed with a descriptive user agent.
2. Parse and normalise dates to UTC.
3. Deduplicate by canonical URL and item ID.
4. Store the summary and metadata; fetch the article only if permitted and needed.
5. Record provenance and retention.
6. Re-check the feed on a schedule; do not re-crawl the whole site.

Sources

• RFC 4287 — Atom
• RSS 2.0 specification
• feedparser documentation
• GDELT DOC 2.0 API
• NewsAPI documentation