Sitemaps and feeds for crawlers: discovery without guessing URLs

A sitemap is an XML file that lists URLs a site wants discovered, optionally with a last-modified date. A feed is an Atom or RSS document that lists recent items with timestamps. Both are discovery mechanisms: they tell a crawler where to start, not what the page contains.

For a crawler, a sitemap is often better than following every link. It gives you a complete URL list, a hint about what changed, and a natural checkpoint. It does not replace the page: you still fetch and parse the URL.

Sitemap basics

A minimal sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/http</loc>
    <lastmod>2026-11-01</lastmod>
  </url>
</urlset>

Rules that matter:

• <loc> must be an absolute URL.
• A sitemap may contain at most 50,000 URLs and be at most 50 MB uncompressed.
• A sitemap index points to other sitemaps; it does not contain page URLs.
• lastmod is a hint. A sitemap that says every page changed today teaches crawlers to ignore it.
• Declare the sitemap in robots.txt so crawlers can find it.

Parse a sitemap in Python

import requests
import xml.etree.ElementTree as ET

url = "http://127.0.0.1:8765/sitemap.xml"
response = requests.get(url, timeout=10)
response.raise_for_status()
root = ET.fromstring(response.content)
namespace = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
locs = [node.text for node in root.findall("sm:url/sm:loc", namespace)]
print(len(locs), locs)

Tested on 2 November 2026 against the local fixture:

2 ['http://127.0.0.1:8765/', 'http://127.0.0.1:8765/guides/http']

ElementTree requires the namespace. A parser that looks for url/loc without it returns nothing, which looks like an empty sitemap.

Handle a sitemap index

A large site splits its sitemap into an index:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemaps/guides.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemaps/blog.xml</loc></sitemap>
</sitemapindex>

Detect the root tag, then fetch each child sitemap:

def sitemap_urls(url: str, session: requests.Session) -> list[str]:
    response = session.get(url, timeout=15)
    response.raise_for_status()
    root = ET.fromstring(response.content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    if root.tag.endswith("sitemapindex"):
        urls = []
        for loc in root.findall("sm:sitemap/sm:loc", ns):
            urls.extend(sitemap_urls(loc.text, session))
        return urls
    return [node.text for node in root.findall("sm:url/sm:loc", ns)]

Add a recursion depth limit and a seen set. A malformed index that points to itself will otherwise loop forever.

Feeds

An Atom feed is useful for a blog or news site because it gives recent items with dates. The structure is namespaced:

import feedparser  # pip install feedparser

feed = feedparser.parse("https://example.com/feed.xml")
for entry in feed.entries[:10]:
    print(entry.get("published"), entry.get("link"), entry.get("title"))

feedparser is the pragmatic choice for real-world feeds, which are often malformed. If you want no dependency, parse with ElementTree and the Atom namespace, but expect to handle missing dates and relative links.

Discovery strategy

A practical order:

1. Read robots.txt for Sitemap: lines.
2. Fetch the sitemap index, then the child sitemaps.
3. Normalise and deduplicate URLs.
4. Compare lastmod with the last crawl; fetch new/changed URLs first.
5. Fall back to link discovery from the homepage for pages missing from the sitemap.
6. Store the sitemap URL, fetch time, and per-URL lastmod so the next run can skip unchanged pages.

Do not trust a sitemap blindly. It can list redirects, 404s, noindex pages, or URLs on another host. Validate the status and canonical before storing a record.

Feeds vs sitemaps

Use more than one. A sitemap plus a feed plus a homepage crawl covers most public content sites.

Build your own sitemap honestly

If you publish a site, generate the sitemap from the same data that renders the pages. Set lastmod from the page’s real modification date. Split by section when the list grows. Declare it in robots.txt. Do not include noindex, redirect, or 404 URLs.

The sitemap validator checks structure, duplicates, and date formats in your browser. It cannot check that a URL returns 200; crawl the URLs separately.

Sources

• sitemaps.org protocol
• RFC 4287 — Atom
• RFC 9309