HTML and the DOM: how to turn a page into a tree you can query

HTML is the text a server sends; the DOM (Document Object Model) is the tree a parser builds from that text. A crawler parses HTML into a tree so it can select elements by structure, attributes, and text instead of matching raw strings.

The distinction explains a common bug. DevTools shows the DOM after the browser has parsed the HTML and run JavaScript. Your scraper sees the HTML the server sent. If the element was created by a script, it is in DevTools and not in your response.

The tree in one example

This HTML:

<article class="card" data-id="42">
  <h2>HTTP for crawlers</h2>
  <p class="summary">Status codes, headers, redirects.</p>
  <a href="/learn/http-for-crawlers">Read the lesson</a>
</article>

becomes a tree with an article element, three children (h2, p, a), attributes on the article and paragraph, and text nodes inside the headings and links. A selector such as article.card h2 walks that tree; it does not search the string.

Elements, attributes, and text

• An element is a tag and its content: <p>…</p>.
• An attribute is a name/value pair on the opening tag: class="card", href="/learn".
• A text node is the text between tags. It is not an element, and some APIs treat it differently.
• Whitespace between tags is usually preserved in the text. Normalise it before comparing or storing.

A parser may add elements the source did not write. Browsers insert html, head, and body if they are missing, and they repair some malformed nesting. lxml and html.parser make different repairs. That is why “it parses in the browser” is not a guarantee that it parses the same way in your scraper.

Parse, then select

The standard-library html.parser is pure Python and forgiving. lxml is faster and handles broken markup differently. selectolax is faster still, with a smaller API. For a first crawler, BeautifulSoup with html.parser is the easiest to debug.

from bs4 import BeautifulSoup

html = """
<article class="card" data-id="42">
  <h2>HTTP for crawlers</h2>
  <p class="summary">Status codes, headers, redirects.</p>
  <a href="/learn/http-for-crawlers">Read the lesson</a>
</article>
"""

soup = BeautifulSoup(html, "html.parser")
card = soup.select_one("article.card")
print(card["data-id"])
print(card.select_one("h2").get_text(strip=True))
print(card.select_one("a")["href"])
print(card.select_one("p.summary").get_text(strip=True))

Tested on 2 November 2026 with BeautifulSoup 4.14.3:

42
HTTP for crawlers
/learn/http-for-crawlers
Status codes, headers, redirects.

select_one returns the first match or None. select returns a list. get_text(strip=True) collapses leading and trailing whitespace; it does not collapse internal whitespace, so use " ".join(text.split()) when you need a clean single line.

The same tree with lxml

from lxml import html

doc = html.fromstring("""
<article class="card" data-id="42">
  <h2>HTTP for crawlers</h2>
  <p class="summary">Status codes, headers, redirects.</p>
  <a href="/learn/http-for-crawlers">Read the lesson</a>
</article>
""")

print(doc.xpath("//article[@class='card']/@data-id")[0])
print(doc.xpath("//article[@class='card']//h2/text()")[0].strip())
print(doc.xpath("//article[@class='card']//a/@href")[0])

Tested on 2 November 2026 with lxml 6.0.2:

42
HTTP for crawlers
/learn/http-for-crawlers

The two libraries agree on the result but not on the API. BeautifulSoup is more forgiving and easier to explore interactively; lxml is faster and gives you XPath. The CSS/XPath lesson covers the query languages in detail.

Common parsing problems

Generated class names. A class such as css-1x2y3z may change on every deploy. Prefer stable attributes: data-testid, itemprop, id, a semantic tag, or a link pattern. If the only stable thing is the text “Price”, select by text and walk to the sibling.

Tables. A table row’s meaning often lives in a header cell, not in the cell itself. Parse the headers first, build a column index, then read each row by index. Do not assume the fourth <td> is the price on every page.

Entities and encoding. &, ', and non-breaking spaces are decoded by the parser. If you see & in your output, you are probably reading raw HTML instead of parsed text.

Whitespace and line breaks. A price split across two text nodes will not match a single .text comparison. Join the descendants’ text and normalise it.

Duplicate elements. A page may have a mobile and a desktop version of the same card, both in the DOM, with CSS hiding one. Your selector may match the hidden one first. Check the surrounding container, or select all and choose by visible text/attributes.

Shadow DOM. Content inside a shadow root is not reachable with ordinary CSS selectors from the document. A browser automation tool needs to pierce the shadow root explicitly. A plain HTTP parser sees the host element and the template, not the rendered shadow tree.

HTML source versus rendered DOM

A static response contains the HTML the server sent. A rendered page contains the DOM after scripts run. To tell them apart:

1. Fetch the URL with curl or requests.
2. Search the response for the text you expected.
3. If it is absent, open DevTools → Network and find the request that returns it.
4. If the data arrives as JSON, parse the JSON. If it is built entirely in the browser, consider a headless browser.

Do not start with a browser because the page “looks dynamic”. Start with the response. The static vs JavaScript rendering lesson walks through the decision.

A parser is not a validator

An HTML parser will happily build a tree from a page that is mostly an error message. Check the HTTP status, the page title, and a known element before extracting fields. A parser that returns zero matches on a 200 response is often a soft 404 or a changed layout, not a selector bug.

Key takeaways

• HTML is text; the DOM is a tree. Your scraper sees the source unless you render it.
• Parse with a real parser, not regular expressions, unless the format is genuinely regular.
• Prefer stable attributes over generated class names.
• Normalise whitespace and decode entities before comparing text.
• Check the status and a known element before trusting an empty result.
• Use lxml when you need speed or XPath; use BeautifulSoup when you need forgiveness and readability.