CSS selectors and XPath for scraping, with the pitfalls that break them

CSS selectors match elements by tag, class, id, attribute, and structural relationship. XPath selects nodes with a path language that adds axes, predicates, and functions. CSS is shorter for common cases; XPath is better when the position or text of a nearby element determines the target.

Both are safer than regex for HTML because they operate on the parsed tree. Neither is magic: a selector is only as stable as the attributes it depends on.

CSS selector patterns worth knowing

Use descendant selectors for containers you know, and attribute selectors for stable hooks. Avoid :nth-child for anything whose order can change, and avoid long chains of generated classes.

XPath patterns worth knowing

XPath’s contains(@class, "card") is not the same as CSS .card: it also matches cardigan. Prefer contains(concat(" ", normalize-space(@class), " "), " card ") when class membership must be exact.

The same extraction in both languages

Fixture:

<ul class="results">
  <li class="item" data-sku="A-1">
    <h3>Mechanical keyboard</h3>
    <span class="price">$89.00</span>
    <span class="stock">In stock</span>
  </li>
  <li class="item" data-sku="B-2">
    <h3>USB-C hub</h3>
    <span class="price">$39.50</span>
    <span class="stock">Out of stock</span>
  </li>
</ul>

CSS with BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("fixture.html", encoding="utf-8").read(), "html.parser")
for item in soup.select("ul.results li.item"):
    print(item["data-sku"], item.select_one("h3").get_text(strip=True), item.select_one(".price").get_text(strip=True))

XPath with lxml:

from lxml import html

doc = html.parse("fixture.html")
for item in doc.xpath("//ul[contains(@class, 'results')]/li[contains(@class, 'item')]"):
    sku = item.get("data-sku")
    name = item.xpath(".//h3/text()")[0].strip()
    price = item.xpath(".//span[contains(@class, 'price')]/text()")[0].strip()
    print(sku, name, price)

Tested on 2 November 2026. Both produce:

A-1 Mechanical keyboard $89.00
B-2 USB-C hub $39.50

The XPath version is longer for this case. It becomes shorter when the page does not give you a stable class or attribute, and you need to select “the cell after the header that says Price”.

Selecting by text without breaking

A common task: find the label “Price” and read the value next to it. The HTML may be a definition list, a table, or a pair of divs. Two robust approaches:

CSS, then walk. Select the element whose text is the label, then take its parent and find the value element.

from bs4 import BeautifulSoup

soup = BeautifulSoup(open("fixture.html", encoding="utf-8").read(), "html.parser")
label = soup.find(string=lambda text: text and text.strip() == "Price")
value = label.find_parent().select_one(".value")
print(value.get_text(strip=True))

XPath, then follow. Select the label node and use an axis to reach the value.

from lxml import html

doc = html.parse("fixture.html")
values = doc.xpath("//dt[normalize-space()='Price']/following-sibling::dd[1]/text()")
print(values[0].strip())

Neither is universally better. The CSS version is easier to read; the XPath version expresses the relationship directly. Use whichever matches the structure you actually have.

The pitfalls

Class churn. A class generated by a CSS-in-JS tool can change on every build. Prefer semantic attributes, data-* hooks, itemprop, or the relationship to stable text.

Order assumptions. nth-child(3) breaks when a banner is inserted. Prefer a unique attribute or a text match.

Hidden duplicates. Responsive pages often include a mobile and a desktop copy. Your selector may hit the hidden one. Select all matches and choose the one with a visible/expected attribute, or scope to the container that is actually rendered.

Over-specific chains. div > div > div > article > div > h2 is a description of today’s markup, not a stable contract. Anchor on the nearest stable container and select within it.

Attribute values with spaces. [class="card"] matches only the exact attribute value. Use [class~="card"] or contains(concat(" ", normalize-space(@class), " "), " card ").

Case and namespaces. HTML is case-insensitive for tag names in HTML mode; XML is not. XPath on XHTML may need a namespace map. If //div returns nothing on an XHTML document, check the namespace before rewriting the selector.

Text nodes versus elements. //h2/text() returns text nodes, not elements. If the heading contains <span>, the text may be split. Use string(.) or .get_text() and normalise.

How to build a selector you can trust

1. Find the smallest stable container: an article, a li, a tr, a div with a meaningful data-* attribute.
2. Select within it, not from the document root.
3. Prefer attributes that describe meaning over attributes that describe appearance.
4. Test on at least two pages of the same type, not just the one you opened.
5. Add a check: if the selector matches zero or more than the expected number of records, fail loudly rather than write nulls.
6. Store the fixture and the expected output in your test suite.

The selector tester runs CSS and XPath against pasted HTML in your browser, which is useful for quick experiments without a Python session. The Python requests + BeautifulSoup guide puts these selectors into a complete crawler.