Selenium web scraping in Python: waits, pagination, and browser cleanup

Selenium drives a real browser through the WebDriver protocol. It is the broadest browser-automation stack: many languages, many browsers, and a mature grid. This guide uses Python and headless Chromium to extract a paginated list, with explicit waits and a guaranteed driver shutdown.

Use Selenium when the page needs a real browser. If the data is in the HTML or a JSON endpoint, use an HTTP client and a parser instead.

Install

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install selenium

Tested with Python 3.14.2 and Selenium 4.52.0 on 2 November 2026. Selenium Manager resolves a compatible driver for the browser you launch. In a normal desktop install, webdriver.Chrome() finds Chrome and the driver. In the test, we pointed Selenium at a local Chromium build:

options.binary_location = "/path/to/chrome"

Pin the browser and driver versions in CI; “latest” on both sides is a moving target.

Launch and wait

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")

driver = webdriver.Chrome(options=options)
try:
    driver.get("http://127.0.0.1:8765/items?page=1")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "ul.results li.item"))
    )
    print(driver.title)
    cards = driver.find_elements(By.CSS_SELECTOR, "ul.results li.item")
    print(len(cards), cards[0].get_attribute("data-sku"))
finally:
    driver.quit()

Tested on 2 November 2026:

Items page 1
2 A-1

The try/finally is not decoration. A failed run without quit() leaves a browser process behind, and the next run starts with less memory.

Extract and paginate

from selenium.webdriver.support.ui import WebDriverWait

records = []
driver.get("http://127.0.0.1:8765/items?page=1")
while True:
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "ul.results li.item"))
    )
    for card in driver.find_elements(By.CSS_SELECTOR, "ul.results li.item"):
        records.append({
            "sku": card.get_attribute("data-sku"),
            "name": card.find_element(By.CSS_SELECTOR, "h3").text,
            "price": card.find_element(By.CSS_SELECTOR, ".price").text,
        })

    next_links = driver.find_elements(By.CSS_SELECTOR, 'a[rel="next"]')
    if not next_links:
        break

    previous_url = driver.current_url
    next_links[0].click()
    WebDriverWait(driver, 10).until(lambda current: current.current_url != previous_url)

print(len(records))
for record in records:
    print(record["sku"], record["name"], record["price"])

Tested against the fixture:

5
A-1 Mechanical keyboard $89.00
B-2 USB-C hub $39.50
C-3 Laptop stand $54.00
D-4 Noise-cancelling headphones $199.00
E-5 Webcam $69.00

The wait after clicking is on the URL change, not on a fixed sleep. A fixed sleep that is long enough is slow; one that is short is flaky. Wait for a fact.

Waits that work

Use WebDriverWait(driver, 10) and a specific condition. A bare find_element immediately after click() races the page.

Cleanup and resource limits

• Always driver.quit() in finally.
• One browser per worker; create a new tab/context for a new job.
• Set --disable-gpu, --no-sandbox only where appropriate (containers), and a fixed window size.
• Block images and fonts only if the data does not need them; Chrome options for that are version-dependent.
• Cap concurrency by memory. A Chromium instance can use hundreds of MB.
• Set page-load and script timeouts:

driver.set_page_load_timeout(30)
driver.set_script_timeout(30)

A browser is a rendering tool, not permission. Do not use it to pass a login, paywall, CAPTCHA, or ban. If the site says no, stop.

Selenium vs Playwright vs Puppeteer

• Selenium is the broadest: many languages, real installed browsers, Selenium Grid.
• Playwright has stronger auto-waiting locators, multi-browser downloads, and tracing.
• Puppeteer is Chromium-first and Node-only, with direct CDP access.

If you are starting fresh in Python and do not need Grid, Playwright is usually the smoother API. If you need Ruby, Java, C#, or a real installed browser/version, Selenium is the safer fit. The browser comparison covers the trade-offs.

Sources

• Selenium documentation
• W3C WebDriver specification
• Selenium Manager