Web Scraping in Python Without Losing Your Mind

I've been scraping websites for longer than I care to admit. Started with Beautiful Soup back when Python 2 was still a thing, and honestly not much has changed. The tools got better, the websites got more hostile, but the core workflow is the same: find the data, grab the data, clean the data.

Here's what I've learned doing this stuff in production ( and in anger ) over the years.

Pick the Right Tool for the Job

Not every site needs a headless browser. I see people reaching for Playwright or Selenium for everything and it drives me crazy. If the data is in the static HTML, use requests + BeautifulSoup. It's 50x faster and won't eat your RAM.

My decision tree is dead simple:

Static HTML → requests + BeautifulSoup JavaScript-rendered → Playwright ( headless ) API endpoint exists → just hit the API directly ( seriously, check first )

90% of what I scrape, the simple stack handles it.

The Boilerplate That Actually Works

Here's my base scraper template. I use this as a starting point for everything:

import requests
from bs4 import BeautifulSoup
from time import sleep
from random import uniform

def scrape(url, retries=3):
    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
        "Accept": "text/html,application/xhtml+xml",
        "Accept-Language": "en-US,en;q=0.9",
    }
    
    for attempt in range(retries):
        try:
            resp = requests.get(url, headers=headers, timeout=15)
            resp.raise_for_status()
            return BeautifulSoup(resp.text, "html.parser")
        except requests.RequestException as e:
            print(f"Attempt {attempt+1} failed: {e}")
            sleep(uniform(2, 5))
    
    return None

# Usage
soup = scrape("https://example.com/data")
if soup:
    rows = soup.select("table.data-table tbody tr")
    for row in rows:
        print(row.select_one(".name").text.strip())

The random sleep between retries is not optional. Sites that rate-limit you will ban you if you hammer them. I learned this the hard way.

CSS Selectors Over Regex, Always

I used to parse HTML with regex. Don't do that. BeautifulSoup's CSS selector support is good enough for 99% of cases:

# Bad - regex on HTML
import re
names = re.findall(r'<span class="name">(.*?)</span>', html)

# Good - CSS selectors
names = [el.text.strip() for el in soup.select("span.name")]

# Get by ID
main_content = soup.select_one("#main-content")

# Nested selectors
prices = [p.text for p in soup.select(".product-card .price")]

# Attributes
data_ids = [el["data-id"] for el in soup.select("[data-id]")]

CSS selectors survive minor HTML changes. Regex breaks when someone adds a space to a tag. Real talk: I've refitted three scrapers this month alone because of "minor" site redesigns. Selectors held up. Regex didn't.

Handle Pagination Without Losing Your Mind

Most paginated sites follow a pattern. Here's how I handle it:

def scrape_all(base_url, max_pages=50):
    all_items = []
    
    for page in range(1, max_pages + 1):
        url = f"{base_url}?page={page}"
        soup = scrape(url)
        if not soup:
            break
            
        items = soup.select(".item-card")
        if not items:  # no more pages
            break
            
        for item in items:
            all_items.append({
                "title": item.select_one(".title").text.strip(),
                "price": item.select_one(".price").text.strip(),
            })
        
        sleep(uniform(1, 3))  # be nice
    
    return all_items

results = scrape_all("https://example.com/listings")
print(f"Scraped {len(results)} items")

The max_pages cap exists because I once wrote a scraper without one and it ran for 14 hours on a site with 40,000 pages. Set limits on everything.

When You Need Playwright

Some sites load everything via JavaScript. The HTML you get with requests is an empty shell. That's when Playwright comes in:

from playwright.sync_api import sync_playwright

def scrape_js(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        
        # Wait for the data to actually render
        page.wait_for_selector(".data-loaded", timeout=10000)
        
        html = page.content()
        browser.close()
    
    return BeautifulSoup(html, "html.parser")

soup = scrape_js("https://spa-example.com/dashboard")
# Now parse normally with BeautifulSoup

Playwright is heavier and slower. Only use it when you have to. And always wait_for_selector something specific — networkidle isn't always enough. I've been burned by that more than once.

Save Incrementally or Lose Everything

This one cost me a full day of scraping once. If your script crashes at page 487 out of 500, you don't want to start over. Write results as you go:

import json, csv

def scrape_with_checkpoint(base_url, output_file="results.json"):
    # Load existing results if resuming
    try:
        with open(output_file) as f:
            results = json.load(f)
        start_page = len(results) // 20 + 1  # adjust per page count
        print(f"Resuming from page {start_page}")
    except FileNotFoundError:
        results = []
        start_page = 1
    
    for page in range(start_page, 51):
        soup = scrape(f"{base_url}?page={page}")
        if not soup:
            break
        
        for item in soup.select(".item"):
            results.append({
                "name": item.select_one(".name").text.strip(),
                "url": item.select_one("a")["href"],
            })
        
        # Save after every page
        with open(output_file, "w") as f:
            json.dump(results, f, indent=2)
        
        sleep(uniform(1, 3))
    
    return results

JSON for checkpoints, CSV for final output if you need it in a spreadsheet. Both work. The point is: never hold everything in memory and hope the script finishes.

Respect the Sites You Scrape

A few things I actually do:

Check robots.txt first. If they say no, think hard about whether you need that data.Add delays between requests. sleep(uniform(1, 3)) is the minimum. For smaller sites, go higher.Set a real User-Agent. Not "Python-urllib/3.x". That gets you blocked in 5 seconds.Cache responses locally during development. Don't hit the same page 50 times while debugging your selectors.Don't scrape personal data. Seriously. GDPR and common sense.

Caching During Development

This saves me so much time. Cache responses to disk and only fetch new ones:

import hashlib, os

def cached_scrape(url, cache_dir=".cache"):
    os.makedirs(cache_dir, exist_ok=True)
    cache_key = hashlib.md5(url.encode()).hexdigest()
    cache_file = f"{cache_dir}/{cache_key}.html"
    
    if os.path.exists(cache_file):
        with open(cache_file) as f:
            return BeautifulSoup(f.read(), "html.parser")
    
    soup = scrape(url)
    if soup:
        with open(cache_file, "w") as f:
            f.write(str(soup))
    
    return soup

I test my selectors against cached HTML and only hit the real site once I'm confident they work. The site owner doesn't get hammered, I don't get blocked. Win-win.

Conclusion

Web scraping in Python doesn't need to be complicated. Start simple with requests + BeautifulSoup, reach for Playwright only when you need it, cache everything during development, and save incrementally in production. That's it. The rest is just writing good selectors and not being a jerk to the sites you're scraping.

The full template is on my GitHub ( along with a few more helpers I didn't cover here ).