Web scraping is writing code that reads a website and extracts data from it — automatically.

Instead of manually copying information from a webpage, you write a script that does it for you. It visits the page, reads the HTML, and pulls out the parts you need.

How it works

1. Your script sends an HTTP request to a URL (just like your browser does)
2. The server returns HTML
3. Your script parses the HTML and extracts the data you want
4. You save it, process it, or do something useful with it

A simple example in Python

import urllib.request
import re

# Fetch a page
url = "https://example.com"
html = urllib.request.urlopen(url).read().decode()

# Extract the title
title = re.search(r'<title>(.*?)</title>', html).group(1)
print(title)  # "Example Domain"

For more complex scraping, most people use libraries like BeautifulSoup or Scrapy:

from bs4 import BeautifulSoup
import requests

page = requests.get("https://example.com")
soup = BeautifulSoup(page.content, "html.parser")

# Find all links on the page
for link in soup.find_all("a"):
    print(link.get("href"))

Scraping vs APIs

	Web Scraping	API
Data source	HTML pages	Structured JSON
Reliability	Breaks when site changes	Stable, versioned
Speed	Slower	Faster
Permission	Gray area	Explicitly allowed

Always prefer an API when one exists. Scraping is for when there’s no API available.

When is scraping okay?

• ✅ Public data that anyone can see in a browser
• ✅ The site’s robots.txt doesn’t block it
• ✅ You’re not hammering the server with requests
• ✅ You’re not bypassing login walls or paywalls
• ❌ Don’t scrape personal data (GDPR, privacy laws)
• ❌ Don’t ignore rate limits or robots.txt
• ❌ Don’t resell scraped content as your own

Always check the site’s terms of service.

Scraping without a browser

Some sites load content with JavaScript (single-page apps). Regular HTTP requests won’t see that content. For those, you need a headless browser:

• Playwright — modern, supports all browsers
• Puppeteer — Chrome/Chromium only
• Selenium — older, widely used

Common use cases

• Price monitoring — track product prices across stores
• Job boards — aggregate listings from multiple sites
• Research — collect data for analysis
• Content monitoring — watch for changes on pages
• Lead generation — extract business contact info from directories