Scrape Job Listings from Multiple Sites into One Spreadsheet

To consolidate job listings from multiple websites into a single spreadsheet, you need to extract raw data from each site, normalize varying schemas into uniform key-value pairs (Title, Company, Location, URL, Source), merge the data streams, and output to CSV or Excel.

Standardized Multi-Site Python Pipeline

The standard stack uses requests and BeautifulSoup for HTML parsing, or native JSON handling for internal API endpoints, paired with pandas for data unification and CSV export.

import requests
from bs4 import BeautifulSoup
import pandas as pd

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

def parse_site_a():
    """Extract listings from HTML-based job board."""
    url = "https://example-jobs-site1.com/listings"
    response = requests.get(url, headers=HEADERS, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    
    jobs = []
    for card in soup.select(".job-card"):
        jobs.append({
            "title": card.select_one(".job-title").get_text(strip=True) if card.select_one(".job-title") else None,
            "company": card.select_one(".company-name").get_text(strip=True) if card.select_one(".company-name") else None,
            "location": card.select_one(".location").get_text(strip=True) if card.select_one(".location") else "Remote",
            "url": card.select_one("a")["href"] if card.select_one("a") else None,
            "source": "Site A"
        })
    return jobs

def parse_site_b():
    """Extract listings from a site using an internal JSON endpoint."""
    url = "https://example-jobs-site2.com/api/v1/jobs"
    response = requests.get(url, headers=HEADERS, timeout=10)
    data = response.json()
    
    jobs = []
    for item in data.get("jobs", []):
        jobs.append({
            "title": item.get("title"),
            "company": item.get("company_name"),
            "location": item.get("location_name"),
            "url": f"https://example-jobs-site2.com/jobs/{item.get('id')}",
            "source": "Site B"
        })
    return jobs

# Combine datasets and export
combined_jobs = parse_site_a() + parse_site_b()
df = pd.DataFrame(combined_jobs)

# Deduplicate identical postings across sites
df.drop_duplicates(subset=["title", "company"], inplace=True)

# Save to spreadsheet
df.to_csv("master_job_listings.csv", index=False)

Handling Dynamic (JavaScript-Rendered) Pages

If a target job board relies on client-side rendering (e.g., React or Vue) and standard HTTP GET requests return empty container tags, use playwright to execute browser interactions before scraping nodes.

from playwright.sync_api import sync_playwright

def parse_dynamic_site():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto("https://example-dynamic-jobs.com")
        page.wait_for_selector(".job-tile")
        
        jobs = []
        for element in page.query_selector_all(".job-tile"):
            jobs.append({
                "title": element.query_selector(".title").inner_text(),
                "company": element.query_selector(".company").inner_text(),
                "location": "N/A",
                "url": page.url,
                "source": "Dynamic Site"
            })
        browser.close()
        return jobs

Data Cleaning and Standardization

Different sites format data differently. Before exporting to your target file format, apply basic normalization routines:

• Whitespace Stripping: Strip line breaks (\n) and non-breaking space characters (\xa0) from titles and descriptions.
• URL Formatting: Convert relative paths (/job/123) into absolute URLs using urllib.parse.urljoin.
• Deduplication: Use df.drop_duplicates(subset=["title", "company"]) to eliminate identical listings published across multiple boards.

Technical Constraints and Rate Limiting

Building a persistent multi-site scraper introduces several operational challenges:

• Anti-Bot Countermeasures: Cloudflare, Datadome, and Akamai block generic HTTP clients. You may need browser header spoofing, delayed request intervals (time.sleep), or rotating proxies.
• DOM Instability: Target sites change CSS selectors without notice, breaking static parser logic. Build try-except fallbacks for individual fields.
• Terms of Service: Scraping rules vary. Check each site's robots.txt file to determine crawl permissions.

Need this done fast? order a scraper on FreelanceHunt.