Monitor Competitor Prices Automatically with Python

Automating competitor price monitoring requires three core components: an HTTP client to fetch target web pages, an HTML parser to extract specific price strings, and a scheduler to run extraction jobs at automated intervals. This guide builds a functional pipeline using Python's standard tooling ecosystem.

1. Dependencies and Environment Setup

Install the core libraries for HTTP requests, HTML parsing, and job scheduling:

pip install requests beautifulsoup4 schedule

2. Building the Price Scraper Engine

Target site structure varies, but most standard product pages expose prices within specific CSS selectors. Include custom request headers to mimic standard browser behavior and prevent immediate HTTP 403 blocks.

import re
import json
import requests
from bs4 import BeautifulSoup
from datetime import datetime

TARGET_URL = "https://example.com/product/123"
CSS_SELECTOR = ".product-price"  # Replace with target site's specific CSS class or ID

def fetch_price(url, selector):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9"
    }
    
    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()
    
    soup = BeautifulSoup(response.text, "html.parser")
    element = soup.select_one(selector)
    
    if not element:
        raise ValueError(f"Selector '{selector}' not found on target page.")
        
    # Strip currency symbols and non-numeric characters using regex
    raw_text = element.get_text(strip=True)
    clean_price = float(re.sub(r"[^\d.]", "", raw_text))
    return clean_price

3. Logging and Detecting Price Changes

Compare the scraped value against local stored data (JSON or SQLite) to log records over time and identify drops or price hikes.

LOG_FILE = "price_history.json"

def process_price_check():
    try:
        current_price = fetch_price(TARGET_URL, CSS_SELECTOR)
        timestamp = datetime.now().isoformat()
        
        record = {"timestamp": timestamp, "price": current_price}
        
        # Log continuous records to a local JSON file
        with open(LOG_FILE, "a") as f:
            f.write(json.dumps(record) + "\n")
            
        print(f"[{timestamp}] Price recorded: ${current_price:.2f}")
        
        # Simple threshold alert rule
        TARGET_PRICE = 99.99
        if current_price <= TARGET_PRICE:
            print(f"ALERT: Price fell to threshold: ${current_price:.2f}")

    except Exception as e:
        print(f"Scrape job failed: {e}")

4. Automating Execution Schedules

Run the check continuously in the background using Python’s schedule library, or hand off execution to native operating system utilities.

import time
import schedule

# Run check every 6 hours
schedule.every(6).hours.do(process_price_check)

if __name__ == "__main__":
    print("Starting monitoring daemon...")
    process_price_check()  # Run immediately on start
    while True:
        schedule.run_pending()
        time.sleep(60)

For production deployments on Linux servers, execute the standalone Python script via cron rather than keeping a persistent loop running:

0 */6 * * * /usr/bin/python3 /path/to/scraper.py >> /path/to/scraper.log 2>&1

Technical Limitations and Production Edge Cases

• Dynamic Rendering (JavaScript): If prices load via client-side frameworks (React, Vue) or asynchronous API requests, standard HTML parsing will fail. Use headless browsers like Playwright or inspect Network DevTools to target internal JSON endpoints directly.
• Anti-Bot Challenges: Advanced Bot Management systems (Cloudflare, DataDome) inspect TLS fingerprints and IP reputations. Mitigate blocks using rotating residential proxies and browser-like request signatures.
• Layout Instability: Retailers frequently rewrite markup and CSS class names, which can break static selectors without warning. Implement robust error alerting to monitor extraction failures.

Need this done fast? order a price monitor on FreelanceHunt.