When You Need to Organize 1000 Product Price Comparison Entries...

Imagine this scenario: Your boss assigns you a task this morning—he wants you to extract all product names and prices containing "gaming laptop" from PChome, Momo, and Yahoo Shopping, then compile them into an Excel report for competitor analysis.

If you don't know how to code, you'd have no choice but to open three web pages, search for "gaming laptop," then manually highlight the first laptop's name, press Ctrl+C, switch to Excel, press Ctrl+V; then switch back to the webpage, highlight the price, Ctrl+C, switch to Excel, Ctrl+V.
After repeating this tedious process 10 times, you'd start questioning your life. By the 500th repetition, it's already dark outside, and you might even paste data into the wrong cells due to eye strain.

This is why we need "Web Scrapers."

A web scraper is like a virtual spider you train. You give it a URL and say: "Go fetch all the red text and dollar-sign ($) numbers from this webpage for me."
It can crawl 10 web pages in 3 seconds and neatly organize thousands of data entries without errors.

In Python's ecosystem, two legendary tools dominate web scraping:

1. Requests: Knocks on the webpage's door and downloads its raw HTML code.
2. BeautifulSoup: Acts like a magnifying glass to precisely extract specific text (e.g., product names, prices) from the chaotic HTML jungle.

---

The First Rule of Scraping: Understand the Webpage's "Skeleton (HTML)"

Before unleashing our spider, we must teach it "what to look for."
Right-click any product name on a shopping site and select "Inspect." You'll see a wall of code—this is the webpage's skeleton (HTML).

You might spot something like this:

<div class="product-item">
  <h2 class="title">ASUS ROG Premium Gaming Laptop</h2>
  <span class="price">$45,000</span>
</div>

Now you have a treasure map!
Tell your Python scraper: "Find all containers with class="product-item", then extract text from class="title" and numbers from class="price."

---

Vibe Prompt in Action: Generate a Scraper Script Instantly

In the past, writing Requests + BeautifulSoup scripts required constant English documentation checks, and one wrong tag name could crash your scraper.
Now, we simply describe our treasure map to AI!

【Automated Python Scraper Prompt】
I need a Python scraper script.
Target URL: https://example-shop.com/laptops

Use requests to fetch the page and BeautifulSoup for parsing.
Page structure:

1. Each product is wrapped in <div class="product-item">.
2. Product names are inside <h2 class="title">.
3. Prices are inside <span class="price">.

Extract these and save them as laptops_price.csv using pandas.
Include full try-catch error handling with Chinese comments. If blocked (e.g., HTTP 403), add a browser-mimicking User-Agent header.

AI-Generated Pro Scraper Script:

import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_laptops():
    url = 'https://example-shop.com/laptops'
    # Disguise as Chrome to avoid blocking
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
    }

    try:
        print("Starting scrape...")
        # 1. Fetch page content
        response = requests.get(url, headers=headers)
        response.raise_for_status() # Check for errors (e.g., 404)

        # 2. Parse HTML with BeautifulSoup
        soup = BeautifulSoup(response.text, 'html.parser')

        # 3. Find all product containers
        items = soup.find_all('div', class_='product-item')
        
        data_list = []

        # 4. Extract treasures from each container
        for item in items:
            title_tag = item.find('h2', class_='title')
            price_tag = item.find('span', class_='price')
            
            if title_tag and price_tag:
                title = title_tag.text.strip()
                price = price_tag.text.strip()
                data_list.append({"Product": title, "Price": price})

        # 5. Convert to CSV with Pandas
        df = pd.DataFrame(data_list)
        df.to_csv('laptops_price.csv', index=False, encoding='utf-8-sig')
        print(f"✅ Done! Saved {len(data_list)} entries to laptops_price.csv")

    except Exception as e:
        print(f"🚨 Error: {e}")

# Execute
scrape_laptops()

Ethical and Legal Boundaries (Robots.txt)

With great power comes great responsibility. You might think: "Awesome! I'll loop-scrape a competitor's site 100 times per second to steal all their data!"

Stop right there!
Excessive scraping can crash servers and may qualify as DDoS (Denial-of-Service), which carries legal consequences.

Always follow these unwritten rules:

1. Don't be rude: Add time.sleep(3) to pause between pages. Don't overload servers.
2. Respect Robots.txt: Append /robots.txt to URLs (e.g., google.com/robots.txt). This "house rules" file specifies which pages are scrape-friendly. Be an ethical Vibe Coder.

In the next chapter, we'll transform raw CSV data into colorful visualizations that will wow your boss!