Skip to content

Tutorial.md

Latest commit

 

History

History
948 lines (704 loc) · 19.3 KB

Tutorial.md

File metadata and controls

948 lines (704 loc) · 19.3 KB
Raw

Getting Started Tutorial

This tutorial walks you through building your first web scraper from installation to your first successful scrape.

Time Required: 15-20 minutes Prerequisites: Python 3.12+, Docker Desktop

📚 What You'll Learn

By the end of this tutorial, you'll:

  1. ✅ Set up the scraping platform locally
  2. ✅ Scrape a real website (quotes.toscrape.com)
  3. ✅ Extract structured data using CSS selectors
  4. ✅ View results in MongoDB
  5. ✅ Understand the worker system

Step 1: Installation (5 minutes)

1.1 Install Python Dependencies

# Navigate to the project directory
cd /path/to/scraper

# Install using uv (fastest - recommended)
uv pip install -e ".[dev]"

# OR using pip
pip install -e ".[dev]"

What this does: Installs httpx, BeautifulSoup, pymongo, redis, pydantic, and all dependencies.

1.2 Start MongoDB and Redis

# Start services using Docker Compose
docker compose up -d mongodb redis

# Verify services are running
docker compose ps

Expected output:

NAME                  STATUS    PORTS
scraper-mongodb       Up        0.0.0.0:27017->27017/tcp
scraper-redis         Up        0.0.0.0:6379->6379/tcp

1.3 Verify Setup

python scripts/verify_setup.py

Expected output:

🔍 Verifying scraper setup...

✓ Checking imports...
  ✓ All core modules imported successfully

✓ Checking configuration...
  ✓ MongoDB URI: mongodb://localhost:27017
  ✓ Redis URL: redis://localhost:6379/0
  ✓ Job Queue: scraper:jobs
  ✓ Worker concurrency: 5

✓ Checking MongoDB connection...
  ✓ MongoDB connected: scraper
  ✓ Jobs count: 0
  ✓ Results count: 0

✓ Checking Redis connection...
  ✓ Redis connected: redis://localhost:6379/0
  ✓ Job queue length: 0

==================================================
✅ All checks passed! Setup is complete.

If verification fails: See Troubleshooting section below.

Step 2: Your First Scrape (5 minutes)

We'll scrape quotes.toscrape.com, a website specifically designed for practicing web scraping.

2.1 Inspect the Target Website

Open https://quotes.toscrape.com in your browser.

What we'll extract:

  • Quotes (the text of each quote)
  • Authors (who said each quote)
  • Tags (categories for each quote)

Finding the selectors:

  1. Right-click on a quote → "Inspect Element"
  2. Notice the HTML structure:
<div class="quote">
    <span class="text">"The world as we have created it..."</span>
    <small class="author">Albert Einstein</small>
    <div class="tags">
        <a class="tag">change</a>
        <a class="tag">deep-thoughts</a>
    </div>
</div>
  1. CSS selectors we need:
    • Quotes: span.text
    • Authors: small.author
    • Tags: a.tag

2.2 Enqueue the Scraping Job

python scripts/enqueue_job.py \
  --url "https://quotes.toscrape.com" \
  --type http \
  --selectors '{
    "quotes": {
      "selector": "span.text",
      "type": "css",
      "multiple": true
    },
    "authors": {
      "selector": "small.author",
      "type": "css",
      "multiple": true
    },
    "tags": {
      "selector": "a.tag",
      "type": "css",
      "multiple": true
    }
  }'

Expected output:

✓ Enqueued job a1b2c3d4e5f6
  URL: https://quotes.toscrape.com
  Type: http
  Queue: scraper:jobs

What just happened:

  1. A Job object was created with your URL and extraction rules
  2. The job was serialized to JSON
  3. JSON was pushed to Redis queue scraper:jobs
  4. The job is now waiting for a worker to process it

2.3 Start the Worker

Open a new terminal and run:

python scripts/run_worker.py

Expected output:

2026-02-17 10:00:00 - scraper.worker - INFO - Starting scraper worker...
2026-02-17 10:00:00 - scraper.db - INFO - Connecting to MongoDB at mongodb://localhost:27017
2026-02-17 10:00:00 - scraper.db - INFO - MongoDB connection established
2026-02-17 10:00:00 - scraper.worker - INFO - Worker started successfully
2026-02-17 10:00:00 - scraper.worker - INFO - Entering processing loop (queue: scraper:jobs)
2026-02-17 10:00:01 - scraper.worker - INFO - Received job from scraper:jobs
2026-02-17 10:00:01 - scraper.worker - INFO - Processing job a1b2c3d4: https://quotes.toscrape.com
2026-02-17 10:00:01 - scraper.scrapers.base - INFO - Fetching GET https://quotes.toscrape.com
2026-02-17 10:00:02 - scraper.scrapers.base - INFO - Successfully scraped https://quotes.toscrape.com in 850ms
2026-02-17 10:00:02 - scraper.pipeline - INFO - Stored result for job a1b2c3d4
2026-02-17 10:00:02 - scraper.worker - INFO - Successfully completed job a1b2c3d4

What's happening:

  1. Worker connects to MongoDB and Redis
  2. Worker pops your job from the queue (BRPOP - blocking operation)
  3. Worker checks rate limit for quotes.toscrape.com
  4. Worker fetches the URL using httpx
  5. Worker parses HTML with BeautifulSoup
  6. Worker extracts data using CSS selectors
  7. Worker runs pipeline (clean → transform → validate)
  8. Worker stores result in MongoDB
  9. Worker waits for the next job

Keep this terminal open! The worker will continue processing jobs.

Step 3: View the Results (2 minutes)

Option 1: Mongo Express (Visual Interface)

  1. Start Mongo Express:
docker compose up -d mongo-express

  1. Open in browser: http://localhost:8082

  2. Login:

    • Username: admin
    • Password: admin

  3. Navigate to results:

    • Click "scraper" database
    • Click "results" collection
    • Click "View all documents"

  4. Inspect the result:

{
  "_id": "xyz789...",
  "job_id": "a1b2c3d4e5f6",
  "data": {
    "quotes": [
      "\"The world as we have created it is a process of our thinking...\"",
      "\"It is our choices, Harry, that show what we truly are...\"",
      "\"There are only two ways to live your life...\"",
      // ... more quotes
    ],
    "authors": [
      "Albert Einstein",
      "J.K. Rowling",
      "Albert Einstein",
      // ... more authors
    ],
    "tags": [
      "change",
      "deep-thoughts",
      "thinking",
      // ... more tags
    ]
  },
  "status": "success",
  "status_code": 200,
  "response_time_ms": 850,
  "scraped_at": "2026-02-17T10:00:02.123Z",
  "metadata": {
    "content_type": "text/html; charset=utf-8",
    "content_length": 11053
  }
}

Option 2: MongoDB Shell

# Connect to MongoDB container
docker exec -it scraper-mongodb mongosh scraper

# Query results
db.results.find().pretty()

# Count results
db.results.countDocuments({})

# Find latest result
db.results.find().sort({scraped_at: -1}).limit(1).pretty()

# Exit
exit

Option 3: Python Script

# quick_check.py
import asyncio
from scraper.db import db

async def check_results():
    await db.connect()

    # Get latest result
    result = await db.results.find_one(sort=[("scraped_at", -1)])

    if result:
        print(f"✅ Found result!")
        print(f"   Quotes extracted: {len(result['data'].get('quotes', []))}")
        print(f"   Authors extracted: {len(result['data'].get('authors', []))}")
        print(f"   Response time: {result.get('response_time_ms')}ms")
    else:
        print("❌ No results found")

    await db.disconnect()

asyncio.run(check_results())

Run: python quick_check.py

Step 4: Understanding What Happened

The Data Flow

1. You (enqueue_job.py)
   ↓
   Created a Job object with:
   - URL to scrape
   - Extractor configuration (CSS selectors)
   ↓
2. Redis Queue (scraper:jobs)
   ↓
   Job stored as JSON, waiting for worker
   ↓
3. Worker (run_worker.py)
   ↓
   Popped job from queue
   ↓
4. Rate Limiter
   ↓
   Checked if we can scrape quotes.toscrape.com
   (Limit: 10 requests per 60 seconds)
   ↓
5. HTTP Scraper
   ↓
   Fetched HTML using httpx
   ↓
6. CSS Extractor
   ↓
   Found all <span class="text"> elements → extracted quotes
   Found all <small class="author"> elements → extracted authors
   Found all <a class="tag"> elements → extracted tags
   ↓
7. Pipeline
   ↓
   Cleaned data (stripped whitespace)
   Validated data (checked non-empty)
   ↓
8. MongoDB (results collection)
   ↓
   Result stored with:
   - Extracted data
   - Status code (200)
   - Response time (850ms)
   - Timestamp

Key Concepts

Job: What to scrape

{
  "url": "https://quotes.toscrape.com",
  "selectors": {"quotes": "span.text"}
}

Result: What was scraped

{
  "data": {"quotes": ["quote 1", "quote 2"]},
  "status": "success",
  "response_time_ms": 850
}

Worker: Does the scraping

  • Fetches URLs
  • Extracts data
  • Stores results

Pipeline: Cleans the data

  • Removes whitespace
  • Removes empty values
  • Validates structure

Step 5: Scrape Another Website (3 minutes)

Let's scrape a product page to extract price and description.

5.1 Scrape books.toscrape.com

python scripts/enqueue_job.py \
  --url "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" \
  --type http \
  --selectors '{
    "title": {
      "selector": "h1",
      "type": "css"
    },
    "price": {
      "selector": "p.price_color",
      "type": "css"
    },
    "availability": {
      "selector": "p.availability",
      "type": "css"
    },
    "description": {
      "selector": "#product_description + p",
      "type": "css"
    },
    "rating": {
      "selector": "p.star-rating",
      "type": "css",
      "attribute": "class"
    }
  }'

Watch the worker terminal - you'll see it process the new job immediately!

5.2 Check Results

# In MongoDB shell
db.results.find().sort({scraped_at: -1}).limit(1).pretty()

Expected data:

{
  "data": {
    "title": "A Light in the Attic",
    "price": "£51.77",
    "availability": "In stock (22 available)",
    "description": "It's hard to imagine a world without A Light in the Attic...",
    "rating": "star-rating Three"
  }
}

Step 6: Scrape a JSON API (2 minutes)

Let's scrape a public API to get different data.

6.1 Scrape JSONPlaceholder API

python scripts/enqueue_job.py \
  --url "https://jsonplaceholder.typicode.com/posts/1" \
  --type api \
  --selectors '{
    "title": {
      "selector": "title",
      "type": "jsonpath"
    },
    "body": {
      "selector": "body",
      "type": "jsonpath"
    },
    "user_id": {
      "selector": "userId",
      "type": "jsonpath"
    }
  }'

6.2 Check Results

The worker will extract data from the JSON response:

{
  "data": {
    "title": "sunt aut facere repellat provident...",
    "body": "quia et suscipit\nsuscipit recusandae...",
    "user_id": 1
  }
}

Advanced Tutorial: XPath Selectors

CSS selectors are simple, but XPath is more powerful for complex queries.

Example: Extract Specific Elements

python scripts/enqueue_job.py \
  --url "https://quotes.toscrape.com" \
  --type http \
  --selectors '{
    "first_quote": {
      "selector": "//div[@class=\"quote\"][1]//span[@class=\"text\"]",
      "type": "xpath"
    },
    "einstein_quotes": {
      "selector": "//div[@class=\"quote\"][.//small[@class=\"author\"][text()=\"Albert Einstein\"]]//span[@class=\"text\"]",
      "type": "xpath",
      "multiple": true
    }
  }'

XPath advantages:

  • Positional selection ([1], [last()])
  • Conditional selection (quotes by specific author)
  • Parent/sibling traversal

Monitoring and Debugging

Check Worker Status

# Worker logs (in worker terminal)
# Shows each job being processed with timing

# Check Redis queue length
redis-cli LLEN scraper:jobs

# Check job count in MongoDB
docker exec -it scraper-mongodb mongosh scraper --eval "db.jobs.countDocuments({})"

# Check result count
docker exec -it scraper-mongodb mongosh scraper --eval "db.results.countDocuments({})"

Debug Failed Scrapes

If a scrape fails, check the result:

// In MongoDB
db.results.find({status: "failed"}).pretty()

Common failures:

  • Timeout: Increase HTTP_TIMEOUT_SECONDS in .env
  • Wrong selector: Verify CSS/XPath selector in browser console
  • Rate limited: Wait or increase rate limits
  • 404 Not Found: Check URL is correct

Enable Debug Logging

# Start worker with debug logging
python scripts/run_worker.py --log-level DEBUG

Debug logs show:

  • Exact HTTP requests sent
  • Response headers and status codes
  • Extraction step-by-step
  • Pipeline transformations

Store Raw HTML for Debugging

# Edit .env
STORE_RAW_HTML=true

# Restart worker
# Ctrl+C to stop, then restart
python scripts/run_worker.py

Now results will include the raw HTML:

{
  "data": {...},
  "raw_html": "<html><head>...</head><body>...</body></html>",
  ...
}

Use this to:

  • Verify what HTML was actually received
  • Test selectors offline
  • Debug JavaScript-rendered content

Multiple Workers (Scaling)

You can run multiple workers to process jobs faster.

Start 3 Workers

# Terminal 1
python scripts/run_worker.py

# Terminal 2
python scripts/run_worker.py

# Terminal 3
python scripts/run_worker.py

What happens:

  • All workers read from the same Redis queue
  • Each job is processed by only one worker (Redis ensures this)
  • Jobs are distributed automatically across workers
  • If one worker crashes, others continue

Test with Multiple Jobs

# Enqueue 10 jobs
for i in {1..10}; do
  python scripts/enqueue_job.py \
    --url "https://quotes.toscrape.com/page/$i/" \
    --type http \
    --selectors '{"quotes": {"selector": "span.text", "type": "css", "multiple": true}}'
done

Watch all worker terminals - you'll see jobs distributed across them!

Rate Limiting in Action

Let's see rate limiting prevent overwhelming a server.

Check Current Rate Limit

# In .env
RATE_LIMIT_DEFAULT_REQUESTS=10
RATE_LIMIT_DEFAULT_WINDOW_SECONDS=60

This means: 10 requests per domain per 60 seconds

Enqueue 20 Jobs to Same Domain

for i in {1..20}; do
  python scripts/enqueue_job.py \
    --url "https://quotes.toscrape.com/page/$i/" \
    --type http \
    --selectors '{"quotes": {"selector": "span.text", "type": "css", "multiple": true}}'
done

Watch the worker logs:

2026-02-17 10:05:00 - scraper.worker - INFO - Processing job 1...
2026-02-17 10:05:01 - scraper.worker - INFO - Processing job 2...
...
2026-02-17 10:05:10 - scraper.worker - INFO - Processing job 10...
2026-02-17 10:05:11 - scraper.rate_limiter - WARNING - Rate limit exceeded for quotes.toscrape.com
2026-02-17 10:05:11 - scraper.rate_limiter - INFO - Rate limiting quotes.toscrape.com: waiting 49.2s
[Worker waits 49 seconds]
2026-02-17 10:06:00 - scraper.worker - INFO - Processing job 11...

This protects:

  • The target server from being overwhelmed
  • Your IP from being blocked
  • Ensures sustainable scraping

Scheduled Scraping (Future Feature)

This is a preview of Phase 2 functionality.

Job with schedule:

{
  "target_url": "https://example.com/price",
  "schedule": "0 */6 * * *",  // Every 6 hours
  "extractor_config": {...}
}

How it works (in Phase 2):

  1. Elixir scheduler reads jobs with schedule field
  2. Every 6 hours, it enqueues the job to Redis
  3. Worker processes it normally
  4. Pipeline compares with previous result
  5. If changed, set metadata.changed = true
  6. Alert system notifies you of changes

Troubleshooting

Issue: "MongoDB connection failed"

Solution:

# Check MongoDB is running
docker compose ps mongodb

# Should show "Up"
# If not, start it:
docker compose up -d mongodb

# Wait 10 seconds for MongoDB to start
sleep 10

# Verify connection
docker exec -it scraper-mongodb mongosh scraper --eval "db.runCommand('ping')"

Issue: "Redis connection failed"

Solution:

# Check Redis is running
docker compose ps redis

# Start if not running
docker compose up -d redis

# Test connection
redis-cli PING
# Should return "PONG"

Issue: Worker not processing jobs

Check 1: Is queue empty?

redis-cli LLEN scraper:jobs
# If 0, enqueue a job

Check 2: Is worker running?

# Check worker terminal - should show "Entering processing loop"
# If not, restart worker

Check 3: Queue name mismatch?

# Check .env
grep REDIS_JOB_QUEUE .env
# Should be: REDIS_JOB_QUEUE=scraper:jobs

# Check what queues exist in Redis
redis-cli KEYS "scraper:*"

Issue: No data extracted (empty results)

Solution:

  1. Enable raw HTML storage:
# Edit .env
STORE_RAW_HTML=true

# Restart worker
  1. Check result:
db.results.findOne({}, {raw_html: 1})

  1. Verify HTML contains your selector:

    • Copy raw_html to a file
    • Open in browser
    • Inspect element structure

  2. Test selector in browser console:

    • Visit the URL in browser
    • F12 → Console
    • document.querySelector("span.text") (for CSS)
    • Should return an element, not null

Issue: "ModuleNotFoundError"

Solution:

# Reinstall dependencies
pip install -e ".[dev]"

# Verify installation
python -c "import scraper; print('OK')"
# Should print "OK"

Next Steps

Congratulations! 🎉 You've completed the tutorial. Here's what to explore next:

1. Read the Full Documentation

2. Customize Extractors

Edit scraper/extractors/json_path.py to add advanced JSONPath features:

# Add filtering support
"items[?(@.price > 10)]"  # Get items where price > 10

# Add recursive descent
"..author"  # Find "author" at any nesting level

3. Add Custom Pipeline Steps

Create custom data transformations:

# scraper/pipeline.py
def parse_price_step(result: Result) -> Result:
    """Convert £19.99 to 19.99"""
    if "price" in result.data:
        price_str = result.data["price"]
        result.data["price"] = float(price_str.replace("£", ""))
    return result

4. Build a Real Scraper

Ideas:

  • Price monitor: Track product prices, alert on drops
  • Job board aggregator: Scrape multiple job sites
  • News monitor: Track article changes
  • API aggregator: Combine data from multiple APIs

5. Prepare for Phase 2

Phase 2 adds:

  • Elixir/Phoenix API for job management
  • Web-based job builder
  • Scheduled scraping (cron-like)
  • Real-time monitoring dashboard

Quick Reference

Start Services

docker compose up -d mongodb redis

Enqueue Job (HTML)

python scripts/enqueue_job.py \
  --url "URL" \
  --type http \
  --selectors '{"field": {"selector": "CSS", "type": "css"}}'

Enqueue Job (API)

python scripts/enqueue_job.py \
  --url "URL" \
  --type api \
  --selectors '{"field": {"selector": "path.to.field", "type": "jsonpath"}}'

Start Worker

python scripts/run_worker.py

Check Results (MongoDB)

docker exec -it scraper-mongodb mongosh scraper
db.results.find().sort({scraped_at: -1}).limit(1).pretty()

Check Queue (Redis)

redis-cli LLEN scraper:jobs

Stop Services

docker compose down

Getting Help

Issues? Check Troubleshooting section first.

Questions? Review the Overview.md for detailed explanations.

Bugs? Report at https://github.com/yourusername/scraper/issues

Happy Scraping! 🕷️