A Python web scraping application that collects book data from multiple pages of a website using Requests and BeautifulSoup. It supports searching, price filtering, and price sorting.
Upgrade a basic web scraper into a more practical data-extraction tool.
The project demonstrates:
Website → HTML → BeautifulSoup → Multiple Pages → Data Processing → Filtering → Sorting
- 🌐 Scrape data from a real website
- 📄 Scrape multiple pages
- 📚 Extract book titles
- 💰 Extract book prices
- ⭐ Extract book ratings
- 🔎 Search books by title
- 💵 Filter books by maximum price
- 📊 Sort books by price
- 🧹 Convert scraped prices from strings to numbers
⚠️ Handle network errors- ⏱️ Use request timeout
- 📦 Store scraped data using Python dictionaries and lists
- Python
- Requests
- BeautifulSoup4
- HTML
- Web Scraping
- Object-Oriented Programming
- Lists
- Dictionaries
- Lambda Functions
- Sorting and Filtering
python-web-scraper/
│
├── main.py
├── web_scraper.py
├── README.md
└── .gitignore
The project uses:
https://books.toscrape.com/
Books to Scrape is a website designed for practicing web scraping.
Install the required libraries:
pip install requests beautifulsoup4Run:
python main.py📚 ADVANCED WEB SCRAPER
1. Scrape Books
2. Search Book
3. Filter by Price
4. Sort by Price
5. Exit
The scraper follows this process:
Website
↓
requests.get()
↓
HTML Response
↓
BeautifulSoup
↓
Find HTML Elements
↓
Extract Book Data
↓
Python List + Dictionaries
↓
Search / Filter / Sort
The project supports scraping multiple pages.
The page URL is generated dynamically:
for page in range(1, pages + 1):
url = f"https://books.toscrape.com/catalogue/page-{page}.html"For example:
pages = 3
generates:
page-1.html
page-2.html
page-3.html
This allows the scraper to collect data from multiple pages instead of only the first page.
The scraper finds book cards using:
soup.find_all(
"article",
class_="product_pod"
)Each book is then processed individually.
The title is extracted using:
title = book.find("h3").find("a")["title"]The scraper navigates through:
article
↓
h3
↓
a
↓
title attribute
The website provides the price as text:
£51.77
The scraper first gets the text:
price_text = book.find(
"p",
class_="price_color"
).get_text(strip=True)Then converts it to a number:
price = float(
price_text.replace("£", "").replace("£", "")
)So:
"£51.77"
becomes:
51.77
This allows Python to compare and sort prices mathematically.
The rating is stored inside the HTML class.
Example:
<p class="star-rating Three">The scraper gets the class:
rating_class = book.find(
"p",
class_="star-rating"
)["class"]Then extracts:
rating = rating_class[1]Result:
Three
Each book is stored as a dictionary:
{
"title": title,
"price": price,
"rating": rating
}All dictionaries are stored in a list:
books = []This makes the scraped information easy to process.
The scraper supports case-insensitive searching:
if keyword.lower() in book["title"].lower():For example:
Search:
python
can find books containing Python in the title.
The scraper can find books below a maximum price.
Example:
if book["price"] <= max_price:
filtered_books.append(book)If the maximum price is:
£20
only books costing £20 or less are returned.
The project uses:
sorted(
books,
key=lambda book: book["price"],
reverse=descending
)The key tells Python to compare books using their price.
For example:
key=lambda book: book["price"]sorts books according to their price.
Finds the first matching HTML element.
soup.find("h3")Finds all matching HTML elements.
soup.find_all("h3")Extracts visible text from an HTML element.
element.get_text(strip=True)Attributes can be accessed using:
element["title"]Used for pagination:
range(1, pages + 1)If pages = 3:
1
2
3
Used to define what value Python should use for sorting:
key=lambda book: book["price"]Sorts a collection:
sorted(books, key=lambda book: book["price"])The project handles request failures:
try:
response = requests.get(url, timeout=10)
except requests.RequestException:
print("Network error. ❌")The timeout=10 prevents the application from waiting forever.
- HTML structure
- HTML parsing
- BeautifulSoup
find()find_all()- HTML attributes
- Text extraction
- Multiple-page scraping
- Classes
- Methods
- Lists
- Dictionaries
- Loops
- Conditions
- String manipulation
float()range()sorted()- Lambda functions
- Exception handling
requests.get()response.textresponse.status_codeRequestException- Request timeout
API
↓
JSON
↓
Python
↓
Analysis
Website
↓
HTML
↓
BeautifulSoup
↓
Python
↓
Analysis
This project builds an important foundation for data collection and automation.
Possible future upgrades:
- 📄 Scrape all available pages automatically
- ⭐ Filter by rating
- 💰 Price range filtering
- 📊 Sort by rating
- 📁 Export data to CSV
- 🗄️ Store scraped data in SQLite
- 🔄 Avoid duplicate records
- 📈 Generate price/rating charts
- 🖥️ Build a Streamlit interface
- ⏰ Schedule automatic scraping
- 🔍 Add category-based scraping
- 📦 Add advanced pagination detection
Ayushman Tiwari
GitHub:
https://github.com/ayushman-ece
Advanced Version Completed ✅
The project now supports multi-page web scraping, book searching, price filtering, price conversion, price sorting, and network error handling.