Architecture

Overview

The Troostwijk Auctions Scraper is a Python-based web scraper that extracts auction lot data using Playwright for browser automation and SQLite for caching.

Core Components

1. Browser Automation (Playwright)

Launches Chromium browser in headless mode
Bypasses Cloudflare protection
Handles dynamic content rendering
Supports network idle detection

2. Cache Manager (SQLite)

Caches every fetched page
Prevents redundant requests
Stores page content, timestamps, and status codes
Auto-cleans entries older than 7 days
Database: cache.db

3. Rate Limiter

Enforces exactly 0.5 seconds between requests
Prevents server overload
Tracks last request time globally

4. Data Extractor

Primary method: Parses __NEXT_DATA__ JSON from Next.js pages
Fallback method: HTML pattern matching with regex
Extracts: title, location, bid info, dates, images, descriptions

5. Output Manager

Exports data in JSON and CSV formats
Saves progress checkpoints every 10 lots
Timestamped filenames for tracking

Data Flow

1. Listing Pages → Extract lot URLs → Store in memory
                                           ↓
2. For each lot URL → Check cache → If cached: use cached content
                          ↓              If not: fetch with rate limit
                          ↓
3. Parse __NEXT_DATA__ JSON → Extract fields → Store in results
                          ↓
4. Every 10 lots → Save progress checkpoint
                          ↓
5. All lots complete → Export final JSON + CSV

Key Design Decisions

Why Playwright?

Handles JavaScript-rendered content (Next.js)
Bypasses Cloudflare protection
More reliable than requests/BeautifulSoup for modern SPAs

Why JSON extraction?

Site uses Next.js with embedded __NEXT_DATA__
JSON is more reliable than HTML pattern matching
Avoids breaking when HTML/CSS changes
Faster parsing

Why SQLite caching?

Persistent across runs
Reduces load on target server
Enables test mode without re-fetching
Respects website resources

File Structure

troost-scraper/
├── main.py                    # Main scraper logic
├── requirements.txt           # Python dependencies
├── README.md                  # Documentation
├── .gitignore                 # Git exclusions
└── output/                    # Generated files (not in git)
    ├── cache.db              # SQLite cache
    ├── *_partial_*.json      # Progress checkpoints
    ├── *_final_*.json        # Final JSON output
    └── *_final_*.csv         # Final CSV output

Classes

`CacheManager`

__init__(db_path) - Initialize cache database
get(url, max_age_hours) - Retrieve cached page
set(url, content, status_code) - Cache a page
clear_old(max_age_hours) - Remove old entries

`TroostwijkScraper`

crawl_auctions(max_pages) - Main entry point
crawl_listing_page(page, page_num) - Extract lot URLs
crawl_lot(page, url) - Scrape individual lot
_extract_nextjs_data(content) - Parse JSON data
_parse_lot_page(content, url) - Extract all fields
save_final_results(data) - Export JSON + CSV

Scalability Notes

Rate limiting prevents IP blocks but slows execution
Caching makes subsequent runs instant for unchanged pages
Progress checkpoints allow resuming after interruption
Async/await used throughout for non-blocking I/O