Troostwijk Auction Scraper

A Java-based web scraper for Dutch auctions on Troostwijk Auctions with 100% free desktop/email notifications, SQLite persistence, and AI-powered object detection.

Features

• Auction Discovery: Automatically discovers active Dutch auctions
• Data Scraping: Fetches detailed lot information via Troostwijk's JSON API
• SQLite Storage: Persists auction data, lots, images, and detected objects
• Image Processing: Downloads and analyzes lot images using OpenCV YOLO object detection
• Free Notifications: Real-time notifications when:
  • Bids change on monitored lots
  • Auctions are closing soon (within 5 minutes)
  • Via desktop notifications (Windows/macOS/Linux system tray) ✅
  • Optionally via email (Gmail SMTP - free) ✅

Dependencies

All dependencies are managed via Maven (see pom.xml):

• jsoup 1.17.2 - HTML parsing and HTTP client
• Jackson 2.17.0 - JSON processing
• SQLite JDBC 3.45.1.0 - Database operations
• JavaMail 1.6.2 - Email notifications (free)
• OpenCV 4.9.0 - Image processing and object detection

Setup

1. Notification Options (Choose One)

Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)

Desktop notifications work out of the box on:

• Windows: System tray notifications
• macOS: Notification Center
• Linux: Desktop environment notifications (GNOME, KDE, etc.)

No configuration required! Just run with default settings:

export NOTIFICATION_CONFIG="desktop"
# Or simply don't set it - desktop is the default

Option B: Desktop + Email Notifications 📧 (Free Gmail)

1. Enable 2-Factor Authentication in your Google Account
2. Go to: Google Account → Security → 2-Step Verification → App passwords
3. Generate an app password for "Mail"
4. Set environment variable:

   export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
   

Format: smtp:username:app_password:recipient_email

Example:

export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"

Note: This is completely free using Gmail's SMTP server. No paid services required!

2. OpenCV Native Libraries

Download and install OpenCV native libraries for your platform:

Windows:

# Download from https://opencv.org/releases/
# Extract and add to PATH or use:
java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar

Linux:

sudo apt-get install libopencv-dev

macOS:

brew install opencv

3. YOLO Model Files

Download YOLO model files for object detection:

mkdir models
cd models

# Download YOLOv4 config
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg

# Download YOLOv4 weights (245 MB)
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights

# Download COCO class names
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names

Building

mvn clean package

This creates:

• target/troostwijk-scraper-1.0-SNAPSHOT.jar - Regular JAR
• target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar - Executable JAR with all dependencies

Running

Quick Start (Desktop Notifications Only)

java -Djava.library.path="/path/to/opencv/lib" \
     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar

With Email Notifications

export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"

java -Djava.library.path="/path/to/opencv/lib" \
     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar

Using Maven

mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"

Project Structure

src/main/java/com/auction/scraper/
├── TroostwijkScraper.java          # Main scraper class
│   ├── Lot                          # Domain model for auction lots
│   ├── DatabaseService              # SQLite operations
│   ├── NotificationService          # Desktop + Email notifications (FREE)
│   └── ObjectDetectionService       # OpenCV YOLO object detection
└── Main.java                        # Entry point

Configuration

Edit TroostwijkScraper.main() to customize:

• Database file: troostwijk.db (SQLite database location)
• YOLO paths: Model configuration and weights files
• Monitoring frequency: Default is every 1 hour
• Closing alerts: Default is 5 minutes before closing

Database Schema

The scraper creates three tables:

sales

• sale_id (PRIMARY KEY)
• title, location, closing_time

lots

• lot_id (PRIMARY KEY)
• sale_id, title, description, manufacturer, type, year
• category, current_bid, currency, url
• closing_time, closing_notified

images

• id (PRIMARY KEY)
• lot_id, url, file_path, labels (detected objects)

Notification Examples

Desktop Notification

![System Tray Notification]

🔔 Kavel bieding update
Nieuw bod op kavel 12345: €150.00 (was €125.00)

Email Notification

From: your.email@gmail.com
To: your.email@gmail.com
Subject: [Troostwijk] Kavel bieding update

Nieuw bod op kavel 12345: €150.00 (was €125.00)

High Priority Alerts (closing soon):

⚠️ Lot nearing closure
Kavel 12345 sluit binnen 5 min.

Why This Approach?

✅ 100% Free - No paid services (Twilio, Pushover, etc.) ✅ No External Dependencies - Desktop notifications built into Java ✅ Works Offline - Desktop notifications don't need internet ✅ Privacy First - Your data stays on your machine ✅ Cross-Platform - Windows, macOS, Linux supported ✅ Optional Email - Add Gmail notifications if you want

Troubleshooting

Desktop Notifications Not Showing

• Windows: Check if Java has notification permissions
• Linux: Ensure you have a desktop environment running (not headless)
• macOS: Check System Preferences → Notifications

Email Not Sending

1. Verify 2FA is enabled in Google Account
2. Confirm you're using an App Password (not your regular Gmail password)
3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
4. Verify the SMTP format: smtp:username:app_password:recipient

Notes

• Desktop notifications require a graphical environment (not headless servers)
• For headless servers, use email-only notifications
• Gmail SMTP is free and has generous limits (500 emails/day)
• OpenCV native libraries must match your platform architecture
• YOLO weights file is ~245 MB

License

This is example code for educational purposes.