236 lines
6.6 KiB
Markdown
236 lines
6.6 KiB
Markdown
# Troostwijk Auction Scraper
|
|
|
|
A Java-based web scraper for Dutch auctions on Troostwijk Auctions with **100% free** desktop/email notifications, SQLite persistence, and AI-powered object detection.
|
|
|
|
## Features
|
|
|
|
- **Auction Discovery**: Automatically discovers active Dutch auctions
|
|
- **Data Scraping**: Fetches detailed lot information via Troostwijk's JSON API
|
|
- **SQLite Storage**: Persists auction data, lots, images, and detected objects
|
|
- **Image Processing**: Downloads and analyzes lot images using OpenCV YOLO object detection
|
|
- **Free Notifications**: Real-time notifications when:
|
|
- Bids change on monitored lots
|
|
- Auctions are closing soon (within 5 minutes)
|
|
- Via desktop notifications (Windows/macOS/Linux system tray) ✅
|
|
- Optionally via email (Gmail SMTP - free) ✅
|
|
|
|
## Dependencies
|
|
|
|
All dependencies are managed via Maven (see `pom.xml`):
|
|
|
|
- **jsoup 1.17.2** - HTML parsing and HTTP client
|
|
- **Jackson 2.17.0** - JSON processing
|
|
- **SQLite JDBC 3.45.1.0** - Database operations
|
|
- **JavaMail 1.6.2** - Email notifications (free)
|
|
- **OpenCV 4.9.0** - Image processing and object detection
|
|
|
|
## Setup
|
|
|
|
### 1. Notification Options (Choose One)
|
|
|
|
#### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)
|
|
|
|
Desktop notifications work out of the box on:
|
|
- **Windows**: System tray notifications
|
|
- **macOS**: Notification Center
|
|
- **Linux**: Desktop environment notifications (GNOME, KDE, etc.)
|
|
|
|
**No configuration required!** Just run with default settings:
|
|
```bash
|
|
export NOTIFICATION_CONFIG="desktop"
|
|
# Or simply don't set it - desktop is the default
|
|
```
|
|
|
|
#### Option B: Desktop + Email Notifications 📧 (Free Gmail)
|
|
|
|
1. Enable 2-Factor Authentication in your Google Account
|
|
2. Go to: **Google Account → Security → 2-Step Verification → App passwords**
|
|
3. Generate an app password for "Mail"
|
|
4. Set environment variable:
|
|
```bash
|
|
export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
|
|
```
|
|
|
|
**Format**: `smtp:username:app_password:recipient_email`
|
|
|
|
**Example**:
|
|
```bash
|
|
export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"
|
|
```
|
|
|
|
**Note**: This is completely free using Gmail's SMTP server. No paid services required!
|
|
|
|
### 2. OpenCV Native Libraries
|
|
|
|
Download and install OpenCV native libraries for your platform:
|
|
|
|
**Windows:**
|
|
```bash
|
|
# Download from https://opencv.org/releases/
|
|
# Extract and add to PATH or use:
|
|
java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar
|
|
```
|
|
|
|
**Linux:**
|
|
```bash
|
|
sudo apt-get install libopencv-dev
|
|
```
|
|
|
|
**macOS:**
|
|
```bash
|
|
brew install opencv
|
|
```
|
|
|
|
### 3. YOLO Model Files
|
|
|
|
Download YOLO model files for object detection:
|
|
|
|
```bash
|
|
mkdir models
|
|
cd models
|
|
|
|
# Download YOLOv4 config
|
|
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
|
|
|
|
# Download YOLOv4 weights (245 MB)
|
|
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
|
|
|
|
# Download COCO class names
|
|
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
|
|
```
|
|
|
|
## Building
|
|
|
|
```bash
|
|
mvn clean package
|
|
```
|
|
|
|
This creates:
|
|
- `target/troostwijk-scraper-1.0-SNAPSHOT.jar` - Regular JAR
|
|
- `target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar` - Executable JAR with all dependencies
|
|
|
|
## Running
|
|
|
|
### Quick Start (Desktop Notifications Only)
|
|
|
|
```bash
|
|
java -Djava.library.path="/path/to/opencv/lib" \
|
|
-jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
|
|
```
|
|
|
|
### With Email Notifications
|
|
|
|
```bash
|
|
export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"
|
|
|
|
java -Djava.library.path="/path/to/opencv/lib" \
|
|
-jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
|
|
```
|
|
|
|
### Using Maven
|
|
|
|
```bash
|
|
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
src/main/java/com/auction/scraper/
|
|
├── TroostwijkScraper.java # Main scraper class
|
|
│ ├── Lot # Domain model for auction lots
|
|
│ ├── DatabaseService # SQLite operations
|
|
│ ├── NotificationService # Desktop + Email notifications (FREE)
|
|
│ └── ObjectDetectionService # OpenCV YOLO object detection
|
|
└── Main.java # Entry point
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Edit `TroostwijkScraper.main()` to customize:
|
|
|
|
- **Database file**: `troostwijk.db` (SQLite database location)
|
|
- **YOLO paths**: Model configuration and weights files
|
|
- **Monitoring frequency**: Default is every 1 hour
|
|
- **Closing alerts**: Default is 5 minutes before closing
|
|
|
|
## Database Schema
|
|
|
|
The scraper creates three tables:
|
|
|
|
**sales**
|
|
- `sale_id` (PRIMARY KEY)
|
|
- `title`, `location`, `closing_time`
|
|
|
|
**lots**
|
|
- `lot_id` (PRIMARY KEY)
|
|
- `sale_id`, `title`, `description`, `manufacturer`, `type`, `year`
|
|
- `category`, `current_bid`, `currency`, `url`
|
|
- `closing_time`, `closing_notified`
|
|
|
|
**images**
|
|
- `id` (PRIMARY KEY)
|
|
- `lot_id`, `url`, `file_path`, `labels` (detected objects)
|
|
|
|
## Notification Examples
|
|
|
|
### Desktop Notification
|
|
![System Tray Notification]
|
|
```
|
|
🔔 Kavel bieding update
|
|
Nieuw bod op kavel 12345: €150.00 (was €125.00)
|
|
```
|
|
|
|
### Email Notification
|
|
```
|
|
From: your.email@gmail.com
|
|
To: your.email@gmail.com
|
|
Subject: [Troostwijk] Kavel bieding update
|
|
|
|
Nieuw bod op kavel 12345: €150.00 (was €125.00)
|
|
```
|
|
|
|
**High Priority Alerts** (closing soon):
|
|
```
|
|
⚠️ Lot nearing closure
|
|
Kavel 12345 sluit binnen 5 min.
|
|
```
|
|
|
|
## Why This Approach?
|
|
|
|
✅ **100% Free** - No paid services (Twilio, Pushover, etc.)
|
|
✅ **No External Dependencies** - Desktop notifications built into Java
|
|
✅ **Works Offline** - Desktop notifications don't need internet
|
|
✅ **Privacy First** - Your data stays on your machine
|
|
✅ **Cross-Platform** - Windows, macOS, Linux supported
|
|
✅ **Optional Email** - Add Gmail notifications if you want
|
|
|
|
## Troubleshooting
|
|
|
|
### Desktop Notifications Not Showing
|
|
|
|
- **Windows**: Check if Java has notification permissions
|
|
- **Linux**: Ensure you have a desktop environment running (not headless)
|
|
- **macOS**: Check System Preferences → Notifications
|
|
|
|
### Email Not Sending
|
|
|
|
1. Verify 2FA is enabled in Google Account
|
|
2. Confirm you're using an **App Password** (not your regular Gmail password)
|
|
3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
|
|
4. Verify the SMTP format: `smtp:username:app_password:recipient`
|
|
|
|
## Notes
|
|
|
|
- Desktop notifications require a graphical environment (not headless servers)
|
|
- For headless servers, use email-only notifications
|
|
- Gmail SMTP is free and has generous limits (500 emails/day)
|
|
- OpenCV native libraries must match your platform architecture
|
|
- YOLO weights file is ~245 MB
|
|
|
|
## License
|
|
|
|
This is example code for educational purposes.
|
|
|
|
|