start

2025-11-26 13:05:04 +01:00
parent 2bc4b21862
commit 47854d8b39
14 changed files with 1717 additions and 12 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,233 @@
+# Troostwijk Auction Scraper
+
+A Java-based web scraper for Dutch auctions on Troostwijk Auctions with **100% free** desktop/email notifications, SQLite persistence, and AI-powered object detection.
+
+## Features
+
+- **Auction Discovery**: Automatically discovers active Dutch auctions
+- **Data Scraping**: Fetches detailed lot information via Troostwijk's JSON API
+- **SQLite Storage**: Persists auction data, lots, images, and detected objects
+- **Image Processing**: Downloads and analyzes lot images using OpenCV YOLO object detection
+- **Free Notifications**: Real-time notifications when:
+  - Bids change on monitored lots
+  - Auctions are closing soon (within 5 minutes)
+  - Via desktop notifications (Windows/macOS/Linux system tray) ✅
+  - Optionally via email (Gmail SMTP - free) ✅
+
+## Dependencies
+
+All dependencies are managed via Maven (see `pom.xml`):
+
+- **jsoup 1.17.2** - HTML parsing and HTTP client
+- **Jackson 2.17.0** - JSON processing
+- **SQLite JDBC 3.45.1.0** - Database operations
+- **JavaMail 1.6.2** - Email notifications (free)
+- **OpenCV 4.9.0** - Image processing and object detection
+
+## Setup
+
+### 1. Notification Options (Choose One)
+
+#### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)
+
+Desktop notifications work out of the box on:
+- **Windows**: System tray notifications
+- **macOS**: Notification Center
+- **Linux**: Desktop environment notifications (GNOME, KDE, etc.)
+
+**No configuration required!** Just run with default settings:
+```bash
+export NOTIFICATION_CONFIG="desktop"
+# Or simply don't set it - desktop is the default
+```
+
+#### Option B: Desktop + Email Notifications 📧 (Free Gmail)
+
+1. Enable 2-Factor Authentication in your Google Account
+2. Go to: **Google Account → Security → 2-Step Verification → App passwords**
+3. Generate an app password for "Mail"
+4. Set environment variable:
+   ```bash
+   export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
+   ```
+
+**Format**: `smtp:username:app_password:recipient_email`
+
+**Example**:
+```bash
+export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"
+```
+
+**Note**: This is completely free using Gmail's SMTP server. No paid services required!
+
+### 2. OpenCV Native Libraries
+
+Download and install OpenCV native libraries for your platform:
+
+**Windows:**
+```bash
+# Download from https://opencv.org/releases/
+# Extract and add to PATH or use:
+java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar
+```
+
+**Linux:**
+```bash
+sudo apt-get install libopencv-dev
+```
+
+**macOS:**
+```bash
+brew install opencv
+```
+
+### 3. YOLO Model Files
+
+Download YOLO model files for object detection:
+
+```bash
+mkdir models
+cd models
+
+# Download YOLOv4 config
+wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
+
+# Download YOLOv4 weights (245 MB)
+wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
+
+# Download COCO class names
+wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
+```
+
+## Building
+
+```bash
+mvn clean package
+```
+
+This creates:
+- `target/troostwijk-scraper-1.0-SNAPSHOT.jar` - Regular JAR
+- `target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar` - Executable JAR with all dependencies
+
+## Running
+
+### Quick Start (Desktop Notifications Only)
+
+```bash
+java -Djava.library.path="/path/to/opencv/lib" \
+     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
+```
+
+### With Email Notifications
+
+```bash
+export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"
+
+java -Djava.library.path="/path/to/opencv/lib" \
+     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
+```
+
+### Using Maven
+
+```bash
+mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
+```
+
+## Project Structure
+
+```
+src/main/java/com/auction/scraper/
+├── TroostwijkScraper.java          # Main scraper class
+│   ├── Lot                          # Domain model for auction lots
+│   ├── DatabaseService              # SQLite operations
+│   ├── NotificationService          # Desktop + Email notifications (FREE)
+│   └── ObjectDetectionService       # OpenCV YOLO object detection
+└── Main.java                        # Entry point
+```
+
+## Configuration
+
+Edit `TroostwijkScraper.main()` to customize:
+
+- **Database file**: `troostwijk.db` (SQLite database location)
+- **YOLO paths**: Model configuration and weights files
+- **Monitoring frequency**: Default is every 1 hour
+- **Closing alerts**: Default is 5 minutes before closing
+
+## Database Schema
+
+The scraper creates three tables:
+
+**sales**
+- `sale_id` (PRIMARY KEY)
+- `title`, `location`, `closing_time`
+
+**lots**
+- `lot_id` (PRIMARY KEY)
+- `sale_id`, `title`, `description`, `manufacturer`, `type`, `year`
+- `category`, `current_bid`, `currency`, `url`
+- `closing_time`, `closing_notified`
+
+**images**
+- `id` (PRIMARY KEY)
+- `lot_id`, `url`, `file_path`, `labels` (detected objects)
+
+## Notification Examples
+
+### Desktop Notification
+![System Tray Notification]
+```
+🔔 Kavel bieding update
+Nieuw bod op kavel 12345: €150.00 (was €125.00)
+```
+
+### Email Notification
+```
+From: your.email@gmail.com
+To: your.email@gmail.com
+Subject: [Troostwijk] Kavel bieding update
+
+Nieuw bod op kavel 12345: €150.00 (was €125.00)
+```
+
+**High Priority Alerts** (closing soon):
+```
+⚠️ Lot nearing closure
+Kavel 12345 sluit binnen 5 min.
+```
+
+## Why This Approach?
+
+✅ **100% Free** - No paid services (Twilio, Pushover, etc.)
+✅ **No External Dependencies** - Desktop notifications built into Java
+✅ **Works Offline** - Desktop notifications don't need internet
+✅ **Privacy First** - Your data stays on your machine
+✅ **Cross-Platform** - Windows, macOS, Linux supported
+✅ **Optional Email** - Add Gmail notifications if you want
+
+## Troubleshooting
+
+### Desktop Notifications Not Showing
+
+- **Windows**: Check if Java has notification permissions
+- **Linux**: Ensure you have a desktop environment running (not headless)
+- **macOS**: Check System Preferences → Notifications
+
+### Email Not Sending
+
+1. Verify 2FA is enabled in Google Account
+2. Confirm you're using an **App Password** (not your regular Gmail password)
+3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
+4. Verify the SMTP format: `smtp:username:app_password:recipient`
+
+## Notes
+
+- Desktop notifications require a graphical environment (not headless servers)
+- For headless servers, use email-only notifications
+- Gmail SMTP is free and has generous limits (500 emails/day)
+- OpenCV native libraries must match your platform architecture
+- YOLO weights file is ~245 MB
+
+## License
+
+This is example code for educational purposes.