diff --git a/QUICKSTART.md b/QUICKSTART.md new file mode 100644 index 0000000..572e887 --- /dev/null +++ b/QUICKSTART.md @@ -0,0 +1,191 @@ +# Quick Start Guide + +Get the scraper running in minutes without downloading YOLO models! + +## Minimal Setup (No Object Detection) + +The scraper works perfectly fine **without** YOLO object detection. You can run it immediately and add object detection later if needed. + +### Step 1: Run the Scraper + +```bash +# Using Maven +mvn clean compile exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper" +``` + +Or in IntelliJ IDEA: +1. Open `TroostwijkScraper.java` +2. Right-click on the `main` method +3. Select "Run 'TroostwijkScraper.main()'" + +### What You'll See + +``` +=== Troostwijk Auction Scraper === + +Initializing scraper... +⚠️ Object detection disabled: YOLO model files not found + Expected files: + - models/yolov4.cfg + - models/yolov4.weights + - models/coco.names + Scraper will continue without image analysis. + +[1/3] Discovering Dutch auctions... +✓ Found 5 auctions: [12345, 12346, 12347, 12348, 12349] + +[2/3] Fetching lot details... + Processing sale 12345... + +[3/3] Starting monitoring service... +✓ Monitoring active. Press Ctrl+C to stop. +``` + +### Step 2: Test Desktop Notifications + +The scraper will automatically send desktop notifications when: +- A new bid is placed on a monitored lot +- An auction is closing within 5 minutes + +**No setup required** - desktop notifications work out of the box! + +--- + +## Optional: Add Email Notifications + +If you want email notifications in addition to desktop notifications: + +```bash +# Set environment variable +export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:app_password:your.email@gmail.com" + +# Then run the scraper +mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper" +``` + +**Get Gmail App Password:** +1. Enable 2FA in Google Account +2. Go to: Google Account → Security → 2-Step Verification → App passwords +3. Generate password for "Mail" +4. Use that password (not your regular Gmail password) + +--- + +## Optional: Add Object Detection Later + +If you want AI-powered image analysis to detect objects in auction photos: + +### 1. Create models directory +```bash +mkdir models +cd models +``` + +### 2. Download YOLO files +```bash +# YOLOv4 config (small) +curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg + +# YOLOv4 weights (245 MB - takes a few minutes) +curl -LO https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights + +# COCO class names +curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names +``` + +### 3. Run again +```bash +mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper" +``` + +Now you'll see: +``` +✓ Object detection enabled with YOLO +``` + +The scraper will now analyze auction images and detect objects like: +- Vehicles (cars, trucks, forklifts) +- Equipment (machines, tools) +- Furniture +- Electronics +- And 80+ other object types + +--- + +## Features Without Object Detection + +Even without YOLO, the scraper provides: + +✅ **Full auction scraping** - Discovers all Dutch auctions +✅ **Lot tracking** - Monitors bids and closing times +✅ **Desktop notifications** - Real-time alerts +✅ **SQLite database** - All data persisted locally +✅ **Image downloading** - Saves all lot images +✅ **Scheduled monitoring** - Automatic updates every hour + +Object detection simply adds: +- AI-powered image analysis +- Automatic object labeling +- Searchable image database + +--- + +## Database Location + +The scraper creates `troostwijk.db` in your current directory with: +- All auction data +- Lot details (title, description, bids, etc.) +- Downloaded image paths +- Object labels (if detection enabled) + +View the database with any SQLite browser: +```bash +sqlite3 troostwijk.db +.tables +SELECT * FROM lots LIMIT 5; +``` + +--- + +## Stopping the Scraper + +Press **Ctrl+C** to stop the monitoring service. + +--- + +## Next Steps + +1. ✅ **Run the scraper** without YOLO to test it +2. ✅ **Verify desktop notifications** work +3. ⚙️ **Optional**: Add email notifications +4. ⚙️ **Optional**: Download YOLO models for object detection +5. 🔧 **Customize**: Edit monitoring frequency, closing alerts, etc. + +--- + +## Troubleshooting + +### Desktop notifications not appearing? +- **Windows**: Check if Java has notification permissions +- **Linux**: Ensure desktop environment is running (not headless) +- **macOS**: Check System Preferences → Notifications + +### OpenCV warnings? +These are normal and can be ignored: +``` +WARNING: A restricted method in java.lang.System has been called +WARNING: Use --enable-native-access=ALL-UNNAMED to avoid warning +``` + +The scraper works fine despite these warnings. + +--- + +## Full Documentation + +See [README.md](README.md) for complete documentation including: +- Email setup details +- YOLO installation guide +- Configuration options +- Database schema +- API endpoints diff --git a/README.md b/README.md new file mode 100644 index 0000000..e7f1ffa --- /dev/null +++ b/README.md @@ -0,0 +1,233 @@ +# Troostwijk Auction Scraper + +A Java-based web scraper for Dutch auctions on Troostwijk Auctions with **100% free** desktop/email notifications, SQLite persistence, and AI-powered object detection. + +## Features + +- **Auction Discovery**: Automatically discovers active Dutch auctions +- **Data Scraping**: Fetches detailed lot information via Troostwijk's JSON API +- **SQLite Storage**: Persists auction data, lots, images, and detected objects +- **Image Processing**: Downloads and analyzes lot images using OpenCV YOLO object detection +- **Free Notifications**: Real-time notifications when: + - Bids change on monitored lots + - Auctions are closing soon (within 5 minutes) + - Via desktop notifications (Windows/macOS/Linux system tray) ✅ + - Optionally via email (Gmail SMTP - free) ✅ + +## Dependencies + +All dependencies are managed via Maven (see `pom.xml`): + +- **jsoup 1.17.2** - HTML parsing and HTTP client +- **Jackson 2.17.0** - JSON processing +- **SQLite JDBC 3.45.1.0** - Database operations +- **JavaMail 1.6.2** - Email notifications (free) +- **OpenCV 4.9.0** - Image processing and object detection + +## Setup + +### 1. Notification Options (Choose One) + +#### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup) + +Desktop notifications work out of the box on: +- **Windows**: System tray notifications +- **macOS**: Notification Center +- **Linux**: Desktop environment notifications (GNOME, KDE, etc.) + +**No configuration required!** Just run with default settings: +```bash +export NOTIFICATION_CONFIG="desktop" +# Or simply don't set it - desktop is the default +``` + +#### Option B: Desktop + Email Notifications 📧 (Free Gmail) + +1. Enable 2-Factor Authentication in your Google Account +2. Go to: **Google Account → Security → 2-Step Verification → App passwords** +3. Generate an app password for "Mail" +4. Set environment variable: + ```bash + export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com" + ``` + +**Format**: `smtp:username:app_password:recipient_email` + +**Example**: +```bash +export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com" +``` + +**Note**: This is completely free using Gmail's SMTP server. No paid services required! + +### 2. OpenCV Native Libraries + +Download and install OpenCV native libraries for your platform: + +**Windows:** +```bash +# Download from https://opencv.org/releases/ +# Extract and add to PATH or use: +java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar +``` + +**Linux:** +```bash +sudo apt-get install libopencv-dev +``` + +**macOS:** +```bash +brew install opencv +``` + +### 3. YOLO Model Files + +Download YOLO model files for object detection: + +```bash +mkdir models +cd models + +# Download YOLOv4 config +wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg + +# Download YOLOv4 weights (245 MB) +wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights + +# Download COCO class names +wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names +``` + +## Building + +```bash +mvn clean package +``` + +This creates: +- `target/troostwijk-scraper-1.0-SNAPSHOT.jar` - Regular JAR +- `target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar` - Executable JAR with all dependencies + +## Running + +### Quick Start (Desktop Notifications Only) + +```bash +java -Djava.library.path="/path/to/opencv/lib" \ + -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar +``` + +### With Email Notifications + +```bash +export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com" + +java -Djava.library.path="/path/to/opencv/lib" \ + -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar +``` + +### Using Maven + +```bash +mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper" +``` + +## Project Structure + +``` +src/main/java/com/auction/scraper/ +├── TroostwijkScraper.java # Main scraper class +│ ├── Lot # Domain model for auction lots +│ ├── DatabaseService # SQLite operations +│ ├── NotificationService # Desktop + Email notifications (FREE) +│ └── ObjectDetectionService # OpenCV YOLO object detection +└── Main.java # Entry point +``` + +## Configuration + +Edit `TroostwijkScraper.main()` to customize: + +- **Database file**: `troostwijk.db` (SQLite database location) +- **YOLO paths**: Model configuration and weights files +- **Monitoring frequency**: Default is every 1 hour +- **Closing alerts**: Default is 5 minutes before closing + +## Database Schema + +The scraper creates three tables: + +**sales** +- `sale_id` (PRIMARY KEY) +- `title`, `location`, `closing_time` + +**lots** +- `lot_id` (PRIMARY KEY) +- `sale_id`, `title`, `description`, `manufacturer`, `type`, `year` +- `category`, `current_bid`, `currency`, `url` +- `closing_time`, `closing_notified` + +**images** +- `id` (PRIMARY KEY) +- `lot_id`, `url`, `file_path`, `labels` (detected objects) + +## Notification Examples + +### Desktop Notification +![System Tray Notification] +``` +🔔 Kavel bieding update +Nieuw bod op kavel 12345: €150.00 (was €125.00) +``` + +### Email Notification +``` +From: your.email@gmail.com +To: your.email@gmail.com +Subject: [Troostwijk] Kavel bieding update + +Nieuw bod op kavel 12345: €150.00 (was €125.00) +``` + +**High Priority Alerts** (closing soon): +``` +⚠️ Lot nearing closure +Kavel 12345 sluit binnen 5 min. +``` + +## Why This Approach? + +✅ **100% Free** - No paid services (Twilio, Pushover, etc.) +✅ **No External Dependencies** - Desktop notifications built into Java +✅ **Works Offline** - Desktop notifications don't need internet +✅ **Privacy First** - Your data stays on your machine +✅ **Cross-Platform** - Windows, macOS, Linux supported +✅ **Optional Email** - Add Gmail notifications if you want + +## Troubleshooting + +### Desktop Notifications Not Showing + +- **Windows**: Check if Java has notification permissions +- **Linux**: Ensure you have a desktop environment running (not headless) +- **macOS**: Check System Preferences → Notifications + +### Email Not Sending + +1. Verify 2FA is enabled in Google Account +2. Confirm you're using an **App Password** (not your regular Gmail password) +3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA) +4. Verify the SMTP format: `smtp:username:app_password:recipient` + +## Notes + +- Desktop notifications require a graphical environment (not headless servers) +- For headless servers, use email-only notifications +- Gmail SMTP is free and has generous limits (500 emails/day) +- OpenCV native libraries must match your platform architecture +- YOLO weights file is ~245 MB + +## License + +This is example code for educational purposes. diff --git a/models/coco.names b/models/coco.names new file mode 100644 index 0000000..ca76c80 --- /dev/null +++ b/models/coco.names @@ -0,0 +1,80 @@ +person +bicycle +car +motorbike +aeroplane +bus +train +truck +boat +traffic light +fire hydrant +stop sign +parking meter +bench +bird +cat +dog +horse +sheep +cow +elephant +bear +zebra +giraffe +backpack +umbrella +handbag +tie +suitcase +frisbee +skis +snowboard +sports ball +kite +baseball bat +baseball glove +skateboard +surfboard +tennis racket +bottle +wine glass +cup +fork +knife +spoon +bowl +banana +apple +sandwich +orange +broccoli +carrot +hot dog +pizza +donut +cake +chair +sofa +pottedplant +bed +diningtable +toilet +tvmonitor +laptop +mouse +remote +keyboard +cell phone +microwave +oven +toaster +sink +refrigerator +book +clock +vase +scissors +teddy bear +hair drier +toothbrush diff --git a/models/yolov4.cfg b/models/yolov4.cfg new file mode 100644 index 0000000..a7be12b --- /dev/null +++ b/models/yolov4.cfg @@ -0,0 +1,1158 @@ +[net] +batch=64 +subdivisions=8 +# Training +#width=512 +#height=512 +width=608 +height=608 +channels=3 +momentum=0.949 +decay=0.0005 +angle=0 +saturation = 1.5 +exposure = 1.5 +hue=.1 + +learning_rate=0.0013 +burn_in=1000 +max_batches = 500500 +policy=steps +steps=400000,450000 +scales=.1,.1 + +#cutmix=1 +mosaic=1 + +#:104x104 54:52x52 85:26x26 104:13x13 for 416 + +[convolutional] +batch_normalize=1 +filters=32 +size=3 +stride=1 +pad=1 +activation=mish + +# Downsample + +[convolutional] +batch_normalize=1 +filters=64 +size=3 +stride=2 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -2 + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=32 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -1,-7 + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +# Downsample + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=2 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -2 + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=64 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=64 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -1,-10 + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +# Downsample + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=2 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -2 + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=128 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -1,-28 + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +# Downsample + +[convolutional] +batch_normalize=1 +filters=512 +size=3 +stride=2 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -2 + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=256 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -1,-28 + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +# Downsample + +[convolutional] +batch_normalize=1 +filters=1024 +size=3 +stride=2 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -2 + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[convolutional] +batch_normalize=1 +filters=512 +size=3 +stride=1 +pad=1 +activation=mish + +[shortcut] +from=-3 +activation=linear + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=mish + +[route] +layers = -1,-16 + +[convolutional] +batch_normalize=1 +filters=1024 +size=1 +stride=1 +pad=1 +activation=mish + +########################## + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=1024 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +### SPP ### +[maxpool] +stride=1 +size=5 + +[route] +layers=-2 + +[maxpool] +stride=1 +size=9 + +[route] +layers=-4 + +[maxpool] +stride=1 +size=13 + +[route] +layers=-1,-3,-5,-6 +### End SPP ### + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=1024 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[upsample] +stride=2 + +[route] +layers = 85 + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[route] +layers = -1, -3 + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=512 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=512 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=leaky + +[upsample] +stride=2 + +[route] +layers = 54 + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=leaky + +[route] +layers = -1, -3 + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=256 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=256 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=128 +size=1 +stride=1 +pad=1 +activation=leaky + +########################## + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=256 +activation=leaky + +[convolutional] +size=1 +stride=1 +pad=1 +filters=255 +activation=linear + + +[yolo] +mask = 0,1,2 +anchors = 12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401 +classes=80 +num=9 +jitter=.3 +ignore_thresh = .7 +truth_thresh = 1 +scale_x_y = 1.2 +iou_thresh=0.213 +cls_normalizer=1.0 +iou_normalizer=0.07 +iou_loss=ciou +nms_kind=greedynms +beta_nms=0.6 +max_delta=5 + + +[route] +layers = -4 + +[convolutional] +batch_normalize=1 +size=3 +stride=2 +pad=1 +filters=256 +activation=leaky + +[route] +layers = -1, -16 + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=512 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=512 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=256 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=512 +activation=leaky + +[convolutional] +size=1 +stride=1 +pad=1 +filters=255 +activation=linear + + +[yolo] +mask = 3,4,5 +anchors = 12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401 +classes=80 +num=9 +jitter=.3 +ignore_thresh = .7 +truth_thresh = 1 +scale_x_y = 1.1 +iou_thresh=0.213 +cls_normalizer=1.0 +iou_normalizer=0.07 +iou_loss=ciou +nms_kind=greedynms +beta_nms=0.6 +max_delta=5 + + +[route] +layers = -4 + +[convolutional] +batch_normalize=1 +size=3 +stride=2 +pad=1 +filters=512 +activation=leaky + +[route] +layers = -1, -37 + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=1024 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=1024 +activation=leaky + +[convolutional] +batch_normalize=1 +filters=512 +size=1 +stride=1 +pad=1 +activation=leaky + +[convolutional] +batch_normalize=1 +size=3 +stride=1 +pad=1 +filters=1024 +activation=leaky + +[convolutional] +size=1 +stride=1 +pad=1 +filters=255 +activation=linear + + +[yolo] +mask = 6,7,8 +anchors = 12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401 +classes=80 +num=9 +jitter=.3 +ignore_thresh = .7 +truth_thresh = 1 +random=1 +scale_x_y = 1.05 +iou_thresh=0.213 +cls_normalizer=1.0 +iou_normalizer=0.07 +iou_loss=ciou +nms_kind=greedynms +beta_nms=0.6 +max_delta=5 + diff --git a/models/yolov4.weights b/models/yolov4.weights new file mode 100644 index 0000000..921f440 Binary files /dev/null and b/models/yolov4.weights differ diff --git a/src/main/java/com/auction/scraper/TroostwijkScraper.java b/src/main/java/com/auction/scraper/TroostwijkScraper.java index b5ad33f..6d64fd8 100644 --- a/src/main/java/com/auction/scraper/TroostwijkScraper.java +++ b/src/main/java/com/auction/scraper/TroostwijkScraper.java @@ -354,6 +354,8 @@ public class TroostwijkScraper { * discovers Dutch auctions, scrapes lots, and begins monitoring. */ public static void main(String[] args) throws Exception { + System.out.println("=== Troostwijk Auction Scraper ===\n"); + // Configuration parameters (replace with your own values) String databaseFile = "troostwijk.db"; @@ -366,27 +368,34 @@ public class TroostwijkScraper { // Example: "smtp:your.email@gmail.com:abcd1234efgh5678:recipient@example.com" // Get app password: Google Account > Security > 2-Step Verification > App passwords - String yoloCfg = "models/yolov4.cfg"; // path to YOLO config file - String yoloWeights = "models/yolov4.weights"; // path to YOLO weights file - String yoloClasses = "models/coco.names"; // list of class names + // YOLO model paths (optional - scraper works without object detection) + String yoloCfg = "models/yolov4.cfg"; + String yoloWeights = "models/yolov4.weights"; + String yoloClasses = "models/coco.names"; // Load native OpenCV library System.loadLibrary(Core.NATIVE_LIBRARY_NAME); + System.out.println("Initializing scraper..."); TroostwijkScraper scraper = new TroostwijkScraper(databaseFile, notificationConfig, "", yoloCfg, yoloWeights, yoloClasses); // Step 1: Discover auctions in NL + System.out.println("\n[1/3] Discovering Dutch auctions..."); List auctions = scraper.discoverDutchAuctions(); - System.out.println("Found auctions: " + auctions); + System.out.println("✓ Found " + auctions.size() + " auctions: " + auctions); // Step 2: Fetch lots for each auction + System.out.println("\n[2/3] Fetching lot details..."); for (int saleId : auctions) { + System.out.println(" Processing sale " + saleId + "..."); scraper.fetchLotsForSale(saleId); } // Step 3: Start monitoring bids and closures + System.out.println("\n[3/3] Starting monitoring service..."); scraper.scheduleMonitoring(); + System.out.println("✓ Monitoring active. Press Ctrl+C to stop.\n"); } // ---------------------------------------------------------------------- @@ -710,23 +719,53 @@ public class TroostwijkScraper { } /** - * Service for performing object detection on images using OpenCV’s DNN + * Service for performing object detection on images using OpenCV's DNN * module. The DNN module can load pre‑trained models from several * frameworks (Darknet, TensorFlow, ONNX, etc.)【784097309529506†L209-L233】. Here * we load a YOLO model (Darknet) by specifying the configuration and * weights files. For each image we run a forward pass and return a * list of detected class labels. + * + * If model files are not found, the service operates in disabled mode + * and returns empty lists. */ static class ObjectDetectionService { private final Net net; private final List classNames; + private final boolean enabled; + ObjectDetectionService(String cfgPath, String weightsPath, String classNamesPath) throws IOException { - // Load network - this.net = Dnn.readNetFromDarknet(cfgPath, weightsPath); - this.net.setPreferableBackend(DNN_BACKEND_OPENCV); - this.net.setPreferableTarget(DNN_TARGET_CPU); - // Load class names (one per line) - this.classNames = Files.readAllLines(Paths.get(classNamesPath)); + // Check if model files exist + Path cfgFile = Paths.get(cfgPath); + Path weightsFile = Paths.get(weightsPath); + Path classNamesFile = Paths.get(classNamesPath); + + if (!Files.exists(cfgFile) || !Files.exists(weightsFile) || !Files.exists(classNamesFile)) { + System.out.println("⚠️ Object detection disabled: YOLO model files not found"); + System.out.println(" Expected files:"); + System.out.println(" - " + cfgPath); + System.out.println(" - " + weightsPath); + System.out.println(" - " + classNamesPath); + System.out.println(" Scraper will continue without image analysis."); + this.enabled = false; + this.net = null; + this.classNames = new ArrayList<>(); + return; + } + + try { + // Load network + this.net = Dnn.readNetFromDarknet(cfgPath, weightsPath); + this.net.setPreferableBackend(DNN_BACKEND_OPENCV); + this.net.setPreferableTarget(DNN_TARGET_CPU); + // Load class names (one per line) + this.classNames = Files.readAllLines(classNamesFile); + this.enabled = true; + System.out.println("✓ Object detection enabled with YOLO"); + } catch (Exception e) { + System.err.println("⚠️ Object detection disabled: " + e.getMessage()); + throw new IOException("Failed to initialize object detection", e); + } } /** * Detects objects in the given image file and returns a list of @@ -736,9 +775,13 @@ public class TroostwijkScraper { * post‑processing【784097309529506†L324-L344】. * * @param imagePath absolute path to the image - * @return list of detected class names + * @return list of detected class names (empty if detection disabled) */ List detectObjects(String imagePath) { + if (!enabled) { + return new ArrayList<>(); + } + List labels = new ArrayList<>(); Mat image = Imgcodecs.imread(imagePath); if (image.empty()) return labels; diff --git a/target/classes/com/auction/scraper/Main.class b/target/classes/com/auction/scraper/Main.class new file mode 100644 index 0000000..d400324 Binary files /dev/null and b/target/classes/com/auction/scraper/Main.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper$DatabaseService.class b/target/classes/com/auction/scraper/TroostwijkScraper$DatabaseService.class new file mode 100644 index 0000000..7069d86 Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper$DatabaseService.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper$Lot.class b/target/classes/com/auction/scraper/TroostwijkScraper$Lot.class new file mode 100644 index 0000000..9af6273 Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper$Lot.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService$1.class b/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService$1.class new file mode 100644 index 0000000..3ebb139 Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService$1.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService.class b/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService.class new file mode 100644 index 0000000..526cd75 Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper$NotificationService.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper$ObjectDetectionService.class b/target/classes/com/auction/scraper/TroostwijkScraper$ObjectDetectionService.class new file mode 100644 index 0000000..f82c6a4 Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper$ObjectDetectionService.class differ diff --git a/target/classes/com/auction/scraper/TroostwijkScraper.class b/target/classes/com/auction/scraper/TroostwijkScraper.class new file mode 100644 index 0000000..c04fa8f Binary files /dev/null and b/target/classes/com/auction/scraper/TroostwijkScraper.class differ diff --git a/troostwijk.db b/troostwijk.db new file mode 100644 index 0000000..729611b Binary files /dev/null and b/troostwijk.db differ