192 lines
4.7 KiB
Markdown
192 lines
4.7 KiB
Markdown
# Quick Start Guide
|
|
|
|
Get the scraper running in minutes without downloading YOLO models!
|
|
|
|
## Minimal Setup (No Object Detection)
|
|
|
|
The scraper works perfectly fine **without** YOLO object detection. You can run it immediately and add object detection later if needed.
|
|
|
|
### Step 1: Run the Scraper
|
|
|
|
```bash
|
|
# Using Maven
|
|
mvn clean compile exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
```
|
|
|
|
Or in IntelliJ IDEA:
|
|
1. Open `TroostwijkScraper.java`
|
|
2. Right-click on the `main` method
|
|
3. Select "Run 'TroostwijkScraper.main()'"
|
|
|
|
### What You'll See
|
|
|
|
```
|
|
=== Troostwijk Auction Scraper ===
|
|
|
|
Initializing scraper...
|
|
⚠️ Object detection disabled: YOLO model files not found
|
|
Expected files:
|
|
- models/yolov4.cfg
|
|
- models/yolov4.weights
|
|
- models/coco.names
|
|
Scraper will continue without image analysis.
|
|
|
|
[1/3] Discovering Dutch auctions...
|
|
✓ Found 5 auctions: [12345, 12346, 12347, 12348, 12349]
|
|
|
|
[2/3] Fetching lot details...
|
|
Processing sale 12345...
|
|
|
|
[3/3] Starting monitoring service...
|
|
✓ Monitoring active. Press Ctrl+C to stop.
|
|
```
|
|
|
|
### Step 2: Test Desktop Notifications
|
|
|
|
The scraper will automatically send desktop notifications when:
|
|
- A new bid is placed on a monitored lot
|
|
- An auction is closing within 5 minutes
|
|
|
|
**No setup required** - desktop notifications work out of the box!
|
|
|
|
---
|
|
|
|
## Optional: Add Email Notifications
|
|
|
|
If you want email notifications in addition to desktop notifications:
|
|
|
|
```bash
|
|
# Set environment variable
|
|
export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:app_password:your.email@gmail.com"
|
|
|
|
# Then run the scraper
|
|
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
```
|
|
|
|
**Get Gmail App Password:**
|
|
1. Enable 2FA in Google Account
|
|
2. Go to: Google Account → Security → 2-Step Verification → App passwords
|
|
3. Generate password for "Mail"
|
|
4. Use that password (not your regular Gmail password)
|
|
|
|
---
|
|
|
|
## Optional: Add Object Detection Later
|
|
|
|
If you want AI-powered image analysis to detect objects in auction photos:
|
|
|
|
### 1. Create models directory
|
|
```bash
|
|
mkdir models
|
|
cd models
|
|
```
|
|
|
|
### 2. Download YOLO files
|
|
```bash
|
|
# YOLOv4 config (small)
|
|
curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
|
|
|
|
# YOLOv4 weights (245 MB - takes a few minutes)
|
|
curl -LO https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
|
|
|
|
# COCO class names
|
|
curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
|
|
```
|
|
|
|
### 3. Run again
|
|
```bash
|
|
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
```
|
|
|
|
Now you'll see:
|
|
```
|
|
✓ Object detection enabled with YOLO
|
|
```
|
|
|
|
The scraper will now analyze auction images and detect objects like:
|
|
- Vehicles (cars, trucks, forklifts)
|
|
- Equipment (machines, tools)
|
|
- Furniture
|
|
- Electronics
|
|
- And 80+ other object types
|
|
|
|
---
|
|
|
|
## Features Without Object Detection
|
|
|
|
Even without YOLO, the scraper provides:
|
|
|
|
✅ **Full auction scraping** - Discovers all Dutch auctions
|
|
✅ **Lot tracking** - Monitors bids and closing times
|
|
✅ **Desktop notifications** - Real-time alerts
|
|
✅ **SQLite database** - All data persisted locally
|
|
✅ **Image downloading** - Saves all lot images
|
|
✅ **Scheduled monitoring** - Automatic updates every hour
|
|
|
|
Object detection simply adds:
|
|
- AI-powered image analysis
|
|
- Automatic object labeling
|
|
- Searchable image database
|
|
|
|
---
|
|
|
|
## Database Location
|
|
|
|
The scraper creates `troostwijk.db` in your current directory with:
|
|
- All auction data
|
|
- Lot details (title, description, bids, etc.)
|
|
- Downloaded image paths
|
|
- Object labels (if detection enabled)
|
|
|
|
View the database with any SQLite browser:
|
|
```bash
|
|
sqlite3 troostwijk.db
|
|
.tables
|
|
SELECT * FROM lots LIMIT 5;
|
|
```
|
|
|
|
---
|
|
|
|
## Stopping the Scraper
|
|
|
|
Press **Ctrl+C** to stop the monitoring service.
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. ✅ **Run the scraper** without YOLO to test it
|
|
2. ✅ **Verify desktop notifications** work
|
|
3. ⚙️ **Optional**: Add email notifications
|
|
4. ⚙️ **Optional**: Download YOLO models for object detection
|
|
5. 🔧 **Customize**: Edit monitoring frequency, closing alerts, etc.
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Desktop notifications not appearing?
|
|
- **Windows**: Check if Java has notification permissions
|
|
- **Linux**: Ensure desktop environment is running (not headless)
|
|
- **macOS**: Check System Preferences → Notifications
|
|
|
|
### OpenCV warnings?
|
|
These are normal and can be ignored:
|
|
```
|
|
WARNING: A restricted method in java.lang.System has been called
|
|
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid warning
|
|
```
|
|
|
|
The scraper works fine despite these warnings.
|
|
|
|
---
|
|
|
|
## Full Documentation
|
|
|
|
See [README.md](../README.md) for complete documentation including:
|
|
- Email setup details
|
|
- YOLO installation guide
|
|
- Configuration options
|
|
- Database schema
|
|
- API endpoints
|