# Troostwijk Auction Scraper
A Java-based web scraper for Dutch auctions on Troostwijk Auctions with **100% free** desktop/email notifications, SQLite persistence, and AI-powered object detection.
## Features
- **Auction Discovery**: Automatically discovers active Dutch auctions
- **Data Scraping**: Fetches detailed lot information via Troostwijk's JSON API
- **SQLite Storage**: Persists auction data, lots, images, and detected objects
- **Image Processing**: Downloads and analyzes lot images using OpenCV YOLO object detection
- **Free Notifications**: Real-time notifications when:
- Bids change on monitored lots
- Auctions are closing soon (within 5 minutes)
- Via desktop notifications (Windows/macOS/Linux system tray) β
- Optionally via email (Gmail SMTP - free) β
## Dependencies
All dependencies are managed via Maven (see `pom.xml`):
- **jsoup 1.17.2** - HTML parsing and HTTP client
- **Jackson 2.17.0** - JSON processing
- **SQLite JDBC 3.45.1.0** - Database operations
- **JavaMail 1.6.2** - Email notifications (free)
- **OpenCV 4.9.0** - Image processing and object detection
## Quick Start
### Development: Sync Production Data
To work with real production data locally:
```powershell
# Linux/Mac (Bash)
./scripts/sync-production-data.sh --db-only
```
See [scripts/README.md](scripts/README.md) for full documentation.
## Setup
### 1. Notification Options (Choose One)
#### Option A: Desktop Notifications Only β (Recommended - Zero Setup)
Desktop notifications work out of the box on:
- **Windows**: System tray notifications
- **macOS**: Notification Center
- **Linux**: Desktop environment notifications (GNOME, KDE, etc.)
**No configuration required!** Just run with default settings:
```bash
export NOTIFICATION_CONFIG="desktop"
# Or simply don't set it - desktop is the default
```
#### Option B: Desktop + Email Notifications π§ (Free Gmail)
1. Enable 2-Factor Authentication in your Google Account
2. Go to: **Google Account β Security β 2-Step Verification β App passwords**
3. Generate an app password for "Mail"
4. Set environment variable:
```bash
export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
```
**Format**: `smtp:username:app_password:recipient_email`
**Example**:
```bash
export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"
```
**Note**: This is completely free using Gmail's SMTP server. No paid services required!
### 2. OpenCV Native Libraries
Download and install OpenCV native libraries for your platform:
**Windows:**
```bash
# Download from https://opencv.org/releases/
# Extract and add to PATH or use:
java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar
```
**Linux:**
```bash
sudo apt-get install libopencv-dev
```
**macOS:**
```bash
brew install opencv
```
### 3. YOLO Model Files
Download YOLO model files for object detection:
```bash
mkdir /mnt/okcomputer/output/models
cd /mnt/okcomputer/output/models
# Download YOLOv4 config
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
# Download YOLOv4 weights (245 MB)
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
# Download COCO class names
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
```
## Building
```bash
mvn clean package
```
This creates:
- `../build/auctiora/auctiora-1.0-SNAPSHOT.jar` - Regular JAR
- `../build/auctiora/auctiora-1.0-SNAPSHOT-jar-with-dependencies.jar` - Executable JAR with all dependencies
## Running
### Quick Start (Desktop Notifications Only)
```bash
java -Djava.library.path="/path/to/opencv/lib" \
-jar ../build/auctiora/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
```
### With Email Notifications
```bash
export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"
java -Djava.library.path="/path/to/opencv/lib" \
-jar ../build/auctiora/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
```
### Using Maven
```bash
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
```
## System Architecture & Integration Flow
> **π Complete Integration Flowchart**: See [docs/INTEGRATION_FLOWCHART.md](docs/INTEGRATION_FLOWCHART.md) for the detailed intelligence integration diagram with GraphQL API fields, analytics, and dashboard features.
### Quick Overview
```
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β COMPLETE SYSTEM INTEGRATION DIAGRAM β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PHASE 1: EXTERNAL SCRAPER (Python/Playwright) - ARCHITECTURE-TROOSTWIJK β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββ
βΌ βΌ βΌ
[Listing Pages] [Auction Pages] [Lot Pages]
/auctions?page=N /a/auction-id /l/lot-id
β β β
β Extract URLs β Parse __NEXT_DATA__ β Parse __NEXT_DATA__
ββββββββββββββββββββββββββββββΆβ JSON β JSON
β β β
β βΌ βΌ
β ββββββββββββββββββ ββββββββββββββββββ
β β INSERT auctionsβ β INSERT lots β
β β to SQLite β β INSERT images β
β ββββββββββββββββββ β (URLs only) β
β β ββββββββββββββββββ
β β β
βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββ
βΌ
ββββββββββββββββββββ
β SQLITE DATABASE β
β output/cache.db β
ββββββββββββββββββββ
β
βββββββββββββββββββΌββββββββββββββββββ
βΌ βΌ βΌ
[auctions table] [lots table] [images table]
- auction_id - lot_id - id
- title - auction_id - lot_id
- location - title - url
- lots_count - current_bid - local_path
- closing_time - bid_count - downloaded=0
- closing_time
β
βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββ
β PHASE 2: MONITORING & PROCESSING (Java) - THIS PROJECT β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββΌββββββββββββββββββ
βΌ βΌ βΌ
[TroostwijkMonitor] [DatabaseService] [ScraperDataAdapter]
β β β
β Read lots β Query lots β Transform data
β every hour β Import images β TEXT β INTEGER
β β β "β¬123" β 123.0
βββββββββββββββββββ΄ββββββββββββββββββ
β
βββββββββββββββββββββββββββΌββββββββββββββββββββββββββ
βΌ βΌ βΌ
[Bid Monitoring] [Image Processing] [Closing Alerts]
Check API every 1h Download images Check < 5 min
β β β
β New bid? β Process via β Time critical?
ββ[YES]βββββββββββ β ObjectDetection ββ[YES]βββββ
β β β β β
βΌ β βΌ β β
[Update current_bid] β ββββββββββββββββββββ β β
in database β β YOLO Detection β β β
β β OpenCV DNN β β β
β ββββββββββββββββββββ β β
β β β β
β β Detect objects β β
β ββ[vehicle] β β
β ββ[furniture] β β
β ββ[machinery] β β
β β β β
β βΌ β β
β [Save labels to DB] β β
β [Estimate value] β β
β β β β
β β β β
βββββββββββ΄ββββββββββββββββββββββββ΄βββββββββββ
β
βββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββ
β PHASE 3: NOTIFICATION SYSTEM - USER INTERACTION TRIGGERS β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββ΄ββββββββββββββββββ
βΌ βΌ
[NotificationService] [User Decision Points]
β β
βββββββββββββββββββββΌββββββββββββββββββββ β
βΌ βΌ βΌ β
[Desktop Notify] [Email Notify] [Priority Level] β
Windows/macOS/ Gmail SMTP 0=Normal β
Linux system (FREE) 1=High β
tray β
β β β β
βββββββββββββββββββββ΄ββββββββββββββββββββ β
β β
βΌ βΌ
ββββββββββββββββββββ ββββββββββββββββββββ
β USER INTERACTION β β TRIGGER EVENTS: β
β NOTIFICATIONS β β β
ββββββββββββββββββββ ββββββββββββββββββββ
β β
βββββββββββββββββββββΌββββββββββββββββββββ β
βΌ βΌ βΌ β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ β
β 1. BID CHANGE β β 2. OBJECT β β 3. CLOSING β β
β β β DETECTED β β ALERT β β
β "Nieuw bod op β β β β β β
β kavel 12345: β β "Lot contains: β β "Kavel 12345 β β
β β¬150 (was β¬125)"β β - Vehicle β β sluit binnen β β
β β β - Machinery β β 5 min." β β
β Priority: NORMAL β β Est: β¬5000" β β Priority: HIGH β β
β β β β β β β
β Action needed: β β Action needed: β β Action needed: β β
β βΈ Place bid? β β βΈ Review item? β β βΈ Place final β β
β βΈ Monitor? β β βΈ Confirm value? β β bid? β β
β βΈ Ignore? β β βΈ Add to watch? β β βΈ Let expire? β β
ββββββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββββββ β
β β β β
βββββββββββββββββββββ΄ββββββββββββββββββββ΄ββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β USER ACTIONS & EXCEPTIONS β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Additional interaction points: β
β β
β 4. VIEWING DAY QUESTIONS β
β "Bezichtiging op [date] - kunt u aanwezig zijn?" β
β Action: βΈ Confirm attendance βΈ Request alternative βΈ Decline β
β β
β 5. ITEM RECOGNITION CONFIRMATION β
β "Detected: [object] - Is deze correcte identificatie?" β
β Action: βΈ Confirm βΈ Correct label βΈ Add notes β
β β
β 6. VALUE ESTIMATE APPROVAL β
β "Geschatte waarde: β¬X - Akkoord?" β
β Action: βΈ Accept βΈ Adjust βΈ Request manual review β
β β
β 7. EXCEPTION HANDLING β
β "Afwijkende sluitingstijd / locatiewijziging / special terms" β
β Action: βΈ Acknowledge βΈ Update preferences βΈ Withdraw interest β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β OBJECT DETECTION & VALUE ESTIMATION PIPELINE β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
[Downloaded Image] β [ImageProcessingService]
β β
β βΌ
β [ObjectDetectionService]
β β
β ββ Load YOLO model
β ββ Run inference (416x416)
β ββ Post-process detections
β β (confidence > 0.5)
β β
β βΌ
β ββββββββββββββββββββββββ
β β Detected Objects: β
β β - person β
β β - car β
β β - truck β
β β - furniture β
β β - machinery β
β β - electronics β
β β (80 COCO classes) β
β ββββββββββββββββββββββββ
β β
β βΌ
β [Value Estimation Logic]
β (Future enhancement)
β β
β ββ Match objects to auction categories
β ββ Historical price analysis
β ββ Condition assessment
β ββ Market trends
β β
β βΌ
β ββββββββββββββββββββββββ
β β Estimated Value: β
β β β¬X - β¬Y range β
β β Confidence: 75% β
β ββββββββββββββββββββββββ
β β
ββββββββββββββββββββββββ΄β [Save to DB]
β
βΌ
[Trigger notification if
value > threshold]
```
```mermaid
flowchart TD
subgraph P1["PHASE 1: EXTERNAL SCRAPER (Python/Playwright)"]
direction LR
A1[Listing Pages
/auctions?page=N] --> A2[Extract URLs]
B1[Auction Pages
/a/auction-id] --> B2[Parse __NEXT_DATA__ JSON]
C1[Lot Pages
/l/lot-id] --> C2[Parse __NEXT_DATA__ JSON]
A2 --> D1[INSERT auctions to SQLite]
B2 --> D1
C2 --> D2[INSERT lots & image URLs]
D1 --> DB[(SQLite Database
output/cache.db)]
D2 --> DB
end
DB --> P2_Entry
subgraph P2["PHASE 2: MONITORING & PROCESSING (Java)"]
direction TB
P2_Entry[Data Ready] --> Monitor[TroostwijkMonitor
Read lots every hour]
P2_Entry --> DBService[DatabaseService
Query & Import]
P2_Entry --> Adapter[ScraperDataAdapter
Transform TEXT β INTEGER]
Monitor --> BM[Bid Monitoring
Check API every 1h]
DBService --> IP[Image Processing
Download & Analyze]
Adapter --> DataForNotify[Formatted Data]
BM --> BidUpdate{New bid?}
BidUpdate -->|Yes| UpdateDB[Update current_bid in DB]
UpdateDB --> NotifyTrigger1
IP --> Detection[Object Detection
YOLO/OpenCV DNN]
Detection --> ObjectCheck{Detect objects?}
ObjectCheck -->|Vehicle| Save1[Save labels & estimate value]
ObjectCheck -->|Furniture| Save2[Save labels & estimate value]
ObjectCheck -->|Machinery| Save3[Save labels & estimate value]
Save1 --> NotifyTrigger2
Save2 --> NotifyTrigger2
Save3 --> NotifyTrigger2
CA[Closing Alerts
Check < 5 min] --> TimeCheck{Time critical?}
TimeCheck -->|Yes| NotifyTrigger3
end
NotifyTrigger1 --> NS
NotifyTrigger2 --> NS
NotifyTrigger3 --> NS
subgraph P3["PHASE 3: NOTIFICATION SYSTEM"]
NS[NotificationService] --> DN[Desktop Notify
Windows/macOS/Linux]
NS --> EN[Email Notify
Gmail SMTP]
NS --> PL[Set Priority Level
0=Normal, 1=High]
end
DN --> UI[User Interaction & Decisions]
EN --> UI
PL --> UI
subgraph UI_Details[User Decision Points / Trigger Events]
direction LR
E1["1. BID CHANGE
'Nieuw bod op kavel 12345...'
Actions: Place bid? Monitor? Ignore?"]
E2["2. OBJECT DETECTED
'Lot contains: Vehicle...'
Actions: Review? Confirm value?"]
E3["3. CLOSING ALERT
'Kavel 12345 sluit binnen 5 min.'
Actions: Place final bid? Let expire?"]
E4["4. VIEWING DAY QUESTIONS
'Bezichtiging op [date]...'"]
E5["5. ITEM RECOGNITION CONFIRMATION
'Detected: [object]...'"]
E6["6. VALUE ESTIMATE APPROVAL
'Geschatte waarde: β¬X...'"]
E7["7. EXCEPTION HANDLING
'Afwijkende sluitingstijd...'"]
end
UI --> UI_Details
%% Object Detection Sub-Flow Detail
subgraph P2_Detail["Object Detection & Value Estimation Pipeline"]
direction LR
DI[Downloaded Image] --> IPS[ImageProcessingService]
IPS --> ODS[ObjectDetectionService]
ODS --> Load[Load YOLO model]
ODS --> Run[Run inference]
ODS --> Post[Post-process detections
confidence > 0.5]
Post --> ObjList["Detected Objects List
(80 COCO classes)"]
ObjList --> VEL[Value Estimation Logic
Future enhancement]
VEL --> Match[Match to categories]
VEL --> History[Historical price analysis]
VEL --> Condition[Condition assessment]
VEL --> Market[Market trends]
Market --> ValueEst["Estimated Value Range
Confidence: 75%"]
ValueEst --> SaveToDB[Save to Database]
SaveToDB --> TriggerNotify{Value > threshold?}
end
IP -.-> P2_Detail
TriggerNotify -.-> NotifyTrigger2
```
## Integration Hooks & Timing
| Event | Frequency | Trigger | Notification Type | User Action Required |
|--------------------------------|-------------------|----------------------------|----------------------------|------------------------|
| **New auction discovered** | On scrape | Scraper finds new auction | Desktop + Email (optional) | Review auction |
| **Bid change detected** | Every 1 hour | Monitor detects higher bid | Desktop + Email | Place counter-bid? |
| **Closing soon (< 30 min)** | When detected | Time-based check | Desktop + Email | Review lot |
| **Closing imminent (< 5 min)** | When detected | Time-based check | Desktop + Email (HIGH) | Final bid decision |
| **Object detected** | On image process | YOLO finds objects | Desktop + Email | Confirm identification |
| **Value estimated** | After detection | Estimation complete | Desktop + Email | Approve estimate |
| **Viewing day scheduled** | From lot metadata | Scraper extracts date | Desktop + Email | Confirm attendance |
| **Exception/Change** | On update | Scraper detects change | Desktop + Email (HIGH) | Acknowledge |
## Project Structure
```
src/main/java/com/auction/
βββ Main.java # Entry point
βββ TroostwijkMonitor.java # Monitoring & orchestration
βββ DatabaseService.java # SQLite operations
βββ ScraperDataAdapter.java # Schema translation (TEXTβINT, β¬βfloat)
βββ ImageProcessingService.java # Downloads & processes images
βββ ObjectDetectionService.java # OpenCV YOLO detection
βββ NotificationService.java # Desktop + Email notifications (FREE)
βββ Lot.java # Domain model for auction lots
βββ AuctionInfo.java # Domain model for auctions
βββ Console.java # Logging utility
```
## Configuration
Edit `TroostwijkScraper.main()` to customize:
- **Database file**: `troostwijk.db` (SQLite database location)
- **YOLO paths**: Model configuration and weights files
- **Monitoring frequency**: Default is every 1 hour
- **Closing alerts**: Default is 5 minutes before closing
## Database Schema
The scraper creates three tables:
**sales**
- `sale_id` (PRIMARY KEY)
- `title`, `location`, `closing_time`
**lots**
- `lot_id` (PRIMARY KEY)
- `sale_id`, `title`, `description`, `manufacturer`, `type`, `year`
- `category`, `current_bid`, `currency`, `url`
- `closing_time`, `closing_notified`
**images**
- `id` (PRIMARY KEY)
- `lot_id`, `url`, `local_path`, `labels` (detected objects)
## Notification Examples
### Desktop Notification
![System Tray Notification]
```
π Kavel bieding update
Nieuw bod op kavel 12345: β¬150.00 (was β¬125.00)
```
### Email Notification
```
From: your.email@gmail.com
To: your.email@gmail.com
Subject: [Troostwijk] Kavel bieding update
Nieuw bod op kavel 12345: β¬150.00 (was β¬125.00)
```
**High Priority Alerts** (closing soon):
```
β οΈ Lot nearing closure
Kavel 12345 sluit binnen 5 min.
```
## Why This Approach?
β
**100% Free** - No paid services (Twilio, Pushover, etc.)
β
**No External Dependencies** - Desktop notifications built into Java
β
**Works Offline** - Desktop notifications don't need internet
β
**Privacy First** - Your data stays on your machine
β
**Cross-Platform** - Windows, macOS, Linux supported
β
**Optional Email** - Add Gmail notifications if you want
## Troubleshooting
### Desktop Notifications Not Showing
- **Windows**: Check if Java has notification permissions
- **Linux**: Ensure you have a desktop environment running (not headless)
- **macOS**: Check System Preferences β Notifications
### Email Not Sending
1. Verify 2FA is enabled in Google Account
2. Confirm you're using an **App Password** (not your regular Gmail password)
3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
4. Verify the SMTP format: `smtp:username:app_password:recipient`
## Notes
- Desktop notifications require a graphical environment (not headless servers)
- For headless servers, use email-only notifications
- Gmail SMTP is free and has generous limits (500 emails/day)
- OpenCV native libraries must match your platform architecture
- YOLO weights file is ~245 MB
```shell
ssh tour@athena.lan "docker run --rm -v shared-auction-data:/data -v /tmp:/tmp alpine cp /data/cache.db /tmp/cache.db" && scp tour@athena.lan:/tmp/cache.db c:/mnt/okcomputer/cache.db
```
## License
This is example code for educational purposes.