# Troostwijk Auction Scraper

A Java-based web scraper for Dutch auctions on Troostwijk Auctions with **100% free** desktop/email notifications, SQLite persistence, and AI-powered object detection.

## Features

- **Auction Discovery**: Automatically discovers active Dutch auctions
- **Data Scraping**: Fetches detailed lot information via Troostwijk's JSON API
- **SQLite Storage**: Persists auction data, lots, images, and detected objects
- **Image Processing**: Downloads and analyzes lot images using OpenCV YOLO object detection
- **Free Notifications**: Real-time notifications when:
  - Bids change on monitored lots
  - Auctions are closing soon (within 5 minutes)
  - Via desktop notifications (Windows/macOS/Linux system tray) ✅
  - Optionally via email (Gmail SMTP - free) ✅

## Dependencies

All dependencies are managed via Maven (see `pom.xml`):

- **jsoup 1.17.2** - HTML parsing and HTTP client
- **Jackson 2.17.0** - JSON processing
- **SQLite JDBC 3.45.1.0** - Database operations
- **JavaMail 1.6.2** - Email notifications (free)
- **OpenCV 4.9.0** - Image processing and object detection

## Setup

### 1. Notification Options (Choose One)

#### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)

Desktop notifications work out of the box on:
- **Windows**: System tray notifications
- **macOS**: Notification Center
- **Linux**: Desktop environment notifications (GNOME, KDE, etc.)

**No configuration required!** Just run with default settings:
```bash
export NOTIFICATION_CONFIG="desktop"
# Or simply don't set it - desktop is the default
```

#### Option B: Desktop + Email Notifications 📧 (Free Gmail)

1. Enable 2-Factor Authentication in your Google Account
2. Go to: **Google Account → Security → 2-Step Verification → App passwords**
3. Generate an app password for "Mail"
4. Set environment variable:
   ```bash
   export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
   ```

**Format**: `smtp:username:app_password:recipient_email`

**Example**:
```bash
export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"
```

**Note**: This is completely free using Gmail's SMTP server. No paid services required!

### 2. OpenCV Native Libraries

Download and install OpenCV native libraries for your platform:

**Windows:**
```bash
# Download from https://opencv.org/releases/
# Extract and add to PATH or use:
java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar
```

**Linux:**
```bash
sudo apt-get install libopencv-dev
```

**macOS:**
```bash
brew install opencv
```

### 3. YOLO Model Files

Download YOLO model files for object detection:

```bash
mkdir models
cd models

# Download YOLOv4 config
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg

# Download YOLOv4 weights (245 MB)
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights

# Download COCO class names
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
```

## Building

```bash
mvn clean package
```

This creates:
- `target/troostwijk-scraper-1.0-SNAPSHOT.jar` - Regular JAR
- `target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar` - Executable JAR with all dependencies

## Running

### Quick Start (Desktop Notifications Only)

```bash
java -Djava.library.path="/path/to/opencv/lib" \
     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
```

### With Email Notifications

```bash
export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"

java -Djava.library.path="/path/to/opencv/lib" \
     -jar target/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
```

### Using Maven

```bash
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
```

## Project Structure

```
src/main/java/com/auction/scraper/
├── TroostwijkScraper.java          # Main scraper class
│   ├── Lot                          # Domain model for auction lots
│   ├── DatabaseService              # SQLite operations
│   ├── NotificationService          # Desktop + Email notifications (FREE)
│   └── ObjectDetectionService       # OpenCV YOLO object detection
└── Main.java                        # Entry point
```

## Configuration

Edit `TroostwijkScraper.main()` to customize:

- **Database file**: `troostwijk.db` (SQLite database location)
- **YOLO paths**: Model configuration and weights files
- **Monitoring frequency**: Default is every 1 hour
- **Closing alerts**: Default is 5 minutes before closing

## Database Schema

The scraper creates three tables:

**sales**
- `sale_id` (PRIMARY KEY)
- `title`, `location`, `closing_time`

**lots**
- `lot_id` (PRIMARY KEY)
- `sale_id`, `title`, `description`, `manufacturer`, `type`, `year`
- `category`, `current_bid`, `currency`, `url`
- `closing_time`, `closing_notified`

**images**
- `id` (PRIMARY KEY)
- `lot_id`, `url`, `file_path`, `labels` (detected objects)

## Notification Examples

### Desktop Notification
![System Tray Notification]
```
🔔 Kavel bieding update
Nieuw bod op kavel 12345: €150.00 (was €125.00)
```

### Email Notification
```
From: your.email@gmail.com
To: your.email@gmail.com
Subject: [Troostwijk] Kavel bieding update

Nieuw bod op kavel 12345: €150.00 (was €125.00)
```

**High Priority Alerts** (closing soon):
```
⚠️ Lot nearing closure
Kavel 12345 sluit binnen 5 min.
```

## Why This Approach?

✅ **100% Free** - No paid services (Twilio, Pushover, etc.)
✅ **No External Dependencies** - Desktop notifications built into Java
✅ **Works Offline** - Desktop notifications don't need internet
✅ **Privacy First** - Your data stays on your machine
✅ **Cross-Platform** - Windows, macOS, Linux supported
✅ **Optional Email** - Add Gmail notifications if you want

## Troubleshooting

### Desktop Notifications Not Showing

- **Windows**: Check if Java has notification permissions
- **Linux**: Ensure you have a desktop environment running (not headless)
- **macOS**: Check System Preferences → Notifications

### Email Not Sending

1. Verify 2FA is enabled in Google Account
2. Confirm you're using an **App Password** (not your regular Gmail password)
3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
4. Verify the SMTP format: `smtp:username:app_password:recipient`

## Notes

- Desktop notifications require a graphical environment (not headless servers)
- For headless servers, use email-only notifications
- Gmail SMTP is free and has generous limits (500 emails/day)
- OpenCV native libraries must match your platform architecture
- YOLO weights file is ~245 MB

## License

This is example code for educational purposes.