Troostwijk Auction Scraper

A Java-based web scraper for Dutch auctions on Troostwijk Auctions with 100% free desktop/email notifications, SQLite persistence, and AI-powered object detection.

Features

• Auction Discovery: Automatically discovers active Dutch auctions
• Data Scraping: Fetches detailed lot information via Troostwijk's JSON API
• SQLite Storage: Persists auction data, lots, images, and detected objects
• Image Processing: Downloads and analyzes lot images using OpenCV YOLO object detection
• Free Notifications: Real-time notifications when:
  • Bids change on monitored lots
  • Auctions are closing soon (within 5 minutes)
  • Via desktop notifications (Windows/macOS/Linux system tray) ✅
  • Optionally via email (Gmail SMTP - free) ✅

Dependencies

All dependencies are managed via Maven (see pom.xml):

• jsoup 1.17.2 - HTML parsing and HTTP client
• Jackson 2.17.0 - JSON processing
• SQLite JDBC 3.45.1.0 - Database operations
• JavaMail 1.6.2 - Email notifications (free)
• OpenCV 4.9.0 - Image processing and object detection

Quick Start

Development: Sync Production Data

To work with real production data locally:

# Linux/Mac (Bash)
./scripts/sync-production-data.sh --db-only

See scripts/README.md for full documentation.

Setup

1. Notification Options (Choose One)

Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)

Desktop notifications work out of the box on:

• Windows: System tray notifications
• macOS: Notification Center
• Linux: Desktop environment notifications (GNOME, KDE, etc.)

No configuration required! Just run with default settings:

export NOTIFICATION_CONFIG="desktop"
# Or simply don't set it - desktop is the default

Option B: Desktop + Email Notifications 📧 (Free Gmail)

1. Enable 2-Factor Authentication in your Google Account
2. Go to: Google Account → Security → 2-Step Verification → App passwords
3. Generate an app password for "Mail"
4. Set environment variable:

   export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:your_app_password:recipient@example.com"
   

Format: smtp:username:app_password:recipient_email

Example:

export NOTIFICATION_CONFIG="smtp:john.doe@gmail.com:abcd1234efgh5678:john.doe@gmail.com"

Note: This is completely free using Gmail's SMTP server. No paid services required!

2. OpenCV Native Libraries

Download and install OpenCV native libraries for your platform:

Windows:

# Download from https://opencv.org/releases/
# Extract and add to PATH or use:
java -Djava.library.path="C:\opencv\build\java\x64" -jar scraper.jar

Linux:

sudo apt-get install libopencv-dev

macOS:

brew install opencv

3. YOLO Model Files

Download YOLO model files for object detection:

mkdir /mnt/okcomputer/output/models
cd /mnt/okcomputer/output/models

# Download YOLOv4 config
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg

# Download YOLOv4 weights (245 MB)
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights

# Download COCO class names
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names

Building

mvn clean package

This creates:

• ../build/auctiora/auctiora-1.0-SNAPSHOT.jar - Regular JAR
• ../build/auctiora/auctiora-1.0-SNAPSHOT-jar-with-dependencies.jar - Executable JAR with all dependencies

Running

Quick Start (Desktop Notifications Only)

java -Djava.library.path="/path/to/opencv/lib" \
     -jar ../build/auctiora/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar

With Email Notifications

export NOTIFICATION_CONFIG="smtp:your@gmail.com:app_password:your@gmail.com"

java -Djava.library.path="/path/to/opencv/lib" \
     -jar ../build/auctiora/troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar

Using Maven

mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"

System Architecture & Integration Flow

📊 Complete Integration Flowchart: See docs/INTEGRATION_FLOWCHART.md for the detailed intelligence integration diagram with GraphQL API fields, analytics, and dashboard features.

Quick Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                    COMPLETE SYSTEM INTEGRATION DIAGRAM                       │
└─────────────────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────────┐
│  PHASE 1: EXTERNAL SCRAPER (Python/Playwright) - ARCHITECTURE-TROOSTWIJK    │
└──────────────────────────────────────────────────────────────────────────────┘
                                      │
        ┌─────────────────────────────┼─────────────────────────────┐
        ▼                             ▼                             ▼
  [Listing Pages]              [Auction Pages]              [Lot Pages]
  /auctions?page=N             /a/auction-id                /l/lot-id
        │                             │                             │
        │ Extract URLs                │ Parse __NEXT_DATA__         │ Parse __NEXT_DATA__
        ├────────────────────────────▶│ JSON                        │ JSON
        │                             │                             │
        │                             ▼                             ▼
        │                    ┌────────────────┐           ┌────────────────┐
        │                    │ INSERT auctions│           │  INSERT lots   │
        │                    │ to SQLite      │           │  INSERT images │
        │                    └────────────────┘           │  (URLs only)   │
        │                             │                   └────────────────┘
        │                             │                            │
        └─────────────────────────────┴────────────────────────────┘
                                      ▼
                            ┌──────────────────┐
                            │  SQLITE DATABASE │
                            │  output/cache.db │
                            └──────────────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
            [auctions table]   [lots table]    [images table]
            - auction_id       - lot_id        - id
            - title            - auction_id    - lot_id
            - location         - title         - url
            - lots_count       - current_bid   - local_path
            - closing_time     - bid_count     - downloaded=0
                               - closing_time
                                      │
┌─────────────────────────────────────┴─────────────────────────────────────┐
│  PHASE 2: MONITORING & PROCESSING (Java) - THIS PROJECT                   │
└────────────────────────────────────────────────────────────────────────────┘
                                      │
                    ┌─────────────────┼─────────────────┐
                    ▼                 ▼                 ▼
         [TroostwijkMonitor]  [DatabaseService]  [ScraperDataAdapter]
                    │                 │                 │
                    │ Read lots       │ Query lots      │ Transform data
                    │ every hour      │ Import images   │ TEXT → INTEGER
                    │                 │                 │ "€123" → 123.0
                    └─────────────────┴─────────────────┘
                                      │
            ┌─────────────────────────┼─────────────────────────┐
            ▼                         ▼                         ▼
   [Bid Monitoring]         [Image Processing]        [Closing Alerts]
   Check API every 1h       Download images           Check < 5 min
            │                         │                         │
            │ New bid?                │ Process via             │ Time critical?
            ├─[YES]──────────┐        │ ObjectDetection         ├─[YES]────┐
            │                 │        │                         │          │
            ▼                 │        ▼                         │          │
   [Update current_bid]       │  ┌──────────────────┐          │          │
   in database                │  │ YOLO Detection   │          │          │
                              │  │ OpenCV DNN       │          │          │
                              │  └──────────────────┘          │          │
                              │         │                       │          │
                              │         │ Detect objects        │          │
                              │         ├─[vehicle]            │          │
                              │         ├─[furniture]          │          │
                              │         ├─[machinery]          │          │
                              │         │                       │          │
                              │         ▼                       │          │
                              │  [Save labels to DB]            │          │
                              │  [Estimate value]               │          │
                              │         │                       │          │
                              │         │                       │          │
                              └─────────┴───────────────────────┴──────────┘
                                                │
┌───────────────────────────────────────────────┴────────────────────────────┐
│  PHASE 3: NOTIFICATION SYSTEM - USER INTERACTION TRIGGERS                  │
└────────────────────────────────────────────────────────────────────────────┘
                                                │
                              ┌─────────────────┴─────────────────┐
                              ▼                                   ▼
                   [NotificationService]                [User Decision Points]
                              │                                   │
          ┌───────────────────┼───────────────────┐              │
          ▼                   ▼                   ▼              │
   [Desktop Notify]    [Email Notify]     [Priority Level]       │
   Windows/macOS/      Gmail SMTP          0=Normal              │
   Linux system        (FREE)              1=High                │
   tray                                                           │
          │                   │                   │              │
          └───────────────────┴───────────────────┘              │
                              │                                  │
                              ▼                                  ▼
                    ┌──────────────────┐              ┌──────────────────┐
                    │ USER INTERACTION │              │ TRIGGER EVENTS:  │
                    │ NOTIFICATIONS    │              │                  │
                    └──────────────────┘              └──────────────────┘
                              │                                  │
          ┌───────────────────┼───────────────────┐             │
          ▼                   ▼                   ▼             │
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ 1. BID CHANGE    │ │ 2. OBJECT        │ │ 3. CLOSING       │ │
│                  │ │    DETECTED       │ │    ALERT         │ │
│ "Nieuw bod op    │ │                  │ │                  │ │
│  kavel 12345:    │ │ "Lot contains:   │ │ "Kavel 12345     │ │
│  €150 (was €125)"│ │  - Vehicle       │ │  sluit binnen    │ │
│                  │ │  - Machinery     │ │  5 min."         │ │
│ Priority: NORMAL │ │  Est: €5000"     │ │ Priority: HIGH   │ │
│                  │ │                  │ │                  │ │
│ Action needed:   │ │ Action needed:   │ │ Action needed:   │ │
│ ▸ Place bid?     │ │ ▸ Review item?   │ │ ▸ Place final    │ │
│ ▸ Monitor?       │ │ ▸ Confirm value? │ │   bid?           │ │
│ ▸ Ignore?        │ │ ▸ Add to watch?  │ │ ▸ Let expire?    │ │
└──────────────────┘ └──────────────────┘ └──────────────────┘ │
          │                   │                   │             │
          └───────────────────┴───────────────────┴─────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  USER ACTIONS & EXCEPTIONS                                                  │
├─────────────────────────────────────────────────────────────────────────────┤
│  Additional interaction points:                                             │
│                                                                              │
│  4. VIEWING DAY QUESTIONS                                                   │
│     "Bezichtiging op [date] - kunt u aanwezig zijn?"                        │
│     Action: ▸ Confirm attendance  ▸ Request alternative  ▸ Decline         │
│                                                                              │
│  5. ITEM RECOGNITION CONFIRMATION                                           │
│     "Detected: [object] - Is deze correcte identificatie?"                  │
│     Action: ▸ Confirm  ▸ Correct label  ▸ Add notes                        │
│                                                                              │
│  6. VALUE ESTIMATE APPROVAL                                                 │
│     "Geschatte waarde: €X - Akkoord?"                                       │
│     Action: ▸ Accept  ▸ Adjust  ▸ Request manual review                    │
│                                                                              │
│  7. EXCEPTION HANDLING                                                      │
│     "Afwijkende sluitingstijd / locatiewijziging / special terms"           │
│     Action: ▸ Acknowledge  ▸ Update preferences  ▸ Withdraw interest       │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│  OBJECT DETECTION & VALUE ESTIMATION PIPELINE                               │
└─────────────────────────────────────────────────────────────────────────────┘

[Downloaded Image] → [ImageProcessingService]
         │                      │
         │                      ▼
         │           [ObjectDetectionService]
         │                      │
         │                      ├─ Load YOLO model
         │                      ├─ Run inference (416x416)
         │                      ├─ Post-process detections
         │                      │  (confidence > 0.5)
         │                      │
         │                      ▼
         │           ┌──────────────────────┐
         │           │ Detected Objects:    │
         │           │ - person             │
         │           │ - car                │
         │           │ - truck              │
         │           │ - furniture          │
         │           │ - machinery          │
         │           │ - electronics        │
         │           │ (80 COCO classes)    │
         │           └──────────────────────┘
         │                      │
         │                      ▼
         │           [Value Estimation Logic]
         │           (Future enhancement)
         │                      │
         │                      ├─ Match objects to auction categories
         │                      ├─ Historical price analysis
         │                      ├─ Condition assessment
         │                      ├─ Market trends
         │                      │
         │                      ▼
         │           ┌──────────────────────┐
         │           │ Estimated Value:     │
         │           │ €X - €Y range        │
         │           │ Confidence: 75%      │
         │           └──────────────────────┘
         │                      │
         └──────────────────────┴─ [Save to DB]
                                   │
                                   ▼
                         [Trigger notification if
                          value > threshold]

flowchart TD
    subgraph P1["PHASE 1: EXTERNAL SCRAPER (Python/Playwright)"]
        direction LR
        A1[Listing Pages<br/>/auctions?page=N] --> A2[Extract URLs]
        B1[Auction Pages<br/>/a/auction-id] --> B2[Parse __NEXT_DATA__ JSON]
        C1[Lot Pages<br/>/l/lot-id] --> C2[Parse __NEXT_DATA__ JSON]

        A2 --> D1[INSERT auctions to SQLite]
        B2 --> D1
        C2 --> D2[INSERT lots & image URLs]
        
        D1 --> DB[(SQLite Database<br/>output/cache.db)]
        D2 --> DB
    end

    DB --> P2_Entry

    subgraph P2["PHASE 2: MONITORING & PROCESSING (Java)"]
        direction TB
        P2_Entry[Data Ready] --> Monitor[TroostwijkMonitor<br/>Read lots every hour]
        P2_Entry --> DBService[DatabaseService<br/>Query & Import]
        P2_Entry --> Adapter[ScraperDataAdapter<br/>Transform TEXT → INTEGER]

        Monitor --> BM[Bid Monitoring<br/>Check API every 1h]
        DBService --> IP[Image Processing<br/>Download & Analyze]
        Adapter --> DataForNotify[Formatted Data]

        BM --> BidUpdate{New bid?}
        BidUpdate -->|Yes| UpdateDB[Update current_bid in DB]
        UpdateDB --> NotifyTrigger1

        IP --> Detection[Object Detection<br/>YOLO/OpenCV DNN]
        Detection --> ObjectCheck{Detect objects?}
        ObjectCheck -->|Vehicle| Save1[Save labels & estimate value]
        ObjectCheck -->|Furniture| Save2[Save labels & estimate value]
        ObjectCheck -->|Machinery| Save3[Save labels & estimate value]
        Save1 --> NotifyTrigger2
        Save2 --> NotifyTrigger2
        Save3 --> NotifyTrigger2

        CA[Closing Alerts<br/>Check < 5 min] --> TimeCheck{Time critical?}
        TimeCheck -->|Yes| NotifyTrigger3
    end

    NotifyTrigger1 --> NS
    NotifyTrigger2 --> NS
    NotifyTrigger3 --> NS

    subgraph P3["PHASE 3: NOTIFICATION SYSTEM"]
        NS[NotificationService] --> DN[Desktop Notify<br/>Windows/macOS/Linux]
        NS --> EN[Email Notify<br/>Gmail SMTP]
        NS --> PL[Set Priority Level<br/>0=Normal, 1=High]
    end

    DN --> UI[User Interaction & Decisions]
    EN --> UI
    PL --> UI

    subgraph UI_Details[User Decision Points / Trigger Events]
        direction LR
        E1["1. BID CHANGE<br/>'Nieuw bod op kavel 12345...'<br/>Actions: Place bid? Monitor? Ignore?"]
        E2["2. OBJECT DETECTED<br/>'Lot contains: Vehicle...'<br/>Actions: Review? Confirm value?"]
        E3["3. CLOSING ALERT<br/>'Kavel 12345 sluit binnen 5 min.'<br/>Actions: Place final bid? Let expire?"]
        E4["4. VIEWING DAY QUESTIONS<br/>'Bezichtiging op [date]...'"]
        E5["5. ITEM RECOGNITION CONFIRMATION<br/>'Detected: [object]...'"]
        E6["6. VALUE ESTIMATE APPROVAL<br/>'Geschatte waarde: €X...'"]
        E7["7. EXCEPTION HANDLING<br/>'Afwijkende sluitingstijd...'"]
    end

    UI --> UI_Details

    %% Object Detection Sub-Flow Detail
    subgraph P2_Detail["Object Detection & Value Estimation Pipeline"]
        direction LR
        DI[Downloaded Image] --> IPS[ImageProcessingService]
        IPS --> ODS[ObjectDetectionService]
        ODS --> Load[Load YOLO model]
        ODS --> Run[Run inference]
        ODS --> Post[Post-process detections<br/>confidence > 0.5]
        Post --> ObjList["Detected Objects List<br/>(80 COCO classes)"]
        ObjList --> VEL[Value Estimation Logic<br/>Future enhancement]
        VEL --> Match[Match to categories]
        VEL --> History[Historical price analysis]
        VEL --> Condition[Condition assessment]
        VEL --> Market[Market trends]
        Market --> ValueEst["Estimated Value Range<br/>Confidence: 75%"]
        ValueEst --> SaveToDB[Save to Database]
        SaveToDB --> TriggerNotify{Value > threshold?}
    end

    IP -.-> P2_Detail
    TriggerNotify -.-> NotifyTrigger2

Integration Hooks & Timing

Event	Frequency	Trigger	Notification Type	User Action Required
New auction discovered	On scrape	Scraper finds new auction	Desktop + Email (optional)	Review auction
Bid change detected	Every 1 hour	Monitor detects higher bid	Desktop + Email	Place counter-bid?
Closing soon (< 30 min)	When detected	Time-based check	Desktop + Email	Review lot
Closing imminent (< 5 min)	When detected	Time-based check	Desktop + Email (HIGH)	Final bid decision
Object detected	On image process	YOLO finds objects	Desktop + Email	Confirm identification
Value estimated	After detection	Estimation complete	Desktop + Email	Approve estimate
Viewing day scheduled	From lot metadata	Scraper extracts date	Desktop + Email	Confirm attendance
Exception/Change	On update	Scraper detects change	Desktop + Email (HIGH)	Acknowledge

Project Structure

src/main/java/com/auction/
├── Main.java                        # Entry point
├── TroostwijkMonitor.java          # Monitoring & orchestration
├── DatabaseService.java             # SQLite operations
├── ScraperDataAdapter.java         # Schema translation (TEXT→INT, €→float)
├── ImageProcessingService.java     # Downloads & processes images
├── ObjectDetectionService.java     # OpenCV YOLO detection
├── NotificationService.java        # Desktop + Email notifications (FREE)
├── Lot.java                        # Domain model for auction lots
├── AuctionInfo.java                # Domain model for auctions
└── Console.java                    # Logging utility

Configuration

Edit TroostwijkScraper.main() to customize:

• Database file: troostwijk.db (SQLite database location)
• YOLO paths: Model configuration and weights files
• Monitoring frequency: Default is every 1 hour
• Closing alerts: Default is 5 minutes before closing

Database Schema

The scraper creates three tables:

sales

• sale_id (PRIMARY KEY)
• title, location, closing_time

lots

• lot_id (PRIMARY KEY)
• sale_id, title, description, manufacturer, type, year
• category, current_bid, currency, url
• closing_time, closing_notified

images

• id (PRIMARY KEY)
• lot_id, url, local_path, labels (detected objects)

Notification Examples

Desktop Notification

![System Tray Notification]

🔔 Kavel bieding update
Nieuw bod op kavel 12345: €150.00 (was €125.00)

Email Notification

From: your.email@gmail.com
To: your.email@gmail.com
Subject: [Troostwijk] Kavel bieding update

Nieuw bod op kavel 12345: €150.00 (was €125.00)

High Priority Alerts (closing soon):

⚠️ Lot nearing closure
Kavel 12345 sluit binnen 5 min.

Why This Approach?

✅ 100% Free - No paid services (Twilio, Pushover, etc.) ✅ No External Dependencies - Desktop notifications built into Java ✅ Works Offline - Desktop notifications don't need internet ✅ Privacy First - Your data stays on your machine ✅ Cross-Platform - Windows, macOS, Linux supported ✅ Optional Email - Add Gmail notifications if you want

Troubleshooting

Desktop Notifications Not Showing

• Windows: Check if Java has notification permissions
• Linux: Ensure you have a desktop environment running (not headless)
• macOS: Check System Preferences → Notifications

Email Not Sending

1. Verify 2FA is enabled in Google Account
2. Confirm you're using an App Password (not your regular Gmail password)
3. Check that "Less secure app access" is NOT needed (app passwords work with 2FA)
4. Verify the SMTP format: smtp:username:app_password:recipient

Notes

• Desktop notifications require a graphical environment (not headless servers)
• For headless servers, use email-only notifications
• Gmail SMTP is free and has generous limits (500 emails/day)
• OpenCV native libraries must match your platform architecture
• YOLO weights file is ~245 MB

ssh tour@athena.lan "docker run --rm -v shared-auction-data:/data -v /tmp:/tmp alpine cp /data/cache.db /tmp/cache.db" && scp tour@athena.lan:/tmp/cache.db c:/mnt/okcomputer/cache.db

License

This is example code for educational purposes.