start
This commit is contained in:
Tour
228
README.md
228
README.md
@@ -133,16 +133,230 @@ java -Djava.library.path="/path/to/opencv/lib" \
|
||||
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
||||
```
|
||||
|
||||
## System Architecture & Integration Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ COMPLETE SYSTEM INTEGRATION DIAGRAM │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 1: EXTERNAL SCRAPER (Python/Playwright) - ARCHITECTURE-TROOSTWIJK │
|
||||
└──────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────┼─────────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
[Listing Pages] [Auction Pages] [Lot Pages]
|
||||
/auctions?page=N /a/auction-id /l/lot-id
|
||||
│ │ │
|
||||
│ Extract URLs │ Parse __NEXT_DATA__ │ Parse __NEXT_DATA__
|
||||
├────────────────────────────▶│ JSON │ JSON
|
||||
│ │ │
|
||||
│ ▼ ▼
|
||||
│ ┌────────────────┐ ┌────────────────┐
|
||||
│ │ INSERT auctions│ │ INSERT lots │
|
||||
│ │ to SQLite │ │ INSERT images │
|
||||
│ └────────────────┘ │ (URLs only) │
|
||||
│ │ └────────────────┘
|
||||
│ │ │
|
||||
└─────────────────────────────┴────────────────────────────┘
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ SQLITE DATABASE │
|
||||
│ troostwijk.db │
|
||||
└──────────────────┘
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
[auctions table] [lots table] [images table]
|
||||
- auction_id - lot_id - id
|
||||
- title - auction_id - lot_id
|
||||
- location - title - url
|
||||
- lots_count - current_bid - local_path
|
||||
- closing_time - bid_count - downloaded=0
|
||||
- closing_time
|
||||
│
|
||||
┌─────────────────────────────────────┴─────────────────────────────────────┐
|
||||
│ PHASE 2: MONITORING & PROCESSING (Java) - THIS PROJECT │
|
||||
└────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
[TroostwijkMonitor] [DatabaseService] [ScraperDataAdapter]
|
||||
│ │ │
|
||||
│ Read lots │ Query lots │ Transform data
|
||||
│ every hour │ Import images │ TEXT → INTEGER
|
||||
│ │ │ "€123" → 123.0
|
||||
└─────────────────┴─────────────────┘
|
||||
│
|
||||
┌─────────────────────────┼─────────────────────────┐
|
||||
▼ ▼ ▼
|
||||
[Bid Monitoring] [Image Processing] [Closing Alerts]
|
||||
Check API every 1h Download images Check < 5 min
|
||||
│ │ │
|
||||
│ New bid? │ Process via │ Time critical?
|
||||
├─[YES]──────────┐ │ ObjectDetection ├─[YES]────┐
|
||||
│ │ │ │ │
|
||||
▼ │ ▼ │ │
|
||||
[Update current_bid] │ ┌──────────────────┐ │ │
|
||||
in database │ │ YOLO Detection │ │ │
|
||||
│ │ OpenCV DNN │ │ │
|
||||
│ └──────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ Detect objects │ │
|
||||
│ ├─[vehicle] │ │
|
||||
│ ├─[furniture] │ │
|
||||
│ ├─[machinery] │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ [Save labels to DB] │ │
|
||||
│ [Estimate value] │ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
└─────────┴───────────────────────┴──────────┘
|
||||
│
|
||||
┌───────────────────────────────────────────────┴────────────────────────────┐
|
||||
│ PHASE 3: NOTIFICATION SYSTEM - USER INTERACTION TRIGGERS │
|
||||
└────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────┴─────────────────┐
|
||||
▼ ▼
|
||||
[NotificationService] [User Decision Points]
|
||||
│ │
|
||||
┌───────────────────┼───────────────────┐ │
|
||||
▼ ▼ ▼ │
|
||||
[Desktop Notify] [Email Notify] [Priority Level] │
|
||||
Windows/macOS/ Gmail SMTP 0=Normal │
|
||||
Linux system (FREE) 1=High │
|
||||
tray │
|
||||
│ │ │ │
|
||||
└───────────────────┴───────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────┐ ┌──────────────────┐
|
||||
│ USER INTERACTION │ │ TRIGGER EVENTS: │
|
||||
│ NOTIFICATIONS │ │ │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
│ │
|
||||
┌───────────────────┼───────────────────┐ │
|
||||
▼ ▼ ▼ │
|
||||
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ 1. BID CHANGE │ │ 2. OBJECT │ │ 3. CLOSING │ │
|
||||
│ │ │ DETECTED │ │ ALERT │ │
|
||||
│ "Nieuw bod op │ │ │ │ │ │
|
||||
│ kavel 12345: │ │ "Lot contains: │ │ "Kavel 12345 │ │
|
||||
│ €150 (was €125)"│ │ - Vehicle │ │ sluit binnen │ │
|
||||
│ │ │ - Machinery │ │ 5 min." │ │
|
||||
│ Priority: NORMAL │ │ Est: €5000" │ │ Priority: HIGH │ │
|
||||
│ │ │ │ │ │ │
|
||||
│ Action needed: │ │ Action needed: │ │ Action needed: │ │
|
||||
│ ▸ Place bid? │ │ ▸ Review item? │ │ ▸ Place final │ │
|
||||
│ ▸ Monitor? │ │ ▸ Confirm value? │ │ bid? │ │
|
||||
│ ▸ Ignore? │ │ ▸ Add to watch? │ │ ▸ Let expire? │ │
|
||||
└──────────────────┘ └──────────────────┘ └──────────────────┘ │
|
||||
│ │ │ │
|
||||
└───────────────────┴───────────────────┴─────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ USER ACTIONS & EXCEPTIONS │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Additional interaction points: │
|
||||
│ │
|
||||
│ 4. VIEWING DAY QUESTIONS │
|
||||
│ "Bezichtiging op [date] - kunt u aanwezig zijn?" │
|
||||
│ Action: ▸ Confirm attendance ▸ Request alternative ▸ Decline │
|
||||
│ │
|
||||
│ 5. ITEM RECOGNITION CONFIRMATION │
|
||||
│ "Detected: [object] - Is deze correcte identificatie?" │
|
||||
│ Action: ▸ Confirm ▸ Correct label ▸ Add notes │
|
||||
│ │
|
||||
│ 6. VALUE ESTIMATE APPROVAL │
|
||||
│ "Geschatte waarde: €X - Akkoord?" │
|
||||
│ Action: ▸ Accept ▸ Adjust ▸ Request manual review │
|
||||
│ │
|
||||
│ 7. EXCEPTION HANDLING │
|
||||
│ "Afwijkende sluitingstijd / locatiewijziging / special terms" │
|
||||
│ Action: ▸ Acknowledge ▸ Update preferences ▸ Withdraw interest │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ OBJECT DETECTION & VALUE ESTIMATION PIPELINE │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
[Downloaded Image] → [ImageProcessingService]
|
||||
│ │
|
||||
│ ▼
|
||||
│ [ObjectDetectionService]
|
||||
│ │
|
||||
│ ├─ Load YOLO model
|
||||
│ ├─ Run inference (416x416)
|
||||
│ ├─ Post-process detections
|
||||
│ │ (confidence > 0.5)
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────────────┐
|
||||
│ │ Detected Objects: │
|
||||
│ │ - person │
|
||||
│ │ - car │
|
||||
│ │ - truck │
|
||||
│ │ - furniture │
|
||||
│ │ - machinery │
|
||||
│ │ - electronics │
|
||||
│ │ (80 COCO classes) │
|
||||
│ └──────────────────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ [Value Estimation Logic]
|
||||
│ (Future enhancement)
|
||||
│ │
|
||||
│ ├─ Match objects to auction categories
|
||||
│ ├─ Historical price analysis
|
||||
│ ├─ Condition assessment
|
||||
│ ├─ Market trends
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────────────┐
|
||||
│ │ Estimated Value: │
|
||||
│ │ €X - €Y range │
|
||||
│ │ Confidence: 75% │
|
||||
│ └──────────────────────┘
|
||||
│ │
|
||||
└──────────────────────┴─ [Save to DB]
|
||||
│
|
||||
▼
|
||||
[Trigger notification if
|
||||
value > threshold]
|
||||
|
||||
```
|
||||
|
||||
## Integration Hooks & Timing
|
||||
|
||||
| Event | Frequency | Trigger | Notification Type | User Action Required |
|
||||
|-------|-----------|---------|-------------------|---------------------|
|
||||
| **New auction discovered** | On scrape | Scraper finds new auction | Desktop + Email (optional) | Review auction |
|
||||
| **Bid change detected** | Every 1 hour | Monitor detects higher bid | Desktop + Email | Place counter-bid? |
|
||||
| **Closing soon (< 30 min)** | When detected | Time-based check | Desktop + Email | Review lot |
|
||||
| **Closing imminent (< 5 min)** | When detected | Time-based check | Desktop + Email (HIGH) | Final bid decision |
|
||||
| **Object detected** | On image process | YOLO finds objects | Desktop + Email | Confirm identification |
|
||||
| **Value estimated** | After detection | Estimation complete | Desktop + Email | Approve estimate |
|
||||
| **Viewing day scheduled** | From lot metadata | Scraper extracts date | Desktop + Email | Confirm attendance |
|
||||
| **Exception/Change** | On update | Scraper detects change | Desktop + Email (HIGH) | Acknowledge |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/main/java/com/auction/scraper/
|
||||
├── TroostwijkScraper.java # Main scraper class
|
||||
│ ├── Lot # Domain model for auction lots
|
||||
│ ├── DatabaseService # SQLite operations
|
||||
│ ├── NotificationService # Desktop + Email notifications (FREE)
|
||||
│ └── ObjectDetectionService # OpenCV YOLO object detection
|
||||
└── Main.java # Entry point
|
||||
src/main/java/com/auction/
|
||||
├── Main.java # Entry point
|
||||
├── TroostwijkMonitor.java # Monitoring & orchestration
|
||||
├── DatabaseService.java # SQLite operations
|
||||
├── ScraperDataAdapter.java # Schema translation (TEXT→INT, €→float)
|
||||
├── ImageProcessingService.java # Downloads & processes images
|
||||
├── ObjectDetectionService.java # OpenCV YOLO detection
|
||||
├── NotificationService.java # Desktop + Email notifications (FREE)
|
||||
├── Lot.java # Domain model for auction lots
|
||||
├── AuctionInfo.java # Domain model for auctions
|
||||
└── Console.java # Logging utility
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Reference in New Issue
Block a user