enrich data
This commit is contained in:
@@ -1,118 +0,0 @@
|
|||||||
# Refactoring Summary: Troostwijk Auction Monitor
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
This project has been refactored to focus on **image processing and monitoring**, removing all auction/lot scraping functionality which is now handled by the external `ARCHITECTURE-TROOSTWIJK-SCRAPER` process.
|
|
||||||
|
|
||||||
## Architecture Changes
|
|
||||||
|
|
||||||
### Removed Components
|
|
||||||
- ❌ **TroostwijkScraper.java** - Removed (replaced by TroostwijkMonitor)
|
|
||||||
- ❌ Auction discovery and scraping logic
|
|
||||||
- ❌ Lot scraping via Playwright/JSoup
|
|
||||||
- ❌ CacheDatabase (can be removed if not used elsewhere)
|
|
||||||
|
|
||||||
### New/Updated Components
|
|
||||||
|
|
||||||
#### New Classes
|
|
||||||
- ✅ **TroostwijkMonitor.java** - Monitors bids and coordinates services (no scraping)
|
|
||||||
- ✅ **ImageProcessingService.java** - Downloads images and runs object detection
|
|
||||||
- ✅ **Console.java** - Simple output utility (renamed from IO to avoid Java 25 conflict)
|
|
||||||
|
|
||||||
#### Modernized Classes
|
|
||||||
- ✅ **AuctionInfo** - Converted to immutable `record`
|
|
||||||
- ✅ **Lot** - Converted to immutable `record` with `minutesUntilClose()` method
|
|
||||||
- ✅ **DatabaseService.java** - Uses modern Java features:
|
|
||||||
- Text blocks (`"""`) for SQL
|
|
||||||
- Record accessor methods
|
|
||||||
- Added `getImagesForLot()` method
|
|
||||||
- Added `processed_at` timestamp to images table
|
|
||||||
- Nested `ImageRecord` record
|
|
||||||
|
|
||||||
#### Preserved Components
|
|
||||||
- ✅ **NotificationService.java** - Desktop/email notifications
|
|
||||||
- ✅ **ObjectDetectionService.java** - YOLO-based object detection
|
|
||||||
- ✅ **Main.java** - Updated to use new architecture
|
|
||||||
|
|
||||||
## Database Schema
|
|
||||||
|
|
||||||
### Populated by External Scraper
|
|
||||||
- `auctions` table - Auction metadata
|
|
||||||
- `lots` table - Lot details with bidding info
|
|
||||||
|
|
||||||
### Populated by This Process
|
|
||||||
- `images` table - Downloaded images with:
|
|
||||||
- `file_path` - Local storage path
|
|
||||||
- `labels` - Detected objects (comma-separated)
|
|
||||||
- `processed_at` - Processing timestamp
|
|
||||||
|
|
||||||
## Modern Java Features Used
|
|
||||||
|
|
||||||
- **Records** - Immutable data carriers (AuctionInfo, Lot, ImageRecord)
|
|
||||||
- **Text Blocks** - Multi-line SQL queries
|
|
||||||
- **var** - Type inference throughout
|
|
||||||
- **Switch expressions** - Where applicable
|
|
||||||
- **Pattern matching** - Ready for future enhancements
|
|
||||||
|
|
||||||
## Responsibilities
|
|
||||||
|
|
||||||
### This Project
|
|
||||||
1. ✅ Image downloading from URLs in database
|
|
||||||
2. ✅ Object detection using YOLO/OpenCV
|
|
||||||
3. ✅ Bid monitoring and change detection
|
|
||||||
4. ✅ Desktop and email notifications
|
|
||||||
5. ✅ Data enrichment with image analysis
|
|
||||||
|
|
||||||
### External ARCHITECTURE-TROOSTWIJK-SCRAPER
|
|
||||||
1. 🔄 Discover auctions from Troostwijk website
|
|
||||||
2. 🔄 Scrape lot details via API
|
|
||||||
3. 🔄 Populate `auctions` and `lots` tables
|
|
||||||
4. 🔄 Share database with this process
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Running the Monitor
|
|
||||||
```bash
|
|
||||||
# With environment variables
|
|
||||||
export DATABASE_FILE=troostwijk.db
|
|
||||||
export NOTIFICATION_CONFIG=desktop # or smtp:user:pass:email
|
|
||||||
|
|
||||||
java -jar troostwijk-monitor.jar
|
|
||||||
```
|
|
||||||
|
|
||||||
### Expected Output
|
|
||||||
```
|
|
||||||
=== Troostwijk Auction Monitor ===
|
|
||||||
|
|
||||||
✓ OpenCV loaded
|
|
||||||
Initializing monitor...
|
|
||||||
|
|
||||||
📊 Current Database State:
|
|
||||||
Total lots in database: 42
|
|
||||||
Total images processed: 0
|
|
||||||
|
|
||||||
[1/2] Processing images...
|
|
||||||
Processing pending images...
|
|
||||||
|
|
||||||
[2/2] Starting bid monitoring...
|
|
||||||
✓ Monitoring service started
|
|
||||||
|
|
||||||
✓ Monitor is running. Press Ctrl+C to stop.
|
|
||||||
|
|
||||||
NOTE: This process expects auction/lot data from the external scraper.
|
|
||||||
Make sure ARCHITECTURE-TROOSTWIJK-SCRAPER is running and populating the database.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Migration Notes
|
|
||||||
|
|
||||||
1. The project now compiles successfully with Java 25
|
|
||||||
2. All scraping logic removed - rely on external scraper
|
|
||||||
3. Shared database architecture for inter-process communication
|
|
||||||
4. Clean separation of concerns
|
|
||||||
5. Modern, maintainable codebase with records and text blocks
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
- Remove `CacheDatabase.java` if not needed
|
|
||||||
- Consider adding API endpoint for external scraper to trigger image processing
|
|
||||||
- Add metrics/logging framework
|
|
||||||
- Consider message queue (e.g., Redis, RabbitMQ) for better inter-process communication
|
|
||||||
Reference in New Issue
Block a user