Compare commits
4 Commits
7600cebcbb
...
3cc0d40fa3
| Author | SHA1 | Date | |
|---|---|---|---|
|
3cc0d40fa3 | ||
|
|
be65f4a5e6 | ||
|
|
3358a2693c | ||
|
|
62cda5c0cb |
2
.mvn/wrapper/maven-wrapper.config
vendored
2
.mvn/wrapper/maven-wrapper.config
vendored
@@ -1 +1 @@
|
|||||||
jvmArguments=--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED
|
jvmArguments=-Djava.util.logging.manager=org.jboss.logmanager.LogManager --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED
|
||||||
@@ -99,8 +99,8 @@ brew install opencv
|
|||||||
Download YOLO model files for object detection:
|
Download YOLO model files for object detection:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mkdir models
|
mkdir /mnt/okcomputer/output/models
|
||||||
cd models
|
cd /mnt/okcomputer/output/models
|
||||||
|
|
||||||
# Download YOLOv4 config
|
# Download YOLOv4 config
|
||||||
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
|
wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
|
||||||
|
|||||||
@@ -1,113 +0,0 @@
|
|||||||
# Database Schema Fix Instructions
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
The server database was created with `BIGINT` primary keys for `auction_id` and `lot_id`, but the scraper uses TEXT IDs like "A7-40063-2". This causes PRIMARY KEY constraint failures.
|
|
||||||
|
|
||||||
## Root Cause
|
|
||||||
- Local DB: `auction_id TEXT PRIMARY KEY`, `lot_id TEXT PRIMARY KEY`
|
|
||||||
- Server DB (old): `auction_id BIGINT PRIMARY KEY`, `lot_id BIGINT PRIMARY KEY`
|
|
||||||
- Scraper data: Uses TEXT IDs like "A7-40063-2", "A1-34732-49"
|
|
||||||
|
|
||||||
This mismatch prevents the scraper from inserting data, resulting in zero bids showing in the UI.
|
|
||||||
|
|
||||||
## Solution
|
|
||||||
|
|
||||||
### Step 1: Backup Server Database
|
|
||||||
```bash
|
|
||||||
ssh tour@192.168.1.149
|
|
||||||
cd /mnt/okcomputer/output
|
|
||||||
cp cache.db cache.db.backup.$(date +%Y%m%d_%H%M%S)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Upload Fix Script
|
|
||||||
From your local machine:
|
|
||||||
```bash
|
|
||||||
scp C:\vibe\auctiora\fix-schema.sql tour@192.168.1.149:/tmp/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 3: Stop the Application
|
|
||||||
```bash
|
|
||||||
ssh tour@192.168.1.149
|
|
||||||
cd /path/to/docker/compose # wherever docker-compose.yml is located
|
|
||||||
docker-compose down
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4: Apply Schema Fix
|
|
||||||
```bash
|
|
||||||
ssh tour@192.168.1.149
|
|
||||||
cd /mnt/okcomputer/output
|
|
||||||
sqlite3 cache.db < /tmp/fix-schema.sql
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 5: Verify Schema
|
|
||||||
```bash
|
|
||||||
sqlite3 cache.db "PRAGMA table_info(auctions);"
|
|
||||||
# Should show: auction_id TEXT PRIMARY KEY
|
|
||||||
|
|
||||||
sqlite3 cache.db "PRAGMA table_info(lots);"
|
|
||||||
# Should show: lot_id TEXT PRIMARY KEY, sale_id TEXT, auction_id TEXT
|
|
||||||
|
|
||||||
# Check data integrity
|
|
||||||
sqlite3 cache.db "SELECT COUNT(*) FROM auctions;"
|
|
||||||
sqlite3 cache.db "SELECT COUNT(*) FROM lots;"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 6: Rebuild Application with Fixed Schema
|
|
||||||
```bash
|
|
||||||
# Build new image with fixed DatabaseService.java
|
|
||||||
cd C:\vibe\auctiora
|
|
||||||
./mvnw clean package -DskipTests
|
|
||||||
|
|
||||||
# Copy new JAR to server
|
|
||||||
scp target/quarkus-app/quarkus-run.jar tour@192.168.1.149:/path/to/app/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 7: Restart Application
|
|
||||||
```bash
|
|
||||||
ssh tour@192.168.1.149
|
|
||||||
cd /path/to/docker/compose
|
|
||||||
docker-compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 8: Verify Fix
|
|
||||||
```bash
|
|
||||||
# Check logs for successful imports
|
|
||||||
docker-compose logs -f --tail=100
|
|
||||||
|
|
||||||
# Should see:
|
|
||||||
# ✓ Imported XXX auctions
|
|
||||||
# ✓ Imported XXXXX lots
|
|
||||||
# No more "PRIMARY KEY constraint failed" errors
|
|
||||||
|
|
||||||
# Check UI at http://192.168.1.149:8081/
|
|
||||||
# Should now show:
|
|
||||||
# - Lots with Bids: > 0
|
|
||||||
# - Total Bid Value: > €0.00
|
|
||||||
# - Average Bid: > €0.00
|
|
||||||
```
|
|
||||||
|
|
||||||
## Alternative: Quick Fix Without Downtime
|
|
||||||
If you can't afford downtime, delete the corrupted database and let it rebuild:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh tour@192.168.1.149
|
|
||||||
cd /mnt/okcomputer/output
|
|
||||||
mv cache.db cache.db.old
|
|
||||||
docker-compose restart
|
|
||||||
|
|
||||||
# The app will create a new database with correct schema
|
|
||||||
# Wait for scraper to re-populate data (may take 10-15 minutes)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Files Changed
|
|
||||||
1. `DatabaseService.java` - Fixed schema definitions (auction_id, lot_id, sale_id now TEXT)
|
|
||||||
2. `fix-schema.sql` - SQL migration script to fix existing database
|
|
||||||
3. `SCHEMA_FIX_INSTRUCTIONS.md` - This file
|
|
||||||
|
|
||||||
## Testing Locally
|
|
||||||
Before deploying to server, test locally:
|
|
||||||
```bash
|
|
||||||
cd C:\vibe\auctiora
|
|
||||||
./mvnw clean test
|
|
||||||
# All tests should pass with new schema
|
|
||||||
```
|
|
||||||
@@ -14,8 +14,8 @@ services:
|
|||||||
- AUCTION_IMAGES_PATH=/mnt/okcomputer/output/images
|
- AUCTION_IMAGES_PATH=/mnt/okcomputer/output/images
|
||||||
|
|
||||||
# Notification configuration
|
# Notification configuration
|
||||||
- AUCTION_NOTIFICATION_CONFIG=desktop
|
# - AUCTION_NOTIFICATION_CONFIG=desktop
|
||||||
|
- AUCTION_NOTIFICATION_CONFIG=smtp:michael.bakker1986@gmail.com:agrepolhlnvhipkv:michael.bakker1986@gmail.com
|
||||||
# Quarkus configuration
|
# Quarkus configuration
|
||||||
- QUARKUS_HTTP_PORT=8081
|
- QUARKUS_HTTP_PORT=8081
|
||||||
- QUARKUS_HTTP_HOST=0.0.0.0
|
- QUARKUS_HTTP_HOST=0.0.0.0
|
||||||
|
|||||||
@@ -1,192 +0,0 @@
|
|||||||
# Database Cleanup Guide
|
|
||||||
|
|
||||||
## Problem: Mixed Data Formats
|
|
||||||
|
|
||||||
Your production database (`cache.db`) contains data from two different scrapers:
|
|
||||||
|
|
||||||
### Valid Data (99.92%)
|
|
||||||
- **Format**: `A1-34732-49` (lot_id) + `c1f44ec2-ad6e-4c98-b0e2-cb1d8ccddcab` (auction_id UUID)
|
|
||||||
- **Count**: 16,794 lots
|
|
||||||
- **Source**: Current GraphQL-based scraper
|
|
||||||
- **Status**: ✅ Clean, with proper auction_id
|
|
||||||
|
|
||||||
### Invalid Data (0.08%)
|
|
||||||
- **Format**: `bmw-550i-4-4-v8-high-executive-...` (slug as lot_id) + `""` (empty auction_id)
|
|
||||||
- **Count**: 13 lots
|
|
||||||
- **Source**: Old legacy scraper
|
|
||||||
- **Status**: ❌ Missing auction_id, causes issues
|
|
||||||
|
|
||||||
## Impact
|
|
||||||
|
|
||||||
These 13 invalid entries:
|
|
||||||
- Cause `NullPointerException` in analytics when grouping by country
|
|
||||||
- Cannot be properly linked to auctions
|
|
||||||
- Skew statistics slightly
|
|
||||||
- May cause issues with intelligence features that rely on auction_id
|
|
||||||
|
|
||||||
## Solution 1: Clean Sync (Recommended)
|
|
||||||
|
|
||||||
The updated sync script now **automatically removes old local data** before syncing:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Windows PowerShell
|
|
||||||
.\scripts\Sync-ProductionData.ps1
|
|
||||||
|
|
||||||
# Linux/Mac
|
|
||||||
./scripts/sync-production-data.sh --db-only
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does**:
|
|
||||||
1. Backs up existing database to `cache.db.backup-YYYYMMDD-HHMMSS`
|
|
||||||
2. **Removes old local database completely**
|
|
||||||
3. Downloads fresh copy from production
|
|
||||||
4. Shows data quality report
|
|
||||||
|
|
||||||
**Output includes**:
|
|
||||||
```
|
|
||||||
Database statistics:
|
|
||||||
┌─────────────┬────────┐
|
|
||||||
│ table_name │ count │
|
|
||||||
├─────────────┼────────┤
|
|
||||||
│ auctions │ 526 │
|
|
||||||
│ lots │ 16807 │
|
|
||||||
│ images │ 536502 │
|
|
||||||
│ cache │ 2134 │
|
|
||||||
└─────────────┴────────┘
|
|
||||||
|
|
||||||
Data quality:
|
|
||||||
┌────────────────────────────────────┬────────┬────────────┐
|
|
||||||
│ metric │ count │ percentage │
|
|
||||||
├────────────────────────────────────┼────────┼────────────┤
|
|
||||||
│ Valid lots │ 16794 │ 99.92% │
|
|
||||||
│ Invalid lots (missing auction_id) │ 13 │ 0.08% │
|
|
||||||
│ Lots with intelligence fields │ 0 │ 0.00% │
|
|
||||||
└────────────────────────────────────┴────────┴────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Solution 2: Manual Cleanup
|
|
||||||
|
|
||||||
If you want to clean your existing local database without re-downloading:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Dry run (see what would be deleted)
|
|
||||||
./scripts/cleanup-database.sh --dry-run
|
|
||||||
|
|
||||||
# Actual cleanup
|
|
||||||
./scripts/cleanup-database.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does**:
|
|
||||||
1. Creates backup before cleanup
|
|
||||||
2. Deletes lots with missing auction_id
|
|
||||||
3. Deletes orphaned images (images without matching lots)
|
|
||||||
4. Compacts database (VACUUM) to reclaim space
|
|
||||||
5. Shows before/after statistics
|
|
||||||
|
|
||||||
**Example output**:
|
|
||||||
```
|
|
||||||
Current database state:
|
|
||||||
┌──────────────────────────────────┬────────┐
|
|
||||||
│ metric │ count │
|
|
||||||
├──────────────────────────────────┼────────┤
|
|
||||||
│ Total lots │ 16807 │
|
|
||||||
│ Valid lots (with auction_id) │ 16794 │
|
|
||||||
│ Invalid lots (missing auction_id) │ 13 │
|
|
||||||
└──────────────────────────────────┴────────┘
|
|
||||||
|
|
||||||
Analyzing data to clean up...
|
|
||||||
→ Invalid lots to delete: 13
|
|
||||||
→ Orphaned images to delete: 0
|
|
||||||
|
|
||||||
This will permanently delete the above records.
|
|
||||||
Continue? (y/N) y
|
|
||||||
|
|
||||||
Cleaning up database...
|
|
||||||
[1/2] Deleting invalid lots...
|
|
||||||
✓ Deleted 13 invalid lots
|
|
||||||
[2/2] Deleting orphaned images...
|
|
||||||
✓ Deleted 0 orphaned images
|
|
||||||
[3/3] Compacting database...
|
|
||||||
✓ Database compacted
|
|
||||||
|
|
||||||
Final database state:
|
|
||||||
┌───────────────┬────────┐
|
|
||||||
│ metric │ count │
|
|
||||||
├───────────────┼────────┤
|
|
||||||
│ Total lots │ 16794 │
|
|
||||||
│ Total images │ 536502 │
|
|
||||||
└───────────────┴────────┘
|
|
||||||
|
|
||||||
Database size: 8.9G
|
|
||||||
```
|
|
||||||
|
|
||||||
## Solution 3: SQL Manual Cleanup
|
|
||||||
|
|
||||||
If you prefer to manually clean using SQL:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
-- Backup first!
|
|
||||||
-- cp cache.db cache.db.backup
|
|
||||||
|
|
||||||
-- Check invalid entries
|
|
||||||
SELECT COUNT(*), 'Invalid' as type
|
|
||||||
FROM lots
|
|
||||||
WHERE auction_id IS NULL OR auction_id = ''
|
|
||||||
UNION ALL
|
|
||||||
SELECT COUNT(*), 'Valid'
|
|
||||||
FROM lots
|
|
||||||
WHERE auction_id IS NOT NULL AND auction_id != '';
|
|
||||||
|
|
||||||
-- Delete invalid lots
|
|
||||||
DELETE FROM lots
|
|
||||||
WHERE auction_id IS NULL OR auction_id = '';
|
|
||||||
|
|
||||||
-- Delete orphaned images
|
|
||||||
DELETE FROM images
|
|
||||||
WHERE lot_id NOT IN (SELECT lot_id FROM lots);
|
|
||||||
|
|
||||||
-- Compact database
|
|
||||||
VACUUM;
|
|
||||||
```
|
|
||||||
|
|
||||||
## Prevention: Production Database Cleanup
|
|
||||||
|
|
||||||
To prevent these invalid entries from accumulating on production, you can:
|
|
||||||
|
|
||||||
1. **Clean production database** (one-time):
|
|
||||||
```bash
|
|
||||||
ssh tour@athena.lan
|
|
||||||
docker run --rm -v shared-auction-data:/data alpine sqlite3 /data/cache.db "DELETE FROM lots WHERE auction_id IS NULL OR auction_id = '';"
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Update scraper** to ensure all lots have auction_id
|
|
||||||
3. **Add validation** in scraper to reject lots without auction_id
|
|
||||||
|
|
||||||
## When to Clean
|
|
||||||
|
|
||||||
### Immediately if:
|
|
||||||
- ❌ Seeing `NullPointerException` in analytics
|
|
||||||
- ❌ Dashboard insights failing
|
|
||||||
- ❌ Country distribution not working
|
|
||||||
|
|
||||||
### Periodically:
|
|
||||||
- 🔄 After syncing from production (if production has invalid data)
|
|
||||||
- 🔄 Weekly/monthly maintenance
|
|
||||||
- 🔄 Before major testing or demos
|
|
||||||
|
|
||||||
## Recommendation
|
|
||||||
|
|
||||||
**Use Solution 1 (Clean Sync)** for simplicity:
|
|
||||||
- ✅ Guarantees clean state
|
|
||||||
- ✅ No manual SQL needed
|
|
||||||
- ✅ Shows data quality report
|
|
||||||
- ✅ Safe (automatic backup)
|
|
||||||
|
|
||||||
The 13 invalid entries are from an old scraper and represent only 0.08% of data, so cleaning them up has minimal impact but prevents future errors.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Related Documentation**:
|
|
||||||
- [Sync Scripts README](../scripts/README.md)
|
|
||||||
- [Data Sync Setup](DATA_SYNC_SETUP.md)
|
|
||||||
- [Database Architecture](../wiki/DATABASE_ARCHITECTURE.md)
|
|
||||||
@@ -1,584 +0,0 @@
|
|||||||
# Implementation Complete ✅
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
All requirements have been successfully implemented:
|
|
||||||
|
|
||||||
### ✅ 1. Test Libraries Added
|
|
||||||
|
|
||||||
**pom.xml updated with:**
|
|
||||||
- JUnit 5 (5.10.1) - Testing framework
|
|
||||||
- Mockito Core (5.8.0) - Mocking framework
|
|
||||||
- Mockito JUnit Jupiter (5.8.0) - JUnit integration
|
|
||||||
- AssertJ (3.24.2) - Fluent assertions
|
|
||||||
|
|
||||||
**Run tests:**
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 2. Paths Configured for Windows
|
|
||||||
|
|
||||||
**Database:**
|
|
||||||
```
|
|
||||||
C:\mnt\okcomputer\output\cache.db
|
|
||||||
```
|
|
||||||
|
|
||||||
**Images:**
|
|
||||||
```
|
|
||||||
C:\mnt\okcomputer\output\images\{saleId}\{lotId}\
|
|
||||||
```
|
|
||||||
|
|
||||||
**Files Updated:**
|
|
||||||
- `Main.java:31` - Database path
|
|
||||||
- `ImageProcessingService.java:52` - Image storage path
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 3. Comprehensive Test Suite (90 Tests)
|
|
||||||
|
|
||||||
| Test File | Tests | Coverage |
|
|
||||||
|-----------|-------|----------|
|
|
||||||
| ScraperDataAdapterTest | 13 | Data transformation, ID parsing, currency |
|
|
||||||
| DatabaseServiceTest | 15 | CRUD operations, concurrency |
|
|
||||||
| ImageProcessingServiceTest | 11 | Download, detection, errors |
|
|
||||||
| ObjectDetectionServiceTest | 10 | YOLO initialization, detection |
|
|
||||||
| NotificationServiceTest | 19 | Desktop/email, priorities |
|
|
||||||
| TroostwijkMonitorTest | 12 | Orchestration, monitoring |
|
|
||||||
| IntegrationTest | 10 | End-to-end workflows |
|
|
||||||
| **TOTAL** | **90** | **Complete system** |
|
|
||||||
|
|
||||||
**Documentation:** See `TEST_SUITE_SUMMARY.md`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 4. Workflow Integration & Orchestration
|
|
||||||
|
|
||||||
**New Component:** `WorkflowOrchestrator.java`
|
|
||||||
|
|
||||||
**4 Automated Workflows:**
|
|
||||||
|
|
||||||
1. **Scraper Data Import** (every 30 min)
|
|
||||||
- Imports auctions, lots, image URLs
|
|
||||||
- Sends notifications for significant data
|
|
||||||
|
|
||||||
2. **Image Processing** (every 1 hour)
|
|
||||||
- Downloads images
|
|
||||||
- Runs YOLO object detection
|
|
||||||
- Saves labels to database
|
|
||||||
|
|
||||||
3. **Bid Monitoring** (every 15 min)
|
|
||||||
- Checks for bid changes
|
|
||||||
- Sends notifications
|
|
||||||
|
|
||||||
4. **Closing Alerts** (every 5 min)
|
|
||||||
- Finds lots closing soon
|
|
||||||
- Sends high-priority notifications
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 5. Running Modes
|
|
||||||
|
|
||||||
**Main.java now supports 4 modes:**
|
|
||||||
|
|
||||||
#### Mode 1: workflow (Default - Recommended)
|
|
||||||
```bash
|
|
||||||
java -jar troostwijk-monitor.jar workflow
|
|
||||||
# OR
|
|
||||||
run-workflow.bat
|
|
||||||
```
|
|
||||||
- Runs all workflows continuously
|
|
||||||
- Built-in scheduling
|
|
||||||
- Best for production
|
|
||||||
|
|
||||||
#### Mode 2: once (For Cron/Task Scheduler)
|
|
||||||
```bash
|
|
||||||
java -jar troostwijk-monitor.jar once
|
|
||||||
# OR
|
|
||||||
run-once.bat
|
|
||||||
```
|
|
||||||
- Runs complete workflow once
|
|
||||||
- Exits after completion
|
|
||||||
- Perfect for external schedulers
|
|
||||||
|
|
||||||
#### Mode 3: legacy (Backward Compatible)
|
|
||||||
```bash
|
|
||||||
java -jar troostwijk-monitor.jar legacy
|
|
||||||
```
|
|
||||||
- Original monitoring approach
|
|
||||||
- Kept for compatibility
|
|
||||||
|
|
||||||
#### Mode 4: status (Quick Check)
|
|
||||||
```bash
|
|
||||||
java -jar troostwijk-monitor.jar status
|
|
||||||
# OR
|
|
||||||
check-status.bat
|
|
||||||
```
|
|
||||||
- Shows current status
|
|
||||||
- Exits immediately
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 6. Windows Scheduling Scripts
|
|
||||||
|
|
||||||
**Batch Scripts Created:**
|
|
||||||
|
|
||||||
1. **run-workflow.bat**
|
|
||||||
- Starts workflow mode
|
|
||||||
- Continuous operation
|
|
||||||
- For manual/startup use
|
|
||||||
|
|
||||||
2. **run-once.bat**
|
|
||||||
- Single execution
|
|
||||||
- For Task Scheduler
|
|
||||||
- Exit code support
|
|
||||||
|
|
||||||
3. **check-status.bat**
|
|
||||||
- Quick status check
|
|
||||||
- Shows database stats
|
|
||||||
|
|
||||||
**PowerShell Automation:**
|
|
||||||
|
|
||||||
4. **setup-windows-task.ps1**
|
|
||||||
- Creates Task Scheduler tasks automatically
|
|
||||||
- Sets up 2 scheduled tasks:
|
|
||||||
- Workflow runner (every 30 min)
|
|
||||||
- Status checker (every 6 hours)
|
|
||||||
|
|
||||||
**Usage:**
|
|
||||||
```powershell
|
|
||||||
# Run as Administrator
|
|
||||||
.\setup-windows-task.ps1
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 7. Event-Driven Triggers
|
|
||||||
|
|
||||||
**WorkflowOrchestrator supports event-driven execution:**
|
|
||||||
|
|
||||||
```java
|
|
||||||
// 1. New auction discovered
|
|
||||||
orchestrator.onNewAuctionDiscovered(auctionInfo);
|
|
||||||
|
|
||||||
// 2. Bid change detected
|
|
||||||
orchestrator.onBidChange(lot, previousBid, newBid);
|
|
||||||
|
|
||||||
// 3. Objects detected in image
|
|
||||||
orchestrator.onObjectsDetected(lotId, labels);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Benefits:**
|
|
||||||
- React immediately to important events
|
|
||||||
- No waiting for next scheduled run
|
|
||||||
- Flexible integration with external systems
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### ✅ 8. Comprehensive Documentation
|
|
||||||
|
|
||||||
**Documentation Created:**
|
|
||||||
|
|
||||||
1. **TEST_SUITE_SUMMARY.md**
|
|
||||||
- Complete test coverage overview
|
|
||||||
- 90 test cases documented
|
|
||||||
- Running instructions
|
|
||||||
- Test patterns explained
|
|
||||||
|
|
||||||
2. **WORKFLOW_GUIDE.md**
|
|
||||||
- Complete workflow integration guide
|
|
||||||
- Running modes explained
|
|
||||||
- Windows Task Scheduler setup
|
|
||||||
- Event-driven triggers
|
|
||||||
- Configuration options
|
|
||||||
- Troubleshooting guide
|
|
||||||
- Advanced integration examples
|
|
||||||
|
|
||||||
3. **README.md** (Updated)
|
|
||||||
- System architecture diagram
|
|
||||||
- Integration flow
|
|
||||||
- User interaction points
|
|
||||||
- Value estimation pipeline
|
|
||||||
- Integration hooks table
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### Option A: Continuous Operation (Recommended)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build
|
|
||||||
mvn clean package
|
|
||||||
|
|
||||||
# Run workflow mode
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar workflow
|
|
||||||
|
|
||||||
# Or use batch script
|
|
||||||
run-workflow.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
**What runs:**
|
|
||||||
- ✅ Data import every 30 min
|
|
||||||
- ✅ Image processing every 1 hour
|
|
||||||
- ✅ Bid monitoring every 15 min
|
|
||||||
- ✅ Closing alerts every 5 min
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option B: Windows Task Scheduler
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
# 1. Build JAR
|
|
||||||
mvn clean package
|
|
||||||
|
|
||||||
# 2. Setup scheduled tasks (run as Admin)
|
|
||||||
.\setup-windows-task.ps1
|
|
||||||
|
|
||||||
# Done! Workflow runs automatically every 30 minutes
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option C: Manual/Cron Execution
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run once
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar once
|
|
||||||
|
|
||||||
# Or
|
|
||||||
run-once.bat
|
|
||||||
|
|
||||||
# Schedule externally (Windows Task Scheduler, cron, etc.)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture Overview
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ External Scraper (Python) │
|
|
||||||
│ Populates: auctions, lots, images tables │
|
|
||||||
└─────────────────────────┬───────────────────────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ SQLite Database │
|
|
||||||
│ C:\mnt\okcomputer\output\cache.db │
|
|
||||||
└─────────────────────────┬───────────────────────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ WorkflowOrchestrator (This System) │
|
|
||||||
│ ┌─────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ Workflow 1: Scraper Import (every 30 min) │ │
|
|
||||||
│ │ Workflow 2: Image Processing (every 1 hour) │ │
|
|
||||||
│ │ Workflow 3: Bid Monitoring (every 15 min) │ │
|
|
||||||
│ │ Workflow 4: Closing Alerts (every 5 min) │ │
|
|
||||||
│ └─────────────────────────────────────────────────────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ┌─────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ ImageProcessingService │ │
|
|
||||||
│ │ - Downloads images │ │
|
|
||||||
│ │ - Stores: C:\mnt\okcomputer\output\images\ │ │
|
|
||||||
│ └─────────────────────────────────────────────────────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ┌─────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ ObjectDetectionService (YOLO) │ │
|
|
||||||
│ │ - Detects objects in images │ │
|
|
||||||
│ │ - Labels: car, truck, machinery, etc. │ │
|
|
||||||
│ └─────────────────────────────────────────────────────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ┌─────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ NotificationService │ │
|
|
||||||
│ │ - Desktop notifications (Windows tray) │ │
|
|
||||||
│ │ - Email notifications (Gmail SMTP) │ │
|
|
||||||
│ └─────────────────────────────────────────────────────┘ │
|
|
||||||
└─────────────────────────┬───────────────────────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ User Notifications │
|
|
||||||
│ - Bid changes │
|
|
||||||
│ - Closing alerts │
|
|
||||||
│ - Object detection results │
|
|
||||||
│ - Value estimates (future) │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Integration Points
|
|
||||||
|
|
||||||
### 1. Database Integration
|
|
||||||
- **Read:** Auctions, lots, image URLs from external scraper
|
|
||||||
- **Write:** Processed images, object labels, notifications
|
|
||||||
|
|
||||||
### 2. File System Integration
|
|
||||||
- **Read:** YOLO model files (models/)
|
|
||||||
- **Write:** Downloaded images (C:\mnt\okcomputer\output\images\)
|
|
||||||
|
|
||||||
### 3. External Scraper Integration
|
|
||||||
- **Mode:** Shared SQLite database
|
|
||||||
- **Frequency:** Scraper populates, monitor enriches
|
|
||||||
|
|
||||||
### 4. Notification Integration
|
|
||||||
- **Desktop:** Windows system tray
|
|
||||||
- **Email:** Gmail SMTP (optional)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
### Run All Tests
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Specific Test
|
|
||||||
```bash
|
|
||||||
mvn test -Dtest=IntegrationTest
|
|
||||||
mvn test -Dtest=WorkflowOrchestratorTest
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Coverage
|
|
||||||
```bash
|
|
||||||
mvn jacoco:prepare-agent test jacoco:report
|
|
||||||
# Report: target/site/jacoco/index.html
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Windows (cmd)
|
|
||||||
set DATABASE_FILE=C:\mnt\okcomputer\output\cache.db
|
|
||||||
set NOTIFICATION_CONFIG=desktop
|
|
||||||
|
|
||||||
# Windows (PowerShell)
|
|
||||||
$env:DATABASE_FILE="C:\mnt\okcomputer\output\cache.db"
|
|
||||||
$env:NOTIFICATION_CONFIG="desktop"
|
|
||||||
|
|
||||||
# For email notifications
|
|
||||||
set NOTIFICATION_CONFIG=smtp:your@gmail.com:app_password:recipient@example.com
|
|
||||||
```
|
|
||||||
|
|
||||||
### Code Configuration
|
|
||||||
|
|
||||||
**Database Path** (`Main.java:31`):
|
|
||||||
```java
|
|
||||||
String databaseFile = System.getenv().getOrDefault(
|
|
||||||
"DATABASE_FILE",
|
|
||||||
"C:\\mnt\\okcomputer\\output\\cache.db"
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Workflow Schedules** (`WorkflowOrchestrator.java`):
|
|
||||||
```java
|
|
||||||
scheduleScraperDataImport(); // Line 65 - every 30 min
|
|
||||||
scheduleImageProcessing(); // Line 95 - every 1 hour
|
|
||||||
scheduleBidMonitoring(); // Line 180 - every 15 min
|
|
||||||
scheduleClosingAlerts(); // Line 215 - every 5 min
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
### Check Status
|
|
||||||
```bash
|
|
||||||
java -jar troostwijk-monitor.jar status
|
|
||||||
```
|
|
||||||
|
|
||||||
**Output:**
|
|
||||||
```
|
|
||||||
📊 Workflow Status:
|
|
||||||
Running: Yes/No
|
|
||||||
Auctions: 25
|
|
||||||
Lots: 150
|
|
||||||
Images: 300
|
|
||||||
Closing soon (< 30 min): 5
|
|
||||||
```
|
|
||||||
|
|
||||||
### View Logs
|
|
||||||
|
|
||||||
Workflows print detailed logs:
|
|
||||||
```
|
|
||||||
📥 [WORKFLOW 1] Importing scraper data...
|
|
||||||
→ Imported 5 auctions
|
|
||||||
→ Imported 25 lots
|
|
||||||
✓ Scraper import completed in 1250ms
|
|
||||||
|
|
||||||
🖼️ [WORKFLOW 2] Processing pending images...
|
|
||||||
→ Processing 50 images
|
|
||||||
✓ Processed 50 images, detected objects in 12
|
|
||||||
|
|
||||||
💰 [WORKFLOW 3] Monitoring bids...
|
|
||||||
→ Checking 150 active lots
|
|
||||||
✓ Bid monitoring completed in 250ms
|
|
||||||
|
|
||||||
⏰ [WORKFLOW 4] Checking closing times...
|
|
||||||
→ Sent 3 closing alerts
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
### Immediate Actions
|
|
||||||
|
|
||||||
1. **Build the project:**
|
|
||||||
```bash
|
|
||||||
mvn clean package
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Run tests:**
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Choose execution mode:**
|
|
||||||
- **Continuous:** `run-workflow.bat`
|
|
||||||
- **Scheduled:** `.\setup-windows-task.ps1` (as Admin)
|
|
||||||
- **Manual:** `run-once.bat`
|
|
||||||
|
|
||||||
4. **Verify setup:**
|
|
||||||
```bash
|
|
||||||
check-status.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
### Future Enhancements
|
|
||||||
|
|
||||||
1. **Value Estimation Algorithm**
|
|
||||||
- Use detected objects to estimate lot value
|
|
||||||
- Historical price analysis
|
|
||||||
- Market trends integration
|
|
||||||
|
|
||||||
2. **Machine Learning**
|
|
||||||
- Train custom YOLO model for auction items
|
|
||||||
- Price prediction based on images
|
|
||||||
- Automatic categorization
|
|
||||||
|
|
||||||
3. **Web Dashboard**
|
|
||||||
- Real-time monitoring
|
|
||||||
- Manual bid placement
|
|
||||||
- Value estimate approval
|
|
||||||
|
|
||||||
4. **API Integration**
|
|
||||||
- Direct Troostwijk API integration
|
|
||||||
- Real-time bid updates
|
|
||||||
- Automatic bid placement
|
|
||||||
|
|
||||||
5. **Advanced Notifications**
|
|
||||||
- SMS notifications (Twilio)
|
|
||||||
- Push notifications (Firebase)
|
|
||||||
- Slack/Discord integration
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files Created/Modified
|
|
||||||
|
|
||||||
### Core Implementation
|
|
||||||
- ✅ `WorkflowOrchestrator.java` - Workflow coordination
|
|
||||||
- ✅ `Main.java` - Updated with 4 running modes
|
|
||||||
- ✅ `ImageProcessingService.java` - Windows paths
|
|
||||||
- ✅ `pom.xml` - Test libraries added
|
|
||||||
|
|
||||||
### Test Suite (90 tests)
|
|
||||||
- ✅ `ScraperDataAdapterTest.java` (13 tests)
|
|
||||||
- ✅ `DatabaseServiceTest.java` (15 tests)
|
|
||||||
- ✅ `ImageProcessingServiceTest.java` (11 tests)
|
|
||||||
- ✅ `ObjectDetectionServiceTest.java` (10 tests)
|
|
||||||
- ✅ `NotificationServiceTest.java` (19 tests)
|
|
||||||
- ✅ `TroostwijkMonitorTest.java` (12 tests)
|
|
||||||
- ✅ `IntegrationTest.java` (10 tests)
|
|
||||||
|
|
||||||
### Windows Scripts
|
|
||||||
- ✅ `run-workflow.bat` - Workflow mode runner
|
|
||||||
- ✅ `run-once.bat` - Once mode runner
|
|
||||||
- ✅ `check-status.bat` - Status checker
|
|
||||||
- ✅ `setup-windows-task.ps1` - Task Scheduler setup
|
|
||||||
|
|
||||||
### Documentation
|
|
||||||
- ✅ `TEST_SUITE_SUMMARY.md` - Test coverage
|
|
||||||
- ✅ `WORKFLOW_GUIDE.md` - Complete workflow guide
|
|
||||||
- ✅ `README.md` - Updated with diagrams
|
|
||||||
- ✅ `IMPLEMENTATION_COMPLETE.md` - This file
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Support & Troubleshooting
|
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
**1. Tests failing**
|
|
||||||
```bash
|
|
||||||
# Ensure Maven dependencies downloaded
|
|
||||||
mvn clean install
|
|
||||||
|
|
||||||
# Run tests with debug info
|
|
||||||
mvn test -X
|
|
||||||
```
|
|
||||||
|
|
||||||
**2. Workflow not starting**
|
|
||||||
```bash
|
|
||||||
# Check if JAR was built
|
|
||||||
dir target\*jar-with-dependencies.jar
|
|
||||||
|
|
||||||
# Rebuild if missing
|
|
||||||
mvn clean package
|
|
||||||
```
|
|
||||||
|
|
||||||
**3. Database not found**
|
|
||||||
```bash
|
|
||||||
# Check path exists
|
|
||||||
dir C:\mnt\okcomputer\output\
|
|
||||||
|
|
||||||
# Create directory if missing
|
|
||||||
mkdir C:\mnt\okcomputer\output
|
|
||||||
```
|
|
||||||
|
|
||||||
**4. Images not downloading**
|
|
||||||
- Check internet connection
|
|
||||||
- Verify image URLs in database
|
|
||||||
- Check Windows Firewall settings
|
|
||||||
|
|
||||||
### Getting Help
|
|
||||||
|
|
||||||
1. Review documentation:
|
|
||||||
- `TEST_SUITE_SUMMARY.md` for tests
|
|
||||||
- `WORKFLOW_GUIDE.md` for workflows
|
|
||||||
- `README.md` for architecture
|
|
||||||
|
|
||||||
2. Check status:
|
|
||||||
```bash
|
|
||||||
check-status.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Review logs in console output
|
|
||||||
|
|
||||||
4. Run tests to verify components:
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
✅ **Test libraries added** (JUnit, Mockito, AssertJ)
|
|
||||||
✅ **90 comprehensive tests created**
|
|
||||||
✅ **Workflow orchestration implemented**
|
|
||||||
✅ **4 running modes** (workflow, once, legacy, status)
|
|
||||||
✅ **Windows scheduling scripts** (batch + PowerShell)
|
|
||||||
✅ **Event-driven triggers** (3 event types)
|
|
||||||
✅ **Complete documentation** (3 guide files)
|
|
||||||
✅ **Windows paths configured** (database + images)
|
|
||||||
|
|
||||||
**The system is production-ready and fully tested! 🎉**
|
|
||||||
@@ -1,478 +0,0 @@
|
|||||||
# Integration Guide: Troostwijk Monitor ↔ Scraper
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This document describes how **Troostwijk Monitor** (this Java project) integrates with the **ARCHITECTURE-TROOSTWIJK-SCRAPER** (Python scraper process).
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ ARCHITECTURE-TROOSTWIJK-SCRAPER (Python) │
|
|
||||||
│ │
|
|
||||||
│ • Discovers auctions from website │
|
|
||||||
│ • Scrapes lot details via Playwright │
|
|
||||||
│ • Parses __NEXT_DATA__ JSON │
|
|
||||||
│ • Stores image URLs (not downloads) │
|
|
||||||
│ │
|
|
||||||
│ ↓ Writes to │
|
|
||||||
└─────────┼───────────────────────────────────────────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ SHARED SQLite DATABASE │
|
|
||||||
│ (troostwijk.db) │
|
|
||||||
│ │
|
|
||||||
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
|
|
||||||
│ │ auctions │ │ lots │ │ images │ │
|
|
||||||
│ │ (Scraper) │ │ (Scraper) │ │ (Both) │ │
|
|
||||||
│ └────────────────┘ └────────────────┘ └────────────────┘ │
|
|
||||||
│ │
|
|
||||||
│ ↑ Reads from ↓ Writes to │
|
|
||||||
└─────────┼──────────────────────────────┼──────────────────────┘
|
|
||||||
│ │
|
|
||||||
│ ▼
|
|
||||||
┌─────────┴──────────────────────────────────────────────────────┐
|
|
||||||
│ TROOSTWIJK MONITOR (Java - This Project) │
|
|
||||||
│ │
|
|
||||||
│ • Reads auction/lot data from database │
|
|
||||||
│ • Downloads images from URLs │
|
|
||||||
│ • Runs YOLO object detection │
|
|
||||||
│ • Monitors bid changes │
|
|
||||||
│ • Sends notifications │
|
|
||||||
└─────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Database Schema Mapping
|
|
||||||
|
|
||||||
### Scraper Schema → Monitor Schema
|
|
||||||
|
|
||||||
The scraper and monitor use **slightly different schemas** that need to be reconciled:
|
|
||||||
|
|
||||||
| Scraper Table | Monitor Table | Integration Notes |
|
|
||||||
|---------------|---------------|-----------------------------------------------|
|
|
||||||
| `auctions` | `auctions` | ✅ **Compatible** - same structure |
|
|
||||||
| `lots` | `lots` | ⚠️ **Needs mapping** - field name differences |
|
|
||||||
| `images` | `images` | ⚠️ **Partial overlap** - different purposes |
|
|
||||||
| `cache` | N/A | ❌ Monitor doesn't use cache |
|
|
||||||
|
|
||||||
### Field Mapping: `auctions` Table
|
|
||||||
|
|
||||||
| Scraper Field | Monitor Field | Notes |
|
|
||||||
|--------------------------|-------------------------------|---------------------------------------------------------------------|
|
|
||||||
| `auction_id` (TEXT) | `auction_id` (INTEGER) | ⚠️ **TYPE MISMATCH** - Scraper uses "A7-39813", Monitor expects INT |
|
|
||||||
| `url` | `url` | ✅ Compatible |
|
|
||||||
| `title` | `title` | ✅ Compatible |
|
|
||||||
| `location` | `location`, `city`, `country` | ⚠️ Monitor splits into 3 fields |
|
|
||||||
| `lots_count` | `lot_count` | ⚠️ Name difference |
|
|
||||||
| `first_lot_closing_time` | `closing_time` | ⚠️ Name difference |
|
|
||||||
| `scraped_at` | `discovered_at` | ⚠️ Name + type difference (TEXT vs INTEGER timestamp) |
|
|
||||||
|
|
||||||
### Field Mapping: `lots` Table
|
|
||||||
|
|
||||||
| Scraper Field | Monitor Field | Notes |
|
|
||||||
|----------------------|----------------------|--------------------------------------------------|
|
|
||||||
| `lot_id` (TEXT) | `lot_id` (INTEGER) | ⚠️ **TYPE MISMATCH** - "A1-28505-5" vs INT |
|
|
||||||
| `auction_id` | `sale_id` | ⚠️ Different name |
|
|
||||||
| `url` | `url` | ✅ Compatible |
|
|
||||||
| `title` | `title` | ✅ Compatible |
|
|
||||||
| `current_bid` (TEXT) | `current_bid` (REAL) | ⚠️ **TYPE MISMATCH** - "€123.45" vs 123.45 |
|
|
||||||
| `bid_count` | N/A | ℹ️ Monitor doesn't track |
|
|
||||||
| `closing_time` | `closing_time` | ⚠️ Format difference (TEXT vs LocalDateTime) |
|
|
||||||
| `viewing_time` | N/A | ℹ️ Monitor doesn't track |
|
|
||||||
| `pickup_date` | N/A | ℹ️ Monitor doesn't track |
|
|
||||||
| `location` | N/A | ℹ️ Monitor doesn't track lot location separately |
|
|
||||||
| `description` | `description` | ✅ Compatible |
|
|
||||||
| `category` | `category` | ✅ Compatible |
|
|
||||||
| N/A | `manufacturer` | ℹ️ Monitor has additional field |
|
|
||||||
| N/A | `type` | ℹ️ Monitor has additional field |
|
|
||||||
| N/A | `year` | ℹ️ Monitor has additional field |
|
|
||||||
| N/A | `currency` | ℹ️ Monitor has additional field |
|
|
||||||
| N/A | `closing_notified` | ℹ️ Monitor tracking field |
|
|
||||||
|
|
||||||
### Field Mapping: `images` Table
|
|
||||||
|
|
||||||
| Scraper Field | Monitor Field | Notes |
|
|
||||||
|------------------------|--------------------------|----------------------------------------|
|
|
||||||
| `id` | `id` | ✅ Compatible |
|
|
||||||
| `lot_id` | `lot_id` | ⚠️ Type difference (TEXT vs INTEGER) |
|
|
||||||
| `url` | `url` | ✅ Compatible |
|
|
||||||
| `local_path` | `Local_path` | ⚠️ Different name |
|
|
||||||
| `downloaded` (INTEGER) | N/A | ℹ️ Monitor uses `processed_at` instead |
|
|
||||||
| N/A | `labels` (TEXT) | ℹ️ Monitor adds detected objects |
|
|
||||||
| N/A | `processed_at` (INTEGER) | ℹ️ Monitor tracking field |
|
|
||||||
|
|
||||||
## Integration Options
|
|
||||||
|
|
||||||
### Option 1: Database Schema Adapter (Recommended)
|
|
||||||
|
|
||||||
Create a compatibility layer that transforms scraper data to monitor format.
|
|
||||||
|
|
||||||
**Implementation:**
|
|
||||||
```java
|
|
||||||
// Add to DatabaseService.java
|
|
||||||
class ScraperDataAdapter {
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Imports auction from scraper format to monitor format
|
|
||||||
*/
|
|
||||||
static AuctionInfo fromScraperAuction(ResultSet rs) throws SQLException {
|
|
||||||
// Parse "A7-39813" → 39813
|
|
||||||
String auctionIdStr = rs.getString("auction_id");
|
|
||||||
int auctionId = extractNumericId(auctionIdStr);
|
|
||||||
|
|
||||||
// Split "Cluj-Napoca, RO" → city="Cluj-Napoca", country="RO"
|
|
||||||
String location = rs.getString("location");
|
|
||||||
String[] parts = location.split(",\\s*");
|
|
||||||
String city = parts.length > 0 ? parts[0] : "";
|
|
||||||
String country = parts.length > 1 ? parts[1] : "";
|
|
||||||
|
|
||||||
return new AuctionInfo(
|
|
||||||
auctionId,
|
|
||||||
rs.getString("title"),
|
|
||||||
location,
|
|
||||||
city,
|
|
||||||
country,
|
|
||||||
rs.getString("url"),
|
|
||||||
extractTypePrefix(auctionIdStr), // "A7-39813" → "A7"
|
|
||||||
rs.getInt("lots_count"),
|
|
||||||
parseTimestamp(rs.getString("first_lot_closing_time"))
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Imports lot from scraper format to monitor format
|
|
||||||
*/
|
|
||||||
static Lot fromScraperLot(ResultSet rs) throws SQLException {
|
|
||||||
// Parse "A1-28505-5" → 285055 (combine numbers)
|
|
||||||
String lotIdStr = rs.getString("lot_id");
|
|
||||||
int lotId = extractNumericId(lotIdStr);
|
|
||||||
|
|
||||||
// Parse "A7-39813" → 39813
|
|
||||||
String auctionIdStr = rs.getString("auction_id");
|
|
||||||
int saleId = extractNumericId(auctionIdStr);
|
|
||||||
|
|
||||||
// Parse "€123.45" → 123.45
|
|
||||||
String currentBidStr = rs.getString("current_bid");
|
|
||||||
double currentBid = parseBid(currentBidStr);
|
|
||||||
|
|
||||||
return new Lot(
|
|
||||||
saleId,
|
|
||||||
lotId,
|
|
||||||
rs.getString("title"),
|
|
||||||
rs.getString("description"),
|
|
||||||
"", // manufacturer - not in scraper
|
|
||||||
"", // type - not in scraper
|
|
||||||
0, // year - not in scraper
|
|
||||||
rs.getString("category"),
|
|
||||||
currentBid,
|
|
||||||
"EUR", // currency - inferred from €
|
|
||||||
rs.getString("url"),
|
|
||||||
parseTimestamp(rs.getString("closing_time")),
|
|
||||||
false // not yet notified
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
private static int extractNumericId(String id) {
|
|
||||||
// "A7-39813" → 39813
|
|
||||||
// "A1-28505-5" → 285055
|
|
||||||
return Integer.parseInt(id.replaceAll("[^0-9]", ""));
|
|
||||||
}
|
|
||||||
|
|
||||||
private static String extractTypePrefix(String id) {
|
|
||||||
// "A7-39813" → "A7"
|
|
||||||
int dashIndex = id.indexOf('-');
|
|
||||||
return dashIndex > 0 ? id.substring(0, dashIndex) : "";
|
|
||||||
}
|
|
||||||
|
|
||||||
private static double parseBid(String bid) {
|
|
||||||
// "€123.45" → 123.45
|
|
||||||
// "No bids" → 0.0
|
|
||||||
if (bid == null || bid.contains("No")) return 0.0;
|
|
||||||
return Double.parseDouble(bid.replaceAll("[^0-9.]", ""));
|
|
||||||
}
|
|
||||||
|
|
||||||
private static LocalDateTime parseTimestamp(String timestamp) {
|
|
||||||
if (timestamp == null) return null;
|
|
||||||
// Parse scraper's timestamp format
|
|
||||||
return LocalDateTime.parse(timestamp);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option 2: Unified Schema (Better Long-term)
|
|
||||||
|
|
||||||
Modify **both** scraper and monitor to use a unified schema.
|
|
||||||
|
|
||||||
**Create**: `SHARED_SCHEMA.sql`
|
|
||||||
```sql
|
|
||||||
-- Unified schema that both projects use
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS auctions (
|
|
||||||
auction_id TEXT PRIMARY KEY, -- Use TEXT to support "A7-39813"
|
|
||||||
auction_id_numeric INTEGER, -- For monitor's integer needs
|
|
||||||
title TEXT NOT NULL,
|
|
||||||
location TEXT, -- Full: "Cluj-Napoca, RO"
|
|
||||||
city TEXT, -- Parsed: "Cluj-Napoca"
|
|
||||||
country TEXT, -- Parsed: "RO"
|
|
||||||
url TEXT NOT NULL,
|
|
||||||
type TEXT, -- "A7", "A1"
|
|
||||||
lot_count INTEGER DEFAULT 0,
|
|
||||||
closing_time TEXT, -- ISO 8601 format
|
|
||||||
scraped_at INTEGER, -- Unix timestamp
|
|
||||||
discovered_at INTEGER -- Unix timestamp (same as scraped_at)
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS lots (
|
|
||||||
lot_id TEXT PRIMARY KEY, -- Use TEXT: "A1-28505-5"
|
|
||||||
lot_id_numeric INTEGER, -- For monitor's integer needs
|
|
||||||
auction_id TEXT, -- FK: "A7-39813"
|
|
||||||
sale_id INTEGER, -- For monitor (same as auction_id_numeric)
|
|
||||||
title TEXT,
|
|
||||||
description TEXT,
|
|
||||||
manufacturer TEXT,
|
|
||||||
type TEXT,
|
|
||||||
year INTEGER,
|
|
||||||
category TEXT,
|
|
||||||
current_bid_text TEXT, -- "€123.45" or "No bids"
|
|
||||||
current_bid REAL, -- 123.45
|
|
||||||
bid_count INTEGER,
|
|
||||||
currency TEXT DEFAULT 'EUR',
|
|
||||||
url TEXT UNIQUE,
|
|
||||||
closing_time TEXT,
|
|
||||||
viewing_time TEXT,
|
|
||||||
pickup_date TEXT,
|
|
||||||
location TEXT,
|
|
||||||
closing_notified INTEGER DEFAULT 0,
|
|
||||||
scraped_at TEXT,
|
|
||||||
FOREIGN KEY (auction_id) REFERENCES auctions(auction_id)
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS images (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT, -- FK: "A1-28505-5"
|
|
||||||
url TEXT, -- Image URL from website
|
|
||||||
local_path TEXT, -- Local path after download
|
|
||||||
labels TEXT, -- Detected objects (comma-separated)
|
|
||||||
downloaded INTEGER DEFAULT 0, -- 0=pending, 1=downloaded
|
|
||||||
processed_at INTEGER, -- Unix timestamp when processed
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Indexes
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_auctions_country ON auctions(country);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_lots_auction_id ON lots(auction_id);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_images_lot_id ON images(lot_id);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_images_downloaded ON images(downloaded);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option 3: API Integration (Most Flexible)
|
|
||||||
|
|
||||||
Have the scraper expose a REST API for the monitor to query.
|
|
||||||
|
|
||||||
```python
|
|
||||||
# In scraper: Add Flask API endpoint
|
|
||||||
@app.route('/api/auctions', methods=['GET'])
|
|
||||||
def get_auctions():
|
|
||||||
"""Returns auctions in monitor-compatible format"""
|
|
||||||
conn = sqlite3.connect(CACHE_DB)
|
|
||||||
cursor = conn.cursor()
|
|
||||||
cursor.execute("SELECT * FROM auctions WHERE location LIKE '%NL%'")
|
|
||||||
|
|
||||||
auctions = []
|
|
||||||
for row in cursor.fetchall():
|
|
||||||
auctions.append({
|
|
||||||
'auctionId': extract_numeric_id(row[0]),
|
|
||||||
'title': row[2],
|
|
||||||
'location': row[3],
|
|
||||||
'city': row[3].split(',')[0] if row[3] else '',
|
|
||||||
'country': row[3].split(',')[1].strip() if ',' in row[3] else '',
|
|
||||||
'url': row[1],
|
|
||||||
'type': row[0].split('-')[0],
|
|
||||||
'lotCount': row[4],
|
|
||||||
'closingTime': row[5]
|
|
||||||
})
|
|
||||||
|
|
||||||
return jsonify(auctions)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Recommended Integration Steps
|
|
||||||
|
|
||||||
### Phase 1: Immediate (Adapter Pattern)
|
|
||||||
1. ✅ Keep separate schemas
|
|
||||||
2. ✅ Create `ScraperDataAdapter` in Monitor
|
|
||||||
3. ✅ Add import methods to `DatabaseService`
|
|
||||||
4. ✅ Monitor reads from scraper's tables using adapter
|
|
||||||
|
|
||||||
### Phase 2: Short-term (Unified Schema)
|
|
||||||
1. 📋 Design unified schema (see Option 2)
|
|
||||||
2. 📋 Update scraper to use unified schema
|
|
||||||
3. 📋 Update monitor to use unified schema
|
|
||||||
4. 📋 Migrate existing data
|
|
||||||
|
|
||||||
### Phase 3: Long-term (API + Event-driven)
|
|
||||||
1. 📋 Add REST API to scraper
|
|
||||||
2. 📋 Add webhook/event notification when new data arrives
|
|
||||||
3. 📋 Monitor subscribes to events
|
|
||||||
4. 📋 Process images asynchronously
|
|
||||||
|
|
||||||
## Current Integration Flow
|
|
||||||
|
|
||||||
### Scraper Process (Python)
|
|
||||||
```bash
|
|
||||||
# 1. Run scraper to populate database
|
|
||||||
cd /path/to/scraper
|
|
||||||
python scraper.py
|
|
||||||
|
|
||||||
# Output:
|
|
||||||
# ✅ Scraped 42 auctions
|
|
||||||
# ✅ Scraped 1,234 lots
|
|
||||||
# ✅ Saved 3,456 image URLs
|
|
||||||
# ✅ Data written to: /mnt/okcomputer/output/cache.db
|
|
||||||
```
|
|
||||||
|
|
||||||
### Monitor Process (Java)
|
|
||||||
```bash
|
|
||||||
# 2. Run monitor to process the data
|
|
||||||
cd /path/to/monitor
|
|
||||||
export DATABASE_FILE=/mnt/okcomputer/output/cache.db
|
|
||||||
java -jar troostwijk-monitor.jar
|
|
||||||
|
|
||||||
# Output:
|
|
||||||
# 📊 Current Database State:
|
|
||||||
# Total lots in database: 1,234
|
|
||||||
# Total images processed: 0
|
|
||||||
#
|
|
||||||
# [1/2] Processing images...
|
|
||||||
# Downloading and analyzing 3,456 images...
|
|
||||||
#
|
|
||||||
# [2/2] Starting bid monitoring...
|
|
||||||
# ✓ Monitoring 1,234 active lots
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Shared Database Path
|
|
||||||
Both processes must point to the same database file:
|
|
||||||
|
|
||||||
**Scraper** (`config.py`):
|
|
||||||
```python
|
|
||||||
CACHE_DB = '/mnt/okcomputer/output/cache.db'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Monitor** (`Main.java`):
|
|
||||||
```java
|
|
||||||
String databaseFile = System.getenv().getOrDefault(
|
|
||||||
"DATABASE_FILE",
|
|
||||||
"/mnt/okcomputer/output/cache.db"
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Recommended Directory Structure
|
|
||||||
```
|
|
||||||
/mnt/okcomputer/
|
|
||||||
├── scraper/ # Python scraper code
|
|
||||||
│ ├── scraper.py
|
|
||||||
│ └── requirements.txt
|
|
||||||
├── monitor/ # Java monitor code
|
|
||||||
│ ├── troostwijk-monitor.jar
|
|
||||||
│ └── models/ # YOLO models
|
|
||||||
│ ├── yolov4.cfg
|
|
||||||
│ ├── yolov4.weights
|
|
||||||
│ └── coco.names
|
|
||||||
└── output/ # Shared data directory
|
|
||||||
├── cache.db # Shared SQLite database
|
|
||||||
└── images/ # Downloaded images
|
|
||||||
├── A1-28505-5/
|
|
||||||
│ ├── 001.jpg
|
|
||||||
│ └── 002.jpg
|
|
||||||
└── ...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring & Coordination
|
|
||||||
|
|
||||||
### Option A: Sequential Execution
|
|
||||||
```bash
|
|
||||||
#!/bin/bash
|
|
||||||
# run-pipeline.sh
|
|
||||||
|
|
||||||
echo "Step 1: Scraping..."
|
|
||||||
python scraper/scraper.py
|
|
||||||
|
|
||||||
echo "Step 2: Processing images..."
|
|
||||||
java -jar monitor/troostwijk-monitor.jar --process-images-only
|
|
||||||
|
|
||||||
echo "Step 3: Starting monitor..."
|
|
||||||
java -jar monitor/troostwijk-monitor.jar --monitor-only
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option B: Separate Services (Docker Compose)
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
services:
|
|
||||||
scraper:
|
|
||||||
build: ./scraper
|
|
||||||
volumes:
|
|
||||||
- ./output:/data
|
|
||||||
environment:
|
|
||||||
- CACHE_DB=/data/cache.db
|
|
||||||
command: python scraper.py
|
|
||||||
|
|
||||||
monitor:
|
|
||||||
build: ./monitor
|
|
||||||
volumes:
|
|
||||||
- ./output:/data
|
|
||||||
environment:
|
|
||||||
- DATABASE_FILE=/data/cache.db
|
|
||||||
- NOTIFICATION_CONFIG=desktop
|
|
||||||
depends_on:
|
|
||||||
- scraper
|
|
||||||
command: java -jar troostwijk-monitor.jar
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option C: Cron-based Scheduling
|
|
||||||
```cron
|
|
||||||
# Scrape every 6 hours
|
|
||||||
0 */6 * * * cd /mnt/okcomputer/scraper && python scraper.py
|
|
||||||
|
|
||||||
# Process images every hour (if new lots found)
|
|
||||||
0 * * * * cd /mnt/okcomputer/monitor && java -jar monitor.jar --process-new
|
|
||||||
|
|
||||||
# Monitor runs continuously
|
|
||||||
@reboot cd /mnt/okcomputer/monitor && java -jar monitor.jar --monitor-only
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Issue: Type Mismatch Errors
|
|
||||||
**Symptom**: Monitor crashes with "INTEGER expected, got TEXT"
|
|
||||||
|
|
||||||
**Solution**: Use adapter pattern (Option 1) or unified schema (Option 2)
|
|
||||||
|
|
||||||
### Issue: Monitor sees no data
|
|
||||||
**Symptom**: "Total lots in database: 0"
|
|
||||||
|
|
||||||
**Check**:
|
|
||||||
1. Is `DATABASE_FILE` env var set correctly?
|
|
||||||
2. Did scraper actually write data?
|
|
||||||
3. Are both processes using the same database file?
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Verify database has data
|
|
||||||
sqlite3 /mnt/okcomputer/output/cache.db "SELECT COUNT(*) FROM lots"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Issue: Images not downloading
|
|
||||||
**Symptom**: "Total images processed: 0" but scraper found images
|
|
||||||
|
|
||||||
**Check**:
|
|
||||||
1. Scraper writes image URLs to `images` table
|
|
||||||
2. Monitor reads from `images` table with `downloaded=0`
|
|
||||||
3. Field name mapping: `local_path` vs `local_path`
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. **Immediate**: Implement `ScraperDataAdapter` for compatibility
|
|
||||||
2. **This Week**: Test end-to-end integration with sample data
|
|
||||||
3. **Next Sprint**: Migrate to unified schema
|
|
||||||
4. **Future**: Add event-driven architecture with webhooks
|
|
||||||
@@ -1,422 +0,0 @@
|
|||||||
# Intelligence Features Implementation Summary
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
This document summarizes the implementation of advanced intelligence features based on 15+ new GraphQL API fields discovered from the Troostwijk auction system.
|
|
||||||
|
|
||||||
## New GraphQL Fields Integrated
|
|
||||||
|
|
||||||
### HIGH PRIORITY FIELDS (Implemented)
|
|
||||||
1. **`followersCount`** (Integer) - Watch count showing bidder interest
|
|
||||||
- Direct indicator of competition
|
|
||||||
- Used for sleeper lot detection
|
|
||||||
- Popularity level classification
|
|
||||||
|
|
||||||
2. **`estimatedFullPrice`** (Object: min/max cents)
|
|
||||||
- Auction house's estimated value range
|
|
||||||
- Used for bargain detection
|
|
||||||
- Price vs estimate analytics
|
|
||||||
|
|
||||||
3. **`nextBidStepInCents`** (Long)
|
|
||||||
- Exact bid increment from API
|
|
||||||
- Precise next bid calculations
|
|
||||||
- Better UX for bidding recommendations
|
|
||||||
|
|
||||||
4. **`condition`** (String)
|
|
||||||
- Direct condition field from API
|
|
||||||
- Better than extracting from attributes
|
|
||||||
- Used in condition scoring
|
|
||||||
|
|
||||||
5. **`categoryInformation`** (Object)
|
|
||||||
- Structured category with path
|
|
||||||
- Better categorization and filtering
|
|
||||||
- Category-based analytics
|
|
||||||
|
|
||||||
6. **`location`** (Object: city, countryCode, etc.)
|
|
||||||
- Structured location data
|
|
||||||
- Proximity filtering capability
|
|
||||||
- Logistics cost calculation
|
|
||||||
|
|
||||||
### MEDIUM PRIORITY FIELDS (Implemented)
|
|
||||||
7. **`biddingStatus`** (Enum) - Detailed bidding status
|
|
||||||
8. **`appearance`** (String) - Visual condition notes
|
|
||||||
9. **`packaging`** (String) - Packaging details
|
|
||||||
10. **`quantity`** (Long) - Lot quantity for bulk items
|
|
||||||
11. **`vat`** (BigDecimal) - VAT percentage
|
|
||||||
12. **`buyerPremiumPercentage`** (BigDecimal) - Buyer premium
|
|
||||||
13. **`remarks`** (String) - Viewing/pickup notes
|
|
||||||
|
|
||||||
## Code Changes
|
|
||||||
|
|
||||||
### 1. Backend - Lot.java (Domain Model)
|
|
||||||
**File**: `src/main/java/auctiora/Lot.java`
|
|
||||||
|
|
||||||
**Changes**:
|
|
||||||
- Added 24 new fields to the Lot record
|
|
||||||
- Implemented 9 intelligence calculation methods:
|
|
||||||
- `calculateTotalCost()` - Bid + VAT + Premium
|
|
||||||
- `calculateNextBid()` - Using API increment
|
|
||||||
- `isBelowEstimate()` - Bargain detection
|
|
||||||
- `isAboveEstimate()` - Overvalued detection
|
|
||||||
- `getInterestToBidRatio()` - Conversion rate
|
|
||||||
- `getPopularityLevel()` - HIGH/MEDIUM/LOW/MINIMAL
|
|
||||||
- `isSleeperLot()` - High interest, low bid
|
|
||||||
- `getEstimatedMidpoint()` - Average of estimate range
|
|
||||||
- `getPriceVsEstimateRatio()` - Price comparison metric
|
|
||||||
|
|
||||||
**Example**:
|
|
||||||
```java
|
|
||||||
public boolean isSleeperLot() {
|
|
||||||
return followersCount != null && followersCount > 10 && currentBid < 100;
|
|
||||||
}
|
|
||||||
|
|
||||||
public double calculateTotalCost() {
|
|
||||||
double base = currentBid > 0 ? currentBid : 0;
|
|
||||||
if (vat != null && vat > 0) {
|
|
||||||
base += (base * vat / 100.0);
|
|
||||||
}
|
|
||||||
if (buyerPremiumPercentage != null && buyerPremiumPercentage > 0) {
|
|
||||||
base += (base * buyerPremiumPercentage / 100.0);
|
|
||||||
}
|
|
||||||
return base;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Backend - AuctionMonitorResource.java (REST API)
|
|
||||||
**File**: `src/main/java/auctiora/AuctionMonitorResource.java`
|
|
||||||
|
|
||||||
**New Endpoints Added**:
|
|
||||||
1. `GET /api/monitor/intelligence/sleepers` - Sleeper lots (high interest, low bids)
|
|
||||||
2. `GET /api/monitor/intelligence/bargains` - Bargain lots (below estimate)
|
|
||||||
3. `GET /api/monitor/intelligence/popular?level={HIGH|MEDIUM|LOW}` - Popular lots
|
|
||||||
4. `GET /api/monitor/intelligence/price-analysis` - Price vs estimate statistics
|
|
||||||
5. `GET /api/monitor/lots/{lotId}/intelligence` - Detailed lot intelligence
|
|
||||||
6. `GET /api/monitor/charts/watch-distribution` - Follower count distribution
|
|
||||||
|
|
||||||
**Enhanced Features**:
|
|
||||||
- Updated insights endpoint to include sleeper, bargain, and popular insights
|
|
||||||
- Added intelligent filtering and sorting for intelligence data
|
|
||||||
- Integrated new fields into existing statistics
|
|
||||||
|
|
||||||
**Example Endpoint**:
|
|
||||||
```java
|
|
||||||
@GET
|
|
||||||
@Path("/intelligence/sleepers")
|
|
||||||
public Response getSleeperLots(@QueryParam("minFollowers") @DefaultValue("10") int minFollowers) {
|
|
||||||
var allLots = db.getAllLots();
|
|
||||||
var sleepers = allLots.stream()
|
|
||||||
.filter(Lot::isSleeperLot)
|
|
||||||
.toList();
|
|
||||||
|
|
||||||
return Response.ok(Map.of(
|
|
||||||
"count", sleepers.size(),
|
|
||||||
"lots", sleepers
|
|
||||||
)).build();
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Frontend - index.html (Intelligence Dashboard)
|
|
||||||
**File**: `src/main/resources/META-INF/resources/index.html`
|
|
||||||
|
|
||||||
**New UI Components**:
|
|
||||||
|
|
||||||
#### Intelligence Dashboard Widgets (3 new cards)
|
|
||||||
1. **Sleeper Lots Widget**
|
|
||||||
- Purple gradient design
|
|
||||||
- Shows count of high-interest, low-bid lots
|
|
||||||
- Click to filter table
|
|
||||||
|
|
||||||
2. **Bargain Lots Widget**
|
|
||||||
- Green gradient design
|
|
||||||
- Shows count of below-estimate lots
|
|
||||||
- Click to filter table
|
|
||||||
|
|
||||||
3. **Popular/Hot Lots Widget**
|
|
||||||
- Orange gradient design
|
|
||||||
- Shows count of high-follower lots
|
|
||||||
- Click to filter table
|
|
||||||
|
|
||||||
#### Enhanced Closing Soon Table
|
|
||||||
**New Columns Added**:
|
|
||||||
1. **Watchers** - Follower count with color-coded badges
|
|
||||||
- Red (50+ followers): High competition
|
|
||||||
- Orange (21-50): Medium competition
|
|
||||||
- Blue (6-20): Some interest
|
|
||||||
- Gray (0-5): Minimal interest
|
|
||||||
|
|
||||||
2. **Est. Range** - Auction house estimate (`€min-€max`)
|
|
||||||
- Shows "DEAL" badge if below estimate
|
|
||||||
|
|
||||||
3. **Total Cost** - True cost including VAT and premium
|
|
||||||
- Hover tooltip shows breakdown
|
|
||||||
- Purple color to stand out
|
|
||||||
|
|
||||||
**JavaScript Functions Added**:
|
|
||||||
- `fetchIntelligenceData()` - Fetches all intelligence metrics
|
|
||||||
- `showSleeperLots()` - Filters table to sleepers
|
|
||||||
- `showBargainLots()` - Filters table to bargains
|
|
||||||
- `showPopularLots()` - Filters table to popular
|
|
||||||
- Enhanced table rendering with smart badges
|
|
||||||
|
|
||||||
**Example Code**:
|
|
||||||
```javascript
|
|
||||||
// Calculate total cost (including VAT and premium)
|
|
||||||
const currentBid = lot.currentBid || 0;
|
|
||||||
const vat = lot.vat || 0;
|
|
||||||
const premium = lot.buyerPremiumPercentage || 0;
|
|
||||||
const totalCost = currentBid * (1 + (vat/100) + (premium/100));
|
|
||||||
|
|
||||||
// Bargain indicator
|
|
||||||
const isBargain = estMin && currentBid < parseFloat(estMin);
|
|
||||||
const bargainBadge = isBargain ?
|
|
||||||
'<span class="ml-1 text-xs bg-green-500 text-white px-1 rounded">DEAL</span>' : '';
|
|
||||||
```
|
|
||||||
|
|
||||||
## Intelligence Features
|
|
||||||
|
|
||||||
### 1. Sleeper Lot Detection
|
|
||||||
**Algorithm**: `followersCount > 10 AND currentBid < 100`
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- Identifies lots with high interest but low current bids
|
|
||||||
- Opportunity to bid strategically before price escalates
|
|
||||||
- Early indicator of undervalued items
|
|
||||||
|
|
||||||
**Dashboard Display**:
|
|
||||||
- Count shown in purple widget
|
|
||||||
- Click to filter table
|
|
||||||
- Purple "eye" icon
|
|
||||||
|
|
||||||
### 2. Bargain Detection
|
|
||||||
**Algorithm**: `currentBid < estimatedMin`
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- Identifies lots priced below auction house estimate
|
|
||||||
- Clear signal of potential good deals
|
|
||||||
- Quantifiable value assessment
|
|
||||||
|
|
||||||
**Dashboard Display**:
|
|
||||||
- Count shown in green widget
|
|
||||||
- "DEAL" badge in table
|
|
||||||
- Click to filter table
|
|
||||||
|
|
||||||
### 3. Popularity Analysis
|
|
||||||
**Algorithm**: Tiered classification by follower count
|
|
||||||
- HIGH: > 50 followers
|
|
||||||
- MEDIUM: 21-50 followers
|
|
||||||
- LOW: 6-20 followers
|
|
||||||
- MINIMAL: 0-5 followers
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- Predict competition level
|
|
||||||
- Identify trending items
|
|
||||||
- Adjust bidding strategy accordingly
|
|
||||||
|
|
||||||
**Dashboard Display**:
|
|
||||||
- Count shown in orange widget
|
|
||||||
- Color-coded badges in table
|
|
||||||
- Click to filter by level
|
|
||||||
|
|
||||||
### 4. True Cost Calculator
|
|
||||||
**Algorithm**: `currentBid × (1 + VAT/100) × (1 + premium/100)`
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- Shows actual out-of-pocket cost
|
|
||||||
- Prevents budget surprises
|
|
||||||
- Enables accurate comparison across lots
|
|
||||||
|
|
||||||
**Dashboard Display**:
|
|
||||||
- Purple "Total Cost" column
|
|
||||||
- Hover tooltip shows breakdown
|
|
||||||
- Updated in real-time
|
|
||||||
|
|
||||||
### 5. Exact Bid Increment
|
|
||||||
**Algorithm**: Uses `nextBidStepInCents` from API, falls back to calculated increment
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- No guesswork on next bid amount
|
|
||||||
- API-provided accuracy
|
|
||||||
- Better bidding UX
|
|
||||||
|
|
||||||
**Implementation**:
|
|
||||||
```java
|
|
||||||
public double calculateNextBid() {
|
|
||||||
if (nextBidStepInCents != null && nextBidStepInCents > 0) {
|
|
||||||
return currentBid + (nextBidStepInCents / 100.0);
|
|
||||||
} else if (bidIncrement != null && bidIncrement > 0) {
|
|
||||||
return currentBid + bidIncrement;
|
|
||||||
}
|
|
||||||
return currentBid * 1.05; // Fallback: 5% increment
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6. Price vs Estimate Analytics
|
|
||||||
**Metrics**:
|
|
||||||
- Total lots with estimates
|
|
||||||
- Count below estimate
|
|
||||||
- Count above estimate
|
|
||||||
- Average price vs estimate percentage
|
|
||||||
|
|
||||||
**Value Proposition**:
|
|
||||||
- Market efficiency analysis
|
|
||||||
- Auction house accuracy tracking
|
|
||||||
- Investment opportunity identification
|
|
||||||
|
|
||||||
**API Endpoint**: `/api/monitor/intelligence/price-analysis`
|
|
||||||
|
|
||||||
## Visual Design
|
|
||||||
|
|
||||||
### Color Scheme
|
|
||||||
- **Purple**: Sleeper lots, total cost (opportunity/value)
|
|
||||||
- **Green**: Bargains, deals (positive value)
|
|
||||||
- **Orange/Red**: Popular/hot lots (competition warning)
|
|
||||||
- **Blue**: Moderate interest (informational)
|
|
||||||
- **Gray**: Minimal interest (neutral)
|
|
||||||
|
|
||||||
### Badge System
|
|
||||||
1. **Watchers Badge**: Color-coded by competition level
|
|
||||||
2. **DEAL Badge**: Green indicator for below-estimate
|
|
||||||
3. **Time Left Badge**: Red/yellow/green by urgency
|
|
||||||
4. **Popularity Badge**: Fire icon for hot lots
|
|
||||||
|
|
||||||
### Interactive Elements
|
|
||||||
- Click widgets to filter table
|
|
||||||
- Hover for detailed tooltips
|
|
||||||
- Smooth scroll to table on filter
|
|
||||||
- Toast notifications for user feedback
|
|
||||||
|
|
||||||
## Performance Considerations
|
|
||||||
|
|
||||||
### API Optimization
|
|
||||||
- All intelligence data fetched in parallel
|
|
||||||
- Cached in dashboard state
|
|
||||||
- Minimal recalculation on render
|
|
||||||
- Efficient stream operations in backend
|
|
||||||
|
|
||||||
### Frontend Optimization
|
|
||||||
- Batch DOM updates
|
|
||||||
- Lazy rendering for large tables
|
|
||||||
- Debounced filter operations
|
|
||||||
- CSS transitions for smooth UX
|
|
||||||
|
|
||||||
## Testing Recommendations
|
|
||||||
|
|
||||||
### Backend Tests
|
|
||||||
1. Test `Lot` intelligence methods with various inputs
|
|
||||||
2. Test API endpoints with mock data
|
|
||||||
3. Test edge cases (null values, zero bids, etc.)
|
|
||||||
4. Performance test with 10k+ lots
|
|
||||||
|
|
||||||
### Frontend Tests
|
|
||||||
1. Test widget click handlers
|
|
||||||
2. Test table rendering with new columns
|
|
||||||
3. Test filter functionality
|
|
||||||
4. Test responsive design on mobile
|
|
||||||
|
|
||||||
### Integration Tests
|
|
||||||
1. End-to-end flow: Scraper → DB → API → Dashboard
|
|
||||||
2. Real-time data refresh
|
|
||||||
3. Concurrent user access
|
|
||||||
4. Load testing
|
|
||||||
|
|
||||||
## Future Enhancements
|
|
||||||
|
|
||||||
### Phase 2 (Bid History)
|
|
||||||
- Implement `bid_history` table scraping
|
|
||||||
- Track bid changes over time
|
|
||||||
- Calculate bid velocity accurately
|
|
||||||
- Identify bid patterns
|
|
||||||
|
|
||||||
### Phase 3 (ML Predictions)
|
|
||||||
- Predict final hammer price
|
|
||||||
- Recommend optimal bid timing
|
|
||||||
- Classify lot categories automatically
|
|
||||||
- Anomaly detection
|
|
||||||
|
|
||||||
### Phase 4 (Mobile)
|
|
||||||
- React Native mobile app
|
|
||||||
- Push notifications
|
|
||||||
- Offline mode
|
|
||||||
- Quick bid functionality
|
|
||||||
|
|
||||||
## Migration Guide
|
|
||||||
|
|
||||||
### Database Migration (Required)
|
|
||||||
The new fields need to be added to the database schema:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
-- Add to lots table
|
|
||||||
ALTER TABLE lots ADD COLUMN followers_count INTEGER DEFAULT 0;
|
|
||||||
ALTER TABLE lots ADD COLUMN estimated_min DECIMAL(12, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN estimated_max DECIMAL(12, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN next_bid_step_in_cents BIGINT;
|
|
||||||
ALTER TABLE lots ADD COLUMN condition TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN category_path TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN city_location TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN country_code TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN bidding_status TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN appearance TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN packaging TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN quantity BIGINT;
|
|
||||||
ALTER TABLE lots ADD COLUMN vat DECIMAL(5, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN buyer_premium_percentage DECIMAL(5, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN remarks TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN starting_bid DECIMAL(12, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN reserve_price DECIMAL(12, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN reserve_met BOOLEAN DEFAULT FALSE;
|
|
||||||
ALTER TABLE lots ADD COLUMN bid_increment DECIMAL(12, 2);
|
|
||||||
ALTER TABLE lots ADD COLUMN view_count INTEGER DEFAULT 0;
|
|
||||||
ALTER TABLE lots ADD COLUMN first_bid_time TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN last_bid_time TEXT;
|
|
||||||
ALTER TABLE lots ADD COLUMN bid_velocity DECIMAL(5, 2);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Scraper Update (Required)
|
|
||||||
The external scraper (Python/Playwright) needs to extract the new fields from GraphQL:
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Extract from __NEXT_DATA__ JSON
|
|
||||||
followers_count = lot_data.get('followersCount')
|
|
||||||
estimated_min = lot_data.get('estimatedFullPrice', {}).get('min', {}).get('cents')
|
|
||||||
estimated_max = lot_data.get('estimatedFullPrice', {}).get('max', {}).get('cents')
|
|
||||||
next_bid_step = lot_data.get('nextBidStepInCents')
|
|
||||||
condition = lot_data.get('condition')
|
|
||||||
# ... etc
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deployment Steps
|
|
||||||
1. Stop the monitor service
|
|
||||||
2. Run database migrations
|
|
||||||
3. Update scraper to extract new fields
|
|
||||||
4. Deploy updated monitor JAR
|
|
||||||
5. Restart services
|
|
||||||
6. Verify data populating in dashboard
|
|
||||||
|
|
||||||
## Performance Metrics
|
|
||||||
|
|
||||||
### Expected Performance
|
|
||||||
- **Intelligence Data Fetch**: < 100ms for 10k lots
|
|
||||||
- **Table Rendering**: < 200ms with all new columns
|
|
||||||
- **Widget Update**: < 50ms
|
|
||||||
- **API Response Time**: < 500ms
|
|
||||||
|
|
||||||
### Resource Usage
|
|
||||||
- **Memory**: +50MB for intelligence calculations
|
|
||||||
- **Database**: +2KB per lot (new columns)
|
|
||||||
- **Network**: +10KB per dashboard refresh
|
|
||||||
|
|
||||||
## Documentation
|
|
||||||
- **Integration Flowchart**: `docs/INTEGRATION_FLOWCHART.md`
|
|
||||||
- **API Documentation**: Auto-generated from JAX-RS annotations
|
|
||||||
- **Database Schema**: `wiki/DATABASE_ARCHITECTURE.md`
|
|
||||||
- **GraphQL Fields**: `wiki/EXPERT_ANALITICS.sql`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Implementation Date**: December 2025
|
|
||||||
**Version**: 2.1
|
|
||||||
**Status**: ✅ Complete - Ready for Testing
|
|
||||||
**Next Steps**:
|
|
||||||
1. Deploy to staging environment
|
|
||||||
2. Run integration tests
|
|
||||||
3. Update scraper to extract new fields
|
|
||||||
4. Deploy to production
|
|
||||||
@@ -1,540 +0,0 @@
|
|||||||
# Quarkus Implementation Complete ✅
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
The Troostwijk Auction Monitor has been fully integrated with **Quarkus Framework** for production-ready deployment with enterprise features.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 What Was Added
|
|
||||||
|
|
||||||
### 1. **Quarkus Dependencies** (pom.xml)
|
|
||||||
|
|
||||||
```xml
|
|
||||||
<!-- Core Quarkus -->
|
|
||||||
<dependency>
|
|
||||||
<groupId>io.quarkus</groupId>
|
|
||||||
<artifactId>quarkus-arc</artifactId> <!-- CDI/DI -->
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>io.quarkus</groupId>
|
|
||||||
<artifactId>quarkus-rest-jackson</artifactId> <!-- REST API -->
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>io.quarkus</groupId>
|
|
||||||
<artifactId>quarkus-scheduler</artifactId> <!-- Cron Scheduling -->
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>io.quarkus</groupId>
|
|
||||||
<artifactId>quarkus-smallrye-health</artifactId> <!-- Health Checks -->
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>io.quarkus</groupId>
|
|
||||||
<artifactId>quarkus-config-yaml</artifactId> <!-- YAML Config -->
|
|
||||||
</dependency>
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. **Configuration** (application.properties)
|
|
||||||
|
|
||||||
```properties
|
|
||||||
# Application
|
|
||||||
quarkus.application.name=troostwijk-scraper
|
|
||||||
quarkus.http.port=8081
|
|
||||||
|
|
||||||
# Auction Monitor Configuration
|
|
||||||
auction.database.path=C:\\mnt\\okcomputer\\output\\cache.db
|
|
||||||
auction.images.path=C:\\mnt\\okcomputer\\output\\images
|
|
||||||
auction.notification.config=desktop
|
|
||||||
|
|
||||||
# YOLO Models
|
|
||||||
auction.yolo.config=models/yolov4.cfg
|
|
||||||
auction.yolo.weights=models/yolov4.weights
|
|
||||||
auction.yolo.classes=models/coco.names
|
|
||||||
|
|
||||||
# Workflow Schedules (Cron Expressions)
|
|
||||||
auction.workflow.scraper-import.cron=0 */30 * * * ? # Every 30 min
|
|
||||||
auction.workflow.image-processing.cron=0 0 * * * ? # Every 1 hour
|
|
||||||
auction.workflow.bid-monitoring.cron=0 */15 * * * ? # Every 15 min
|
|
||||||
auction.workflow.closing-alerts.cron=0 */5 * * * ? # Every 5 min
|
|
||||||
|
|
||||||
# Scheduler
|
|
||||||
quarkus.scheduler.enabled=true
|
|
||||||
|
|
||||||
# Health Checks
|
|
||||||
quarkus.smallrye-health.root-path=/health
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. **Quarkus Scheduler** (QuarkusWorkflowScheduler.java)
|
|
||||||
|
|
||||||
Replaced manual `ScheduledExecutorService` with Quarkus `@Scheduled`:
|
|
||||||
|
|
||||||
```java
|
|
||||||
@ApplicationScoped
|
|
||||||
public class QuarkusWorkflowScheduler {
|
|
||||||
|
|
||||||
@Inject DatabaseService db;
|
|
||||||
@Inject NotificationService notifier;
|
|
||||||
@Inject ObjectDetectionService detector;
|
|
||||||
@Inject ImageProcessingService imageProcessor;
|
|
||||||
|
|
||||||
// Workflow 1: Every 30 minutes
|
|
||||||
@Scheduled(cron = "{auction.workflow.scraper-import.cron}")
|
|
||||||
void importScraperData() { /* ... */ }
|
|
||||||
|
|
||||||
// Workflow 2: Every 1 hour
|
|
||||||
@Scheduled(cron = "{auction.workflow.image-processing.cron}")
|
|
||||||
void processImages() { /* ... */ }
|
|
||||||
|
|
||||||
// Workflow 3: Every 15 minutes
|
|
||||||
@Scheduled(cron = "{auction.workflow.bid-monitoring.cron}")
|
|
||||||
void monitorBids() { /* ... */ }
|
|
||||||
|
|
||||||
// Workflow 4: Every 5 minutes
|
|
||||||
@Scheduled(cron = "{auction.workflow.closing-alerts.cron}")
|
|
||||||
void checkClosingTimes() { /* ... */ }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. **CDI Producer** (AuctionMonitorProducer.java)
|
|
||||||
|
|
||||||
Centralized service creation with dependency injection:
|
|
||||||
|
|
||||||
```java
|
|
||||||
@ApplicationScoped
|
|
||||||
public class AuctionMonitorProducer {
|
|
||||||
|
|
||||||
@Produces @Singleton
|
|
||||||
public DatabaseService produceDatabaseService(
|
|
||||||
@ConfigProperty(name = "auction.database.path") String dbPath) {
|
|
||||||
DatabaseService db = new DatabaseService(dbPath);
|
|
||||||
db.ensureSchema();
|
|
||||||
return db;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Produces @Singleton
|
|
||||||
public NotificationService produceNotificationService(
|
|
||||||
@ConfigProperty(name = "auction.notification.config") String config) {
|
|
||||||
return new NotificationService(config, "");
|
|
||||||
}
|
|
||||||
|
|
||||||
@Produces @Singleton
|
|
||||||
public ObjectDetectionService produceObjectDetectionService(...) { }
|
|
||||||
|
|
||||||
@Produces @Singleton
|
|
||||||
public ImageProcessingService produceImageProcessingService(...) { }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. **REST API** (AuctionMonitorResource.java)
|
|
||||||
|
|
||||||
Full REST API for monitoring and control:
|
|
||||||
|
|
||||||
| Endpoint | Method | Description |
|
|
||||||
|----------|--------|-------------|
|
|
||||||
| `/api/monitor/status` | GET | Get current status |
|
|
||||||
| `/api/monitor/statistics` | GET | Get detailed statistics |
|
|
||||||
| `/api/monitor/trigger/scraper-import` | POST | Trigger scraper import |
|
|
||||||
| `/api/monitor/trigger/image-processing` | POST | Trigger image processing |
|
|
||||||
| `/api/monitor/trigger/bid-monitoring` | POST | Trigger bid monitoring |
|
|
||||||
| `/api/monitor/trigger/closing-alerts` | POST | Trigger closing alerts |
|
|
||||||
| `/api/monitor/auctions` | GET | List auctions |
|
|
||||||
| `/api/monitor/auctions?country=NL` | GET | Filter auctions by country |
|
|
||||||
| `/api/monitor/lots` | GET | List active lots |
|
|
||||||
| `/api/monitor/lots/closing-soon` | GET | Lots closing soon |
|
|
||||||
| `/api/monitor/lots/{id}/images` | GET | Get lot images |
|
|
||||||
| `/api/monitor/test-notification` | POST | Send test notification |
|
|
||||||
|
|
||||||
### 6. **Health Checks** (AuctionMonitorHealthCheck.java)
|
|
||||||
|
|
||||||
Kubernetes-ready health probes:
|
|
||||||
|
|
||||||
```java
|
|
||||||
@Liveness // /health/live
|
|
||||||
public class LivenessCheck implements HealthCheck {
|
|
||||||
public HealthCheckResponse call() {
|
|
||||||
return HealthCheckResponse.up("Auction Monitor is alive");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@Readiness // /health/ready
|
|
||||||
public class ReadinessCheck implements HealthCheck {
|
|
||||||
@Inject DatabaseService db;
|
|
||||||
|
|
||||||
public HealthCheckResponse call() {
|
|
||||||
var auctions = db.getAllAuctions();
|
|
||||||
return HealthCheckResponse.named("database")
|
|
||||||
.up()
|
|
||||||
.withData("auctions", auctions.size())
|
|
||||||
.build();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@Startup // /health/started
|
|
||||||
public class StartupCheck implements HealthCheck { /* ... */ }
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7. **Docker Support**
|
|
||||||
|
|
||||||
#### Dockerfile (Optimized for Quarkus fast-jar)
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
# Build stage
|
|
||||||
FROM maven:3.9-eclipse-temurin-25-alpine AS build
|
|
||||||
WORKDIR /app
|
|
||||||
COPY ../pom.xml ./
|
|
||||||
RUN mvn dependency:go-offline -B
|
|
||||||
COPY ../src ./src/
|
|
||||||
RUN mvn package -DskipTests -Dquarkus.package.jar.type=fast-jar
|
|
||||||
|
|
||||||
# Runtime stage
|
|
||||||
FROM eclipse-temurin:25-jre-alpine
|
|
||||||
WORKDIR /app
|
|
||||||
|
|
||||||
# Copy Quarkus fast-jar structure
|
|
||||||
COPY --from=build /app/target/quarkus-app/lib/ /app/lib/
|
|
||||||
COPY --from=build /app/target/quarkus-app/*.jar /app/
|
|
||||||
COPY --from=build /app/target/quarkus-app/app/ /app/app/
|
|
||||||
COPY --from=build /app/target/quarkus-app/quarkus/ /app/quarkus/
|
|
||||||
|
|
||||||
EXPOSE 8081
|
|
||||||
HEALTHCHECK CMD wget --spider http://localhost:8081/health/live
|
|
||||||
|
|
||||||
ENTRYPOINT ["java", "-jar", "/app/quarkus-run.jar"]
|
|
||||||
```
|
|
||||||
|
|
||||||
#### docker-compose.yml
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
services:
|
|
||||||
auction-monitor:
|
|
||||||
build: ../wiki
|
|
||||||
ports:
|
|
||||||
- "8081:8081"
|
|
||||||
volumes:
|
|
||||||
- ./data/cache.db:/mnt/okcomputer/output/cache.db
|
|
||||||
- ./data/images:/mnt/okcomputer/output/images
|
|
||||||
environment:
|
|
||||||
- AUCTION_DATABASE_PATH=/mnt/okcomputer/output/cache.db
|
|
||||||
- AUCTION_NOTIFICATION_CONFIG=desktop
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "http://localhost:8081/health/live" ]
|
|
||||||
interval: 30s
|
|
||||||
restart: unless-stopped
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8. **Kubernetes Deployment**
|
|
||||||
|
|
||||||
Full Kubernetes manifests:
|
|
||||||
- **Namespace** - Isolated environment
|
|
||||||
- **PersistentVolumeClaim** - Data storage
|
|
||||||
- **ConfigMap** - Configuration
|
|
||||||
- **Secret** - Sensitive data (SMTP credentials)
|
|
||||||
- **Deployment** - Application pods
|
|
||||||
- **Service** - Internal networking
|
|
||||||
- **Ingress** - External access
|
|
||||||
- **HorizontalPodAutoscaler** - Auto-scaling
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 How to Run
|
|
||||||
|
|
||||||
### Development Mode (with live reload)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mvn quarkus:dev
|
|
||||||
|
|
||||||
# Access:
|
|
||||||
# - App: http://localhost:8081
|
|
||||||
# - Dev UI: http://localhost:8081/q/dev/
|
|
||||||
# - API: http://localhost:8081/api/monitor/status
|
|
||||||
# - Health: http://localhost:8081/health
|
|
||||||
```
|
|
||||||
|
|
||||||
### Production Mode (JAR)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build
|
|
||||||
mvn clean package
|
|
||||||
|
|
||||||
# Run
|
|
||||||
java -jar target/quarkus-app/quarkus-run.jar
|
|
||||||
|
|
||||||
# Access: http://localhost:8081
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build
|
|
||||||
docker build -t auction-monitor .
|
|
||||||
|
|
||||||
# Run
|
|
||||||
docker run -p 8081:8081 auction-monitor
|
|
||||||
|
|
||||||
# Access: http://localhost:8081
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker Compose
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Start
|
|
||||||
docker-compose up -d
|
|
||||||
|
|
||||||
# View logs
|
|
||||||
docker-compose logs -f
|
|
||||||
|
|
||||||
# Access: http://localhost:8081
|
|
||||||
```
|
|
||||||
|
|
||||||
### Kubernetes
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Deploy
|
|
||||||
kubectl apply -f k8s/deployment.yaml
|
|
||||||
|
|
||||||
# Port forward
|
|
||||||
kubectl port-forward svc/auction-monitor 8081:8081 -n auction-monitor
|
|
||||||
|
|
||||||
# Access: http://localhost:8081
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📊 Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ QUARKUS APPLICATION │
|
|
||||||
├─────────────────────────────────────────────────────────────┤
|
|
||||||
│ │
|
|
||||||
│ ┌────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ QuarkusWorkflowScheduler (@ApplicationScoped) │ │
|
|
||||||
│ │ ┌──────────────────────────────────────────────┐ │ │
|
|
||||||
│ │ │ @Scheduled(cron = "0 */30 * * * ?") │ │ │
|
|
||||||
│ │ │ importScraperData() │ │ │
|
|
||||||
│ │ ├──────────────────────────────────────────────┤ │ │
|
|
||||||
│ │ │ @Scheduled(cron = "0 0 * * * ?") │ │ │
|
|
||||||
│ │ │ processImages() │ │ │
|
|
||||||
│ │ ├──────────────────────────────────────────────┤ │ │
|
|
||||||
│ │ │ @Scheduled(cron = "0 */15 * * * ?") │ │ │
|
|
||||||
│ │ │ monitorBids() │ │ │
|
|
||||||
│ │ ├──────────────────────────────────────────────┤ │ │
|
|
||||||
│ │ │ @Scheduled(cron = "0 */5 * * * ?") │ │ │
|
|
||||||
│ │ │ checkClosingTimes() │ │ │
|
|
||||||
│ │ └──────────────────────────────────────────────┘ │ │
|
|
||||||
│ └────────────────────────────────────────────────────┘ │
|
|
||||||
│ ▲ │
|
|
||||||
│ │ @Inject │
|
|
||||||
│ ┌───────────────────────┴────────────────────────────┐ │
|
|
||||||
│ │ AuctionMonitorProducer │ │
|
|
||||||
│ │ ┌──────────────────────────────────────────────┐ │ │
|
|
||||||
│ │ │ @Produces @Singleton DatabaseService │ │ │
|
|
||||||
│ │ │ @Produces @Singleton NotificationService │ │ │
|
|
||||||
│ │ │ @Produces @Singleton ObjectDetectionService │ │ │
|
|
||||||
│ │ │ @Produces @Singleton ImageProcessingService │ │ │
|
|
||||||
│ │ └──────────────────────────────────────────────┘ │ │
|
|
||||||
│ └────────────────────────────────────────────────────┘ │
|
|
||||||
│ │
|
|
||||||
│ ┌────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ AuctionMonitorResource (REST API) │ │
|
|
||||||
│ │ ┌──────────────────────────────────────────────┐ │ │
|
|
||||||
│ │ │ GET /api/monitor/status │ │ │
|
|
||||||
│ │ │ GET /api/monitor/statistics │ │ │
|
|
||||||
│ │ │ POST /api/monitor/trigger/* │ │ │
|
|
||||||
│ │ │ GET /api/monitor/auctions │ │ │
|
|
||||||
│ │ │ GET /api/monitor/lots │ │ │
|
|
||||||
│ │ └──────────────────────────────────────────────┘ │ │
|
|
||||||
│ └────────────────────────────────────────────────────┘ │
|
|
||||||
│ │
|
|
||||||
│ ┌────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ AuctionMonitorHealthCheck │ │
|
|
||||||
│ │ ┌──────────────────────────────────────────────┐ │ │
|
|
||||||
│ │ │ @Liveness - /health/live │ │ │
|
|
||||||
│ │ │ @Readiness - /health/ready │ │ │
|
|
||||||
│ │ │ @Startup - /health/started │ │ │
|
|
||||||
│ │ └──────────────────────────────────────────────┘ │ │
|
|
||||||
│ └────────────────────────────────────────────────────┘ │
|
|
||||||
│ │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔧 Key Features
|
|
||||||
|
|
||||||
### 1. **Dependency Injection (CDI)**
|
|
||||||
- Type-safe injection with `@Inject`
|
|
||||||
- Singleton services with `@Produces`
|
|
||||||
- Configuration injection with `@ConfigProperty`
|
|
||||||
|
|
||||||
### 2. **Scheduled Tasks**
|
|
||||||
- Cron-based scheduling with `@Scheduled`
|
|
||||||
- Configurable via properties
|
|
||||||
- No manual thread management
|
|
||||||
|
|
||||||
### 3. **REST API**
|
|
||||||
- JAX-RS endpoints
|
|
||||||
- JSON serialization
|
|
||||||
- Error handling
|
|
||||||
|
|
||||||
### 4. **Health Checks**
|
|
||||||
- Liveness probe (is app alive?)
|
|
||||||
- Readiness probe (is app ready?)
|
|
||||||
- Startup probe (has app started?)
|
|
||||||
|
|
||||||
### 5. **Configuration**
|
|
||||||
- External configuration
|
|
||||||
- Environment variable override
|
|
||||||
- Type-safe config injection
|
|
||||||
|
|
||||||
### 6. **Container Ready**
|
|
||||||
- Optimized Docker image
|
|
||||||
- Fast startup (~0.5s)
|
|
||||||
- Low memory (~50MB)
|
|
||||||
- Health checks included
|
|
||||||
|
|
||||||
### 7. **Cloud Native**
|
|
||||||
- Kubernetes manifests
|
|
||||||
- Auto-scaling support
|
|
||||||
- Ingress configuration
|
|
||||||
- Persistent storage
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📁 Files Created/Modified
|
|
||||||
|
|
||||||
### New Files
|
|
||||||
|
|
||||||
```
|
|
||||||
src/main/java/com/auction/
|
|
||||||
├── QuarkusWorkflowScheduler.java # Quarkus scheduler
|
|
||||||
├── AuctionMonitorProducer.java # CDI producer
|
|
||||||
├── AuctionMonitorResource.java # REST API
|
|
||||||
└── AuctionMonitorHealthCheck.java # Health checks
|
|
||||||
|
|
||||||
src/main/resources/
|
|
||||||
└── application.properties # Configuration
|
|
||||||
|
|
||||||
k8s/
|
|
||||||
├── deployment.yaml # Kubernetes manifests
|
|
||||||
└── README.md # K8s deployment guide
|
|
||||||
|
|
||||||
docker-compose.yml # Docker Compose config
|
|
||||||
Dockerfile # Updated for Quarkus
|
|
||||||
QUARKUS_GUIDE.md # Complete Quarkus guide
|
|
||||||
QUARKUS_IMPLEMENTATION.md # This file
|
|
||||||
```
|
|
||||||
|
|
||||||
### Modified Files
|
|
||||||
|
|
||||||
```
|
|
||||||
pom.xml # Added Quarkus dependencies
|
|
||||||
src/main/resources/application.properties # Added config
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 Benefits of Quarkus
|
|
||||||
|
|
||||||
| Feature | Before | After (Quarkus) |
|
|
||||||
|---------|--------|-----------------|
|
|
||||||
| **Startup Time** | ~3-5 seconds | ~0.5 seconds |
|
|
||||||
| **Memory** | ~200MB | ~50MB |
|
|
||||||
| **Scheduling** | Manual ExecutorService | @Scheduled annotations |
|
|
||||||
| **DI/CDI** | Manual instantiation | @Inject, @Produces |
|
|
||||||
| **REST API** | None | Full JAX-RS API |
|
|
||||||
| **Health Checks** | None | Built-in probes |
|
|
||||||
| **Config** | Hard-coded | External properties |
|
|
||||||
| **Dev Mode** | Manual restart | Live reload |
|
|
||||||
| **Container** | Basic Docker | Optimized fast-jar |
|
|
||||||
| **Cloud Native** | Not ready | K8s ready |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧪 Testing
|
|
||||||
|
|
||||||
### Unit Tests
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
### Integration Tests
|
|
||||||
```bash
|
|
||||||
# Start app
|
|
||||||
mvn quarkus:dev
|
|
||||||
|
|
||||||
# In another terminal
|
|
||||||
curl http://localhost:8081/api/monitor/status
|
|
||||||
curl http://localhost:8081/health
|
|
||||||
curl -X POST http://localhost:8081/api/monitor/trigger/scraper-import
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker Test
|
|
||||||
```bash
|
|
||||||
docker-compose up -d
|
|
||||||
docker-compose logs -f
|
|
||||||
curl http://localhost:8081/api/monitor/status
|
|
||||||
docker-compose down
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📚 Documentation
|
|
||||||
|
|
||||||
1. **QUARKUS_GUIDE.md** - Complete Quarkus usage guide
|
|
||||||
2. **QUARKUS_IMPLEMENTATION.md** - This file (implementation details)
|
|
||||||
3. **k8s/README.md** - Kubernetes deployment guide
|
|
||||||
4. **docker-compose.yml** - Docker Compose reference
|
|
||||||
5. **README.md** - Updated main README
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎉 Summary
|
|
||||||
|
|
||||||
✅ **Quarkus Framework** - Fully integrated
|
|
||||||
✅ **@Scheduled Workflows** - Cron-based scheduling
|
|
||||||
✅ **CDI/Dependency Injection** - Clean architecture
|
|
||||||
✅ **REST API** - Full control interface
|
|
||||||
✅ **Health Checks** - Kubernetes ready
|
|
||||||
✅ **Docker/Compose** - Production containers
|
|
||||||
✅ **Kubernetes** - Cloud deployment
|
|
||||||
✅ **Configuration** - Externalized settings
|
|
||||||
✅ **Documentation** - Complete guides
|
|
||||||
|
|
||||||
**The application is now production-ready with Quarkus! 🚀**
|
|
||||||
|
|
||||||
### Quick Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Development
|
|
||||||
mvn quarkus:dev
|
|
||||||
|
|
||||||
# Production
|
|
||||||
mvn clean package
|
|
||||||
java -jar target/quarkus-app/quarkus-run.jar
|
|
||||||
|
|
||||||
# Docker
|
|
||||||
docker-compose up -d
|
|
||||||
|
|
||||||
# Kubernetes
|
|
||||||
kubectl apply -f k8s/deployment.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### API Access
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Status
|
|
||||||
curl http://localhost:8081/api/monitor/status
|
|
||||||
|
|
||||||
# Statistics
|
|
||||||
curl http://localhost:8081/api/monitor/statistics
|
|
||||||
|
|
||||||
# Health
|
|
||||||
curl http://localhost:8081/health
|
|
||||||
|
|
||||||
# Trigger workflow
|
|
||||||
curl -X POST http://localhost:8081/api/monitor/trigger/scraper-import
|
|
||||||
```
|
|
||||||
|
|
||||||
**Enjoy your Quarkus-powered Auction Monitor! 🎊**
|
|
||||||
@@ -1,191 +0,0 @@
|
|||||||
# Quick Start Guide
|
|
||||||
|
|
||||||
Get the scraper running in minutes without downloading YOLO models!
|
|
||||||
|
|
||||||
## Minimal Setup (No Object Detection)
|
|
||||||
|
|
||||||
The scraper works perfectly fine **without** YOLO object detection. You can run it immediately and add object detection later if needed.
|
|
||||||
|
|
||||||
### Step 1: Run the Scraper
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Using Maven
|
|
||||||
mvn clean compile exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
||||||
```
|
|
||||||
|
|
||||||
Or in IntelliJ IDEA:
|
|
||||||
1. Open `TroostwijkScraper.java`
|
|
||||||
2. Right-click on the `main` method
|
|
||||||
3. Select "Run 'TroostwijkScraper.main()'"
|
|
||||||
|
|
||||||
### What You'll See
|
|
||||||
|
|
||||||
```
|
|
||||||
=== Troostwijk Auction Scraper ===
|
|
||||||
|
|
||||||
Initializing scraper...
|
|
||||||
⚠️ Object detection disabled: YOLO model files not found
|
|
||||||
Expected files:
|
|
||||||
- models/yolov4.cfg
|
|
||||||
- models/yolov4.weights
|
|
||||||
- models/coco.names
|
|
||||||
Scraper will continue without image analysis.
|
|
||||||
|
|
||||||
[1/3] Discovering Dutch auctions...
|
|
||||||
✓ Found 5 auctions: [12345, 12346, 12347, 12348, 12349]
|
|
||||||
|
|
||||||
[2/3] Fetching lot details...
|
|
||||||
Processing sale 12345...
|
|
||||||
|
|
||||||
[3/3] Starting monitoring service...
|
|
||||||
✓ Monitoring active. Press Ctrl+C to stop.
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Test Desktop Notifications
|
|
||||||
|
|
||||||
The scraper will automatically send desktop notifications when:
|
|
||||||
- A new bid is placed on a monitored lot
|
|
||||||
- An auction is closing within 5 minutes
|
|
||||||
|
|
||||||
**No setup required** - desktop notifications work out of the box!
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Optional: Add Email Notifications
|
|
||||||
|
|
||||||
If you want email notifications in addition to desktop notifications:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Set environment variable
|
|
||||||
export NOTIFICATION_CONFIG="smtp:your.email@gmail.com:app_password:your.email@gmail.com"
|
|
||||||
|
|
||||||
# Then run the scraper
|
|
||||||
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Get Gmail App Password:**
|
|
||||||
1. Enable 2FA in Google Account
|
|
||||||
2. Go to: Google Account → Security → 2-Step Verification → App passwords
|
|
||||||
3. Generate password for "Mail"
|
|
||||||
4. Use that password (not your regular Gmail password)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Optional: Add Object Detection Later
|
|
||||||
|
|
||||||
If you want AI-powered image analysis to detect objects in auction photos:
|
|
||||||
|
|
||||||
### 1. Create models directory
|
|
||||||
```bash
|
|
||||||
mkdir models
|
|
||||||
cd models
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Download YOLO files
|
|
||||||
```bash
|
|
||||||
# YOLOv4 config (small)
|
|
||||||
curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg
|
|
||||||
|
|
||||||
# YOLOv4 weights (245 MB - takes a few minutes)
|
|
||||||
curl -LO https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights
|
|
||||||
|
|
||||||
# COCO class names
|
|
||||||
curl -O https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Run again
|
|
||||||
```bash
|
|
||||||
mvn exec:java -Dexec.mainClass="com.auction.scraper.TroostwijkScraper"
|
|
||||||
```
|
|
||||||
|
|
||||||
Now you'll see:
|
|
||||||
```
|
|
||||||
✓ Object detection enabled with YOLO
|
|
||||||
```
|
|
||||||
|
|
||||||
The scraper will now analyze auction images and detect objects like:
|
|
||||||
- Vehicles (cars, trucks, forklifts)
|
|
||||||
- Equipment (machines, tools)
|
|
||||||
- Furniture
|
|
||||||
- Electronics
|
|
||||||
- And 80+ other object types
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Features Without Object Detection
|
|
||||||
|
|
||||||
Even without YOLO, the scraper provides:
|
|
||||||
|
|
||||||
✅ **Full auction scraping** - Discovers all Dutch auctions
|
|
||||||
✅ **Lot tracking** - Monitors bids and closing times
|
|
||||||
✅ **Desktop notifications** - Real-time alerts
|
|
||||||
✅ **SQLite database** - All data persisted locally
|
|
||||||
✅ **Image downloading** - Saves all lot images
|
|
||||||
✅ **Scheduled monitoring** - Automatic updates every hour
|
|
||||||
|
|
||||||
Object detection simply adds:
|
|
||||||
- AI-powered image analysis
|
|
||||||
- Automatic object labeling
|
|
||||||
- Searchable image database
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Database Location
|
|
||||||
|
|
||||||
The scraper creates `troostwijk.db` in your current directory with:
|
|
||||||
- All auction data
|
|
||||||
- Lot details (title, description, bids, etc.)
|
|
||||||
- Downloaded image paths
|
|
||||||
- Object labels (if detection enabled)
|
|
||||||
|
|
||||||
View the database with any SQLite browser:
|
|
||||||
```bash
|
|
||||||
sqlite3 troostwijk.db
|
|
||||||
.tables
|
|
||||||
SELECT * FROM lots LIMIT 5;
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Stopping the Scraper
|
|
||||||
|
|
||||||
Press **Ctrl+C** to stop the monitoring service.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. ✅ **Run the scraper** without YOLO to test it
|
|
||||||
2. ✅ **Verify desktop notifications** work
|
|
||||||
3. ⚙️ **Optional**: Add email notifications
|
|
||||||
4. ⚙️ **Optional**: Download YOLO models for object detection
|
|
||||||
5. 🔧 **Customize**: Edit monitoring frequency, closing alerts, etc.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Desktop notifications not appearing?
|
|
||||||
- **Windows**: Check if Java has notification permissions
|
|
||||||
- **Linux**: Ensure desktop environment is running (not headless)
|
|
||||||
- **macOS**: Check System Preferences → Notifications
|
|
||||||
|
|
||||||
### OpenCV warnings?
|
|
||||||
These are normal and can be ignored:
|
|
||||||
```
|
|
||||||
WARNING: A restricted method in java.lang.System has been called
|
|
||||||
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid warning
|
|
||||||
```
|
|
||||||
|
|
||||||
The scraper works fine despite these warnings.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Full Documentation
|
|
||||||
|
|
||||||
See [README.md](../README.md) for complete documentation including:
|
|
||||||
- Email setup details
|
|
||||||
- YOLO installation guide
|
|
||||||
- Configuration options
|
|
||||||
- Database schema
|
|
||||||
- API endpoints
|
|
||||||
@@ -1,399 +0,0 @@
|
|||||||
# Scraper Refactor Guide - Image Download Integration
|
|
||||||
|
|
||||||
## 🎯 Objective
|
|
||||||
|
|
||||||
Refactor the Troostwijk scraper to **download and store images locally**, eliminating the 57M+ duplicate image problem in the monitoring process.
|
|
||||||
|
|
||||||
## 📋 Current vs. New Architecture
|
|
||||||
|
|
||||||
### **Before** (Current Architecture)
|
|
||||||
```
|
|
||||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
||||||
│ Scraper │────────▶│ Database │◀────────│ Monitor │
|
|
||||||
│ │ │ │ │ │
|
|
||||||
│ Stores URLs │ │ images table │ │ Downloads + │
|
|
||||||
│ downloaded=0 │ │ │ │ Detection │
|
|
||||||
└──────────────┘ └──────────────┘ └──────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
57M+ duplicates!
|
|
||||||
```
|
|
||||||
|
|
||||||
### **After** (New Architecture)
|
|
||||||
```
|
|
||||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
||||||
│ Scraper │────────▶│ Database │◀────────│ Monitor │
|
|
||||||
│ │ │ │ │ │
|
|
||||||
│ Downloads + │ │ images table │ │ Detection │
|
|
||||||
│ Stores path │ │ local_path ✓ │ │ Only │
|
|
||||||
│ downloaded=1 │ │ │ │ │
|
|
||||||
└──────────────┘ └──────────────┘ └──────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
No duplicates!
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🗄️ Database Schema Changes
|
|
||||||
|
|
||||||
### Current Schema (ARCHITECTURE-TROOSTWIJK-SCRAPER.md:113-122)
|
|
||||||
```sql
|
|
||||||
CREATE TABLE images (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT,
|
|
||||||
url TEXT,
|
|
||||||
local_path TEXT, -- Currently NULL
|
|
||||||
downloaded INTEGER -- Currently 0
|
|
||||||
-- Missing: processed_at, labels (added by monitor)
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Required Schema (Already Compatible!)
|
|
||||||
```sql
|
|
||||||
CREATE TABLE images (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT,
|
|
||||||
url TEXT,
|
|
||||||
local_path TEXT, -- ✅ SET by scraper after download
|
|
||||||
downloaded INTEGER, -- ✅ SET to 1 by scraper after download
|
|
||||||
labels TEXT, -- ⚠️ SET by monitor (object detection)
|
|
||||||
processed_at INTEGER, -- ⚠️ SET by monitor (timestamp)
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Good News**: The scraper's schema already has `local_path` and `downloaded` columns! You just need to populate them.
|
|
||||||
|
|
||||||
## 🔧 Implementation Steps
|
|
||||||
|
|
||||||
### **Step 1: Enable Image Downloading in Configuration**
|
|
||||||
|
|
||||||
**File**: Your scraper's config file (e.g., `config.py` or environment variables)
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Current setting
|
|
||||||
DOWNLOAD_IMAGES = False # ❌ Change this!
|
|
||||||
|
|
||||||
# New setting
|
|
||||||
DOWNLOAD_IMAGES = True # ✅ Enable downloads
|
|
||||||
|
|
||||||
# Image storage path
|
|
||||||
IMAGES_DIR = "/mnt/okcomputer/output/images" # Or your preferred path
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 2: Update Image Download Logic**
|
|
||||||
|
|
||||||
Based on ARCHITECTURE-TROOSTWIJK-SCRAPER.md:211-228, you already have the structure. Here's what needs to change:
|
|
||||||
|
|
||||||
**Current Code** (Conceptual):
|
|
||||||
```python
|
|
||||||
# Phase 3: Scrape lot details
|
|
||||||
def scrape_lot(lot_url):
|
|
||||||
lot_data = parse_lot_page(lot_url)
|
|
||||||
|
|
||||||
# Save lot to database
|
|
||||||
db.insert_lot(lot_data)
|
|
||||||
|
|
||||||
# Save image URLs to database (NOT DOWNLOADED)
|
|
||||||
for img_url in lot_data['images']:
|
|
||||||
db.execute("""
|
|
||||||
INSERT INTO images (lot_id, url, downloaded)
|
|
||||||
VALUES (?, ?, 0)
|
|
||||||
""", (lot_data['lot_id'], img_url))
|
|
||||||
```
|
|
||||||
|
|
||||||
**New Code** (Required):
|
|
||||||
```python
|
|
||||||
import os
|
|
||||||
import requests
|
|
||||||
from pathlib import Path
|
|
||||||
import time
|
|
||||||
|
|
||||||
def scrape_lot(lot_url):
|
|
||||||
lot_data = parse_lot_page(lot_url)
|
|
||||||
|
|
||||||
# Save lot to database
|
|
||||||
db.insert_lot(lot_data)
|
|
||||||
|
|
||||||
# Download and save images
|
|
||||||
for idx, img_url in enumerate(lot_data['images'], start=1):
|
|
||||||
try:
|
|
||||||
# Download image
|
|
||||||
local_path = download_image(img_url, lot_data['lot_id'], idx)
|
|
||||||
|
|
||||||
# Insert with local_path and downloaded=1
|
|
||||||
db.execute("""
|
|
||||||
INSERT INTO images (lot_id, url, local_path, downloaded)
|
|
||||||
VALUES (?, ?, ?, 1)
|
|
||||||
ON CONFLICT(lot_id, url) DO UPDATE SET
|
|
||||||
local_path = excluded.local_path,
|
|
||||||
downloaded = 1
|
|
||||||
""", (lot_data['lot_id'], img_url, local_path))
|
|
||||||
|
|
||||||
# Rate limiting (0.5s between downloads)
|
|
||||||
time.sleep(0.5)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Failed to download {img_url}: {e}")
|
|
||||||
# Still insert record but mark as not downloaded
|
|
||||||
db.execute("""
|
|
||||||
INSERT INTO images (lot_id, url, downloaded)
|
|
||||||
VALUES (?, ?, 0)
|
|
||||||
""", (lot_data['lot_id'], img_url))
|
|
||||||
|
|
||||||
def download_image(image_url, lot_id, index):
|
|
||||||
"""
|
|
||||||
Downloads an image and saves it to organized directory structure.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
image_url: Remote URL of the image
|
|
||||||
lot_id: Lot identifier (e.g., "A1-28505-5")
|
|
||||||
index: Image sequence number (1, 2, 3, ...)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Absolute path to saved file
|
|
||||||
"""
|
|
||||||
# Create directory structure: /images/{lot_id}/
|
|
||||||
images_dir = Path(os.getenv('IMAGES_DIR', '/mnt/okcomputer/output/images'))
|
|
||||||
lot_dir = images_dir / lot_id
|
|
||||||
lot_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
# Determine file extension from URL or content-type
|
|
||||||
ext = Path(image_url).suffix or '.jpg'
|
|
||||||
filename = f"{index:03d}{ext}" # 001.jpg, 002.jpg, etc.
|
|
||||||
local_path = lot_dir / filename
|
|
||||||
|
|
||||||
# Download with timeout
|
|
||||||
response = requests.get(image_url, timeout=10)
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
# Save to disk
|
|
||||||
with open(local_path, 'wb') as f:
|
|
||||||
f.write(response.content)
|
|
||||||
|
|
||||||
return str(local_path.absolute())
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 3: Add Unique Constraint to Prevent Duplicates**
|
|
||||||
|
|
||||||
**Migration SQL**:
|
|
||||||
```sql
|
|
||||||
-- Add unique constraint to prevent duplicate image records
|
|
||||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_images_unique
|
|
||||||
ON images(lot_id, url);
|
|
||||||
```
|
|
||||||
|
|
||||||
Add this to your scraper's schema initialization:
|
|
||||||
|
|
||||||
```python
|
|
||||||
def init_database():
|
|
||||||
conn = sqlite3.connect('/mnt/okcomputer/output/cache.db')
|
|
||||||
cursor = conn.cursor()
|
|
||||||
|
|
||||||
# Existing table creation...
|
|
||||||
cursor.execute("""
|
|
||||||
CREATE TABLE IF NOT EXISTS images (...)
|
|
||||||
""")
|
|
||||||
|
|
||||||
# Add unique constraint (NEW)
|
|
||||||
cursor.execute("""
|
|
||||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_images_unique
|
|
||||||
ON images(lot_id, url)
|
|
||||||
""")
|
|
||||||
|
|
||||||
conn.commit()
|
|
||||||
conn.close()
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 4: Handle Image Download Failures Gracefully**
|
|
||||||
|
|
||||||
```python
|
|
||||||
def download_with_retry(image_url, lot_id, index, max_retries=3):
|
|
||||||
"""Downloads image with retry logic."""
|
|
||||||
for attempt in range(max_retries):
|
|
||||||
try:
|
|
||||||
return download_image(image_url, lot_id, index)
|
|
||||||
except requests.exceptions.RequestException as e:
|
|
||||||
if attempt == max_retries - 1:
|
|
||||||
print(f"Failed after {max_retries} attempts: {image_url}")
|
|
||||||
return None # Return None on failure
|
|
||||||
print(f"Retry {attempt + 1}/{max_retries} for {image_url}")
|
|
||||||
time.sleep(2 ** attempt) # Exponential backoff
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Step 5: Update Database Queries**
|
|
||||||
|
|
||||||
Make sure your INSERT uses `INSERT ... ON CONFLICT` to handle re-scraping:
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Good: Handles re-scraping without duplicates
|
|
||||||
db.execute("""
|
|
||||||
INSERT INTO images (lot_id, url, local_path, downloaded)
|
|
||||||
VALUES (?, ?, ?, 1)
|
|
||||||
ON CONFLICT(lot_id, url) DO UPDATE SET
|
|
||||||
local_path = excluded.local_path,
|
|
||||||
downloaded = 1
|
|
||||||
""", (lot_id, img_url, local_path))
|
|
||||||
|
|
||||||
# Bad: Creates duplicates on re-scrape
|
|
||||||
db.execute("""
|
|
||||||
INSERT INTO images (lot_id, url, local_path, downloaded)
|
|
||||||
VALUES (?, ?, ?, 1)
|
|
||||||
""", (lot_id, img_url, local_path))
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📊 Expected Outcomes
|
|
||||||
|
|
||||||
### Before Refactor
|
|
||||||
```sql
|
|
||||||
SELECT COUNT(*) FROM images WHERE downloaded = 0;
|
|
||||||
-- Result: 57,376,293 (57M+ undownloaded!)
|
|
||||||
|
|
||||||
SELECT COUNT(*) FROM images WHERE local_path IS NOT NULL;
|
|
||||||
-- Result: 0 (no files downloaded)
|
|
||||||
```
|
|
||||||
|
|
||||||
### After Refactor
|
|
||||||
```sql
|
|
||||||
SELECT COUNT(*) FROM images WHERE downloaded = 1;
|
|
||||||
-- Result: ~16,807 (one per actual lot image)
|
|
||||||
|
|
||||||
SELECT COUNT(*) FROM images WHERE local_path IS NOT NULL;
|
|
||||||
-- Result: ~16,807 (all downloaded images have paths)
|
|
||||||
|
|
||||||
SELECT COUNT(DISTINCT lot_id, url) FROM images;
|
|
||||||
-- Result: ~16,807 (no duplicates!)
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚀 Deployment Checklist
|
|
||||||
|
|
||||||
### Pre-Deployment
|
|
||||||
- [ ] Back up current database: `cp cache.db cache.db.backup`
|
|
||||||
- [ ] Verify disk space: At least 10GB free for images
|
|
||||||
- [ ] Test download function on 5 sample lots
|
|
||||||
- [ ] Verify `IMAGES_DIR` path exists and is writable
|
|
||||||
|
|
||||||
### Deployment
|
|
||||||
- [ ] Update configuration: `DOWNLOAD_IMAGES = True`
|
|
||||||
- [ ] Run schema migration to add unique index
|
|
||||||
- [ ] Deploy updated scraper code
|
|
||||||
- [ ] Monitor first 100 lots for errors
|
|
||||||
|
|
||||||
### Post-Deployment Verification
|
|
||||||
```sql
|
|
||||||
-- Check download success rate
|
|
||||||
SELECT
|
|
||||||
COUNT(*) as total_images,
|
|
||||||
SUM(CASE WHEN downloaded = 1 THEN 1 ELSE 0 END) as downloaded,
|
|
||||||
SUM(CASE WHEN downloaded = 0 THEN 1 ELSE 0 END) as failed,
|
|
||||||
ROUND(100.0 * SUM(downloaded) / COUNT(*), 2) as success_rate
|
|
||||||
FROM images;
|
|
||||||
|
|
||||||
-- Check for duplicates (should be 0)
|
|
||||||
SELECT lot_id, url, COUNT(*) as dup_count
|
|
||||||
FROM images
|
|
||||||
GROUP BY lot_id, url
|
|
||||||
HAVING COUNT(*) > 1;
|
|
||||||
|
|
||||||
-- Verify file system
|
|
||||||
SELECT COUNT(*) FROM images
|
|
||||||
WHERE downloaded = 1
|
|
||||||
AND local_path IS NOT NULL
|
|
||||||
AND local_path != '';
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔍 Monitoring Process Impact
|
|
||||||
|
|
||||||
The monitoring process (auctiora) will automatically:
|
|
||||||
- ✅ Stop downloading images (network I/O eliminated)
|
|
||||||
- ✅ Only run object detection on `local_path` files
|
|
||||||
- ✅ Query: `WHERE local_path IS NOT NULL AND (labels IS NULL OR labels = '')`
|
|
||||||
- ✅ Update only the `labels` and `processed_at` columns
|
|
||||||
|
|
||||||
**No changes needed in monitoring process!** It's already updated to work with scraper-downloaded images.
|
|
||||||
|
|
||||||
## 🐛 Troubleshooting
|
|
||||||
|
|
||||||
### Problem: "No space left on device"
|
|
||||||
```bash
|
|
||||||
# Check disk usage
|
|
||||||
df -h /mnt/okcomputer/output/images
|
|
||||||
|
|
||||||
# Estimate needed space: ~100KB per image
|
|
||||||
# 16,807 images × 100KB = ~1.6GB
|
|
||||||
```
|
|
||||||
|
|
||||||
### Problem: "Permission denied" when writing images
|
|
||||||
```bash
|
|
||||||
# Fix permissions
|
|
||||||
chmod 755 /mnt/okcomputer/output/images
|
|
||||||
chown -R scraper_user:scraper_group /mnt/okcomputer/output/images
|
|
||||||
```
|
|
||||||
|
|
||||||
### Problem: Images downloading but not recorded in DB
|
|
||||||
```python
|
|
||||||
# Add logging
|
|
||||||
import logging
|
|
||||||
logging.basicConfig(level=logging.INFO)
|
|
||||||
|
|
||||||
def download_image(...):
|
|
||||||
logging.info(f"Downloading {image_url} to {local_path}")
|
|
||||||
# ... download code ...
|
|
||||||
logging.info(f"Saved to {local_path}, size: {os.path.getsize(local_path)} bytes")
|
|
||||||
return local_path
|
|
||||||
```
|
|
||||||
|
|
||||||
### Problem: Duplicate images after refactor
|
|
||||||
```sql
|
|
||||||
-- Find duplicates
|
|
||||||
SELECT lot_id, url, COUNT(*)
|
|
||||||
FROM images
|
|
||||||
GROUP BY lot_id, url
|
|
||||||
HAVING COUNT(*) > 1;
|
|
||||||
|
|
||||||
-- Clean up duplicates (keep newest)
|
|
||||||
DELETE FROM images
|
|
||||||
WHERE id NOT IN (
|
|
||||||
SELECT MAX(id)
|
|
||||||
FROM images
|
|
||||||
GROUP BY lot_id, url
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📈 Performance Comparison
|
|
||||||
|
|
||||||
| Metric | Before (Monitor Downloads) | After (Scraper Downloads) |
|
|
||||||
|----------------------|---------------------------------|---------------------------|
|
|
||||||
| **Image records** | 57,376,293 | ~16,807 |
|
|
||||||
| **Duplicates** | 57,359,486 (99.97%!) | 0 |
|
|
||||||
| **Network I/O** | Monitor process | Scraper process |
|
|
||||||
| **Disk usage** | 0 (URLs only) | ~1.6GB (actual files) |
|
|
||||||
| **Processing speed** | 500ms/image (download + detect) | 100ms/image (detect only) |
|
|
||||||
| **Error handling** | Complex (download failures) | Simple (files exist) |
|
|
||||||
|
|
||||||
## 🎓 Code Examples by Language
|
|
||||||
|
|
||||||
### Python (Most Likely)
|
|
||||||
See **Step 2** above for complete implementation.
|
|
||||||
|
|
||||||
## 📚 References
|
|
||||||
|
|
||||||
- **Current Scraper Architecture**: `wiki/ARCHITECTURE-TROOSTWIJK-SCRAPER.md`
|
|
||||||
- **Database Schema**: `wiki/DATABASE_ARCHITECTURE.md`
|
|
||||||
- **Monitor Changes**: See commit history for `ImageProcessingService.java`, `DatabaseService.java`
|
|
||||||
|
|
||||||
## ✅ Success Criteria
|
|
||||||
|
|
||||||
You'll know the refactor is successful when:
|
|
||||||
|
|
||||||
1. ✅ Database query `SELECT COUNT(*) FROM images` returns ~16,807 (not 57M+)
|
|
||||||
2. ✅ All images have `downloaded = 1` and `local_path IS NOT NULL`
|
|
||||||
3. ✅ No duplicate records: `SELECT lot_id, url, COUNT(*) ... HAVING COUNT(*) > 1` returns 0 rows
|
|
||||||
4. ✅ Monitor logs show "Found N images needing detection" with reasonable numbers
|
|
||||||
5. ✅ Files exist at paths in `local_path` column
|
|
||||||
6. ✅ Monitor process speed increases (100ms vs 500ms per image)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Questions?** Check the troubleshooting section or inspect the monitor's updated code in:
|
|
||||||
- `src/main/java/auctiora/ImageProcessingService.java`
|
|
||||||
- `src/main/java/auctiora/DatabaseService.java:695-719`
|
|
||||||
@@ -1,333 +0,0 @@
|
|||||||
# Test Suite Summary
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
Comprehensive test suite for Troostwijk Auction Monitor with individual test cases for every aspect of the system.
|
|
||||||
|
|
||||||
## Configuration Updates
|
|
||||||
|
|
||||||
### Paths Updated
|
|
||||||
- **Database**: `C:\mnt\okcomputer\output\cache.db`
|
|
||||||
- **Images**: `C:\mnt\okcomputer\output\images\{saleId}\{lotId}\`
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
1. `src/main/java/com/auction/Main.java` - Updated default database path
|
|
||||||
2. `src/main/java/com/auction/ImageProcessingService.java` - Updated image storage path
|
|
||||||
|
|
||||||
## Test Files Created
|
|
||||||
|
|
||||||
### 1. ScraperDataAdapterTest.java (13 test cases)
|
|
||||||
Tests data transformation from external scraper schema to monitor schema:
|
|
||||||
|
|
||||||
- ✅ Extract numeric ID from text format (auction & lot IDs)
|
|
||||||
- ✅ Convert scraper auction format to AuctionInfo
|
|
||||||
- ✅ Handle simple location without country
|
|
||||||
- ✅ Convert scraper lot format to Lot
|
|
||||||
- ✅ Parse bid amounts from various formats (€, $, £, plain numbers)
|
|
||||||
- ✅ Handle missing/null fields gracefully
|
|
||||||
- ✅ Parse various timestamp formats (ISO, SQL)
|
|
||||||
- ✅ Handle invalid timestamps
|
|
||||||
- ✅ Extract type prefix from auction ID
|
|
||||||
- ✅ Handle GBP currency symbol
|
|
||||||
- ✅ Handle "No bids" text
|
|
||||||
- ✅ Parse complex lot IDs (A1-28505-5 → 285055)
|
|
||||||
- ✅ Validate field mapping (lots_count → lotCount, etc.)
|
|
||||||
|
|
||||||
### 2. DatabaseServiceTest.java (15 test cases)
|
|
||||||
Tests database operations and SQLite persistence:
|
|
||||||
|
|
||||||
- ✅ Create database schema successfully
|
|
||||||
- ✅ Insert and retrieve auction
|
|
||||||
- ✅ Update existing auction on conflict (UPSERT)
|
|
||||||
- ✅ Retrieve auctions by country code
|
|
||||||
- ✅ Insert and retrieve lot
|
|
||||||
- ✅ Update lot current bid
|
|
||||||
- ✅ Update lot notification flags
|
|
||||||
- ✅ Insert and retrieve image records
|
|
||||||
- ✅ Count total images
|
|
||||||
- ✅ Handle empty database gracefully
|
|
||||||
- ✅ Handle lots with null closing time
|
|
||||||
- ✅ Retrieve active lots
|
|
||||||
- ✅ Handle concurrent upserts (thread safety)
|
|
||||||
- ✅ Validate foreign key relationships
|
|
||||||
- ✅ Test database indexes performance
|
|
||||||
|
|
||||||
### 3. ImageProcessingServiceTest.java (11 test cases)
|
|
||||||
Tests image downloading and processing pipeline:
|
|
||||||
|
|
||||||
- ✅ Process images for lot with object detection
|
|
||||||
- ✅ Handle image download failure gracefully
|
|
||||||
- ✅ Create directory structure for images
|
|
||||||
- ✅ Save detected objects to database
|
|
||||||
- ✅ Handle empty image list
|
|
||||||
- ✅ Process pending images from database
|
|
||||||
- ✅ Skip lots that already have images
|
|
||||||
- ✅ Handle database errors during image save
|
|
||||||
- ✅ Handle empty detection results
|
|
||||||
- ✅ Handle lots with no existing images
|
|
||||||
- ✅ Capture and verify detection labels
|
|
||||||
|
|
||||||
### 4. ObjectDetectionServiceTest.java (10 test cases)
|
|
||||||
Tests YOLO object detection functionality:
|
|
||||||
|
|
||||||
- ✅ Initialize with missing YOLO models (disabled mode)
|
|
||||||
- ✅ Return empty list when detection is disabled
|
|
||||||
- ✅ Handle invalid image path gracefully
|
|
||||||
- ✅ Handle empty image file
|
|
||||||
- ✅ Initialize successfully with valid model files
|
|
||||||
- ✅ Handle missing class names file
|
|
||||||
- ✅ Detect when model files are missing
|
|
||||||
- ✅ Return unique labels only
|
|
||||||
- ✅ Handle multiple detections in same image
|
|
||||||
- ✅ Respect confidence threshold (0.5)
|
|
||||||
|
|
||||||
### 5. NotificationServiceTest.java (19 test cases)
|
|
||||||
Tests desktop and email notification delivery:
|
|
||||||
|
|
||||||
- ✅ Initialize with desktop-only configuration
|
|
||||||
- ✅ Initialize with SMTP configuration
|
|
||||||
- ✅ Reject invalid SMTP configuration format
|
|
||||||
- ✅ Reject unknown configuration type
|
|
||||||
- ✅ Send desktop notification without error
|
|
||||||
- ✅ Send high priority notification
|
|
||||||
- ✅ Send normal priority notification
|
|
||||||
- ✅ Handle notification when system tray not supported
|
|
||||||
- ✅ Send email notification with valid SMTP config
|
|
||||||
- ✅ Include both desktop and email when SMTP configured
|
|
||||||
- ✅ Handle empty message gracefully
|
|
||||||
- ✅ Handle very long message (1000+ chars)
|
|
||||||
- ✅ Handle special characters in message (€, ⚠️)
|
|
||||||
- ✅ Accept case-insensitive desktop config
|
|
||||||
- ✅ Validate SMTP config parts count
|
|
||||||
- ✅ Handle multiple rapid notifications
|
|
||||||
- ✅ Send bid change notification format
|
|
||||||
- ✅ Send closing alert notification format
|
|
||||||
- ✅ Send object detection notification format
|
|
||||||
|
|
||||||
### 6. TroostwijkMonitorTest.java (12 test cases)
|
|
||||||
Tests monitoring orchestration and coordination:
|
|
||||||
|
|
||||||
- ✅ Initialize monitor successfully
|
|
||||||
- ✅ Print database stats without error
|
|
||||||
- ✅ Process pending images without error
|
|
||||||
- ✅ Handle empty database gracefully
|
|
||||||
- ✅ Track lots in database
|
|
||||||
- ✅ Monitor lots closing soon (< 5 minutes)
|
|
||||||
- ✅ Identify lots with time remaining
|
|
||||||
- ✅ Handle lots without closing time
|
|
||||||
- ✅ Track notification status
|
|
||||||
- ✅ Update bid amounts
|
|
||||||
- ✅ Handle multiple concurrent lot updates
|
|
||||||
- ✅ Handle database with auctions and lots
|
|
||||||
|
|
||||||
### 7. IntegrationTest.java (10 test cases)
|
|
||||||
Tests complete end-to-end workflows:
|
|
||||||
|
|
||||||
- ✅ **Test 1**: Complete scraper data import workflow
|
|
||||||
- Import auction from scraper format
|
|
||||||
- Import multiple lots for auction
|
|
||||||
- Verify data integrity
|
|
||||||
|
|
||||||
- ✅ **Test 2**: Image processing and detection workflow
|
|
||||||
- Add images for lots
|
|
||||||
- Run object detection
|
|
||||||
- Save labels to database
|
|
||||||
|
|
||||||
- ✅ **Test 3**: Bid monitoring and notification workflow
|
|
||||||
- Simulate bid increase
|
|
||||||
- Update database
|
|
||||||
- Send notification
|
|
||||||
- Verify bid was updated
|
|
||||||
|
|
||||||
- ✅ **Test 4**: Closing alert workflow
|
|
||||||
- Create lot closing soon
|
|
||||||
- Send high-priority notification
|
|
||||||
- Mark as notified
|
|
||||||
- Verify notification flag
|
|
||||||
|
|
||||||
- ✅ **Test 5**: Multi-country auction filtering
|
|
||||||
- Add auctions from NL, RO, BE
|
|
||||||
- Filter by country code
|
|
||||||
- Verify filtering works correctly
|
|
||||||
|
|
||||||
- ✅ **Test 6**: Complete monitoring cycle
|
|
||||||
- Print database statistics
|
|
||||||
- Process pending images
|
|
||||||
- Verify database integrity
|
|
||||||
|
|
||||||
- ✅ **Test 7**: Data consistency across services
|
|
||||||
- Verify all auctions have valid data
|
|
||||||
- Verify all lots have valid data
|
|
||||||
- Check referential integrity
|
|
||||||
|
|
||||||
- ✅ **Test 8**: Object detection value estimation workflow
|
|
||||||
- Create lot with detected objects
|
|
||||||
- Add images with labels
|
|
||||||
- Analyze detected objects
|
|
||||||
- Send value estimation notification
|
|
||||||
|
|
||||||
- ✅ **Test 9**: Handle rapid concurrent updates
|
|
||||||
- Concurrent auction insertions
|
|
||||||
- Concurrent lot insertions
|
|
||||||
- Verify all data persisted correctly
|
|
||||||
|
|
||||||
- ✅ **Test 10**: End-to-end notification scenarios
|
|
||||||
- Bid change notification
|
|
||||||
- Closing alert
|
|
||||||
- Object detection notification
|
|
||||||
- Value estimate notification
|
|
||||||
- Viewing day reminder
|
|
||||||
|
|
||||||
## Test Coverage Summary
|
|
||||||
|
|
||||||
| Component | Test Cases | Coverage Areas |
|
|
||||||
|-----------|-----------|----------------|
|
|
||||||
| **ScraperDataAdapter** | 13 | Data transformation, ID parsing, currency parsing, timestamp parsing |
|
|
||||||
| **DatabaseService** | 15 | CRUD operations, concurrency, foreign keys, indexes |
|
|
||||||
| **ImageProcessingService** | 11 | Download, detection integration, error handling |
|
|
||||||
| **ObjectDetectionService** | 10 | YOLO initialization, detection, confidence threshold |
|
|
||||||
| **NotificationService** | 19 | Desktop/Email, priority levels, special chars, formats |
|
|
||||||
| **TroostwijkMonitor** | 12 | Orchestration, monitoring, bid tracking, alerts |
|
|
||||||
| **Integration** | 10 | End-to-end workflows, multi-service coordination |
|
|
||||||
| **TOTAL** | **90** | **Complete system coverage** |
|
|
||||||
|
|
||||||
## Key Testing Patterns
|
|
||||||
|
|
||||||
### 1. Isolation Testing
|
|
||||||
Each component tested independently with mocks:
|
|
||||||
```java
|
|
||||||
mockDb = mock(DatabaseService.class);
|
|
||||||
mockDetector = mock(ObjectDetectionService.class);
|
|
||||||
service = new ImageProcessingService(mockDb, mockDetector);
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Integration Testing
|
|
||||||
Components tested together for realistic scenarios:
|
|
||||||
```java
|
|
||||||
db → imageProcessor → detector → notifier
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Concurrency Testing
|
|
||||||
Thread safety verified with parallel operations:
|
|
||||||
```java
|
|
||||||
Thread t1 = new Thread(() -> db.upsertLot(...));
|
|
||||||
Thread t2 = new Thread(() -> db.upsertLot(...));
|
|
||||||
t1.start(); t2.start();
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Error Handling
|
|
||||||
Graceful degradation tested throughout:
|
|
||||||
```java
|
|
||||||
assertDoesNotThrow(() -> service.process(invalidInput));
|
|
||||||
```
|
|
||||||
|
|
||||||
## Running the Tests
|
|
||||||
|
|
||||||
### Run All Tests
|
|
||||||
```bash
|
|
||||||
mvn test
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Specific Test Class
|
|
||||||
```bash
|
|
||||||
mvn test -Dtest=ScraperDataAdapterTest
|
|
||||||
mvn test -Dtest=IntegrationTest
|
|
||||||
```
|
|
||||||
|
|
||||||
### Run Single Test Method
|
|
||||||
```bash
|
|
||||||
mvn test -Dtest=IntegrationTest#testCompleteScraperImportWorkflow
|
|
||||||
```
|
|
||||||
|
|
||||||
### Generate Coverage Report
|
|
||||||
```bash
|
|
||||||
mvn jacoco:prepare-agent test jacoco:report
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test Data Cleanup
|
|
||||||
All tests use temporary databases that are automatically cleaned up:
|
|
||||||
```java
|
|
||||||
@AfterAll
|
|
||||||
void tearDown() throws Exception {
|
|
||||||
Files.deleteIfExists(Paths.get(testDbPath));
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Integration Scenarios Covered
|
|
||||||
|
|
||||||
### Scenario 1: New Auction Discovery
|
|
||||||
1. External scraper finds new auction
|
|
||||||
2. Data imported via ScraperDataAdapter
|
|
||||||
3. Lots added to database
|
|
||||||
4. Images downloaded
|
|
||||||
5. Object detection runs
|
|
||||||
6. Notification sent to user
|
|
||||||
|
|
||||||
### Scenario 2: Bid Monitoring
|
|
||||||
1. Monitor checks API every hour
|
|
||||||
2. Detects bid increase
|
|
||||||
3. Updates database
|
|
||||||
4. Sends notification
|
|
||||||
5. User can place counter-bid
|
|
||||||
|
|
||||||
### Scenario 3: Closing Alert
|
|
||||||
1. Monitor checks closing times
|
|
||||||
2. Lot closing in < 5 minutes
|
|
||||||
3. High-priority notification sent
|
|
||||||
4. Flag updated to prevent duplicates
|
|
||||||
5. User can place final bid
|
|
||||||
|
|
||||||
### Scenario 4: Value Estimation
|
|
||||||
1. Images downloaded
|
|
||||||
2. YOLO detects objects
|
|
||||||
3. Labels saved to database
|
|
||||||
4. Value estimated (future feature)
|
|
||||||
5. Notification sent with estimate
|
|
||||||
|
|
||||||
## Dependencies Required for Tests
|
|
||||||
|
|
||||||
```xml
|
|
||||||
<dependencies>
|
|
||||||
<!-- JUnit 5 -->
|
|
||||||
<dependency>
|
|
||||||
<groupId>org.junit.jupiter</groupId>
|
|
||||||
<artifactId>junit-jupiter</artifactId>
|
|
||||||
<version>5.10.0</version>
|
|
||||||
<scope>test</scope>
|
|
||||||
</dependency>
|
|
||||||
|
|
||||||
<!-- Mockito -->
|
|
||||||
<dependency>
|
|
||||||
<groupId>org.mockito</groupId>
|
|
||||||
<artifactId>mockito-core</artifactId>
|
|
||||||
<version>5.5.0</version>
|
|
||||||
<scope>test</scope>
|
|
||||||
</dependency>
|
|
||||||
|
|
||||||
<!-- Mockito JUnit Jupiter -->
|
|
||||||
<dependency>
|
|
||||||
<groupId>org.mockito</groupId>
|
|
||||||
<artifactId>mockito-junit-jupiter</artifactId>
|
|
||||||
<version>5.5.0</version>
|
|
||||||
<scope>test</scope>
|
|
||||||
</dependency>
|
|
||||||
</dependencies>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- All tests are independent and can run in any order
|
|
||||||
- Tests use in-memory or temporary databases
|
|
||||||
- No actual HTTP requests made (except in integration tests)
|
|
||||||
- YOLO models are optional (tests work in disabled mode)
|
|
||||||
- Notifications are tested but may not display in headless environments
|
|
||||||
- Tests document expected behavior for each component
|
|
||||||
|
|
||||||
## Future Test Enhancements
|
|
||||||
|
|
||||||
1. **Mock HTTP Server** for realistic image download testing
|
|
||||||
2. **Test Containers** for full database integration
|
|
||||||
3. **Performance Tests** for large datasets (1000+ auctions)
|
|
||||||
4. **Stress Tests** for concurrent monitoring scenarios
|
|
||||||
5. **UI Tests** for notification display (if GUI added)
|
|
||||||
6. **API Tests** for Troostwijk API integration
|
|
||||||
7. **Value Estimation** tests (when algorithm implemented)
|
|
||||||
@@ -1,537 +0,0 @@
|
|||||||
## Troostwijk Auction Monitor - Workflow Integration Guide
|
|
||||||
|
|
||||||
Complete guide for running the auction monitoring system with scheduled workflows, cron jobs, and event-driven triggers.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
1. [Overview](#overview)
|
|
||||||
2. [Running Modes](#running-modes)
|
|
||||||
3. [Workflow Orchestration](#workflow-orchestration)
|
|
||||||
4. [Windows Scheduling](#windows-scheduling)
|
|
||||||
5. [Event-Driven Triggers](#event-driven-triggers)
|
|
||||||
6. [Configuration](#configuration)
|
|
||||||
7. [Monitoring & Debugging](#monitoring--debugging)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The Troostwijk Auction Monitor supports multiple execution modes:
|
|
||||||
|
|
||||||
- **Workflow Mode** (Recommended): Continuous operation with built-in scheduling
|
|
||||||
- **Once Mode**: Single execution for external schedulers (Windows Task Scheduler, cron)
|
|
||||||
- **Legacy Mode**: Original monitoring approach
|
|
||||||
- **Status Mode**: Quick status check
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Running Modes
|
|
||||||
|
|
||||||
### 1. Workflow Mode (Default - Recommended)
|
|
||||||
|
|
||||||
**Runs all workflows continuously with built-in scheduling.**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Windows
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar workflow
|
|
||||||
|
|
||||||
# Or simply (workflow is default)
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
|
|
||||||
|
|
||||||
# Using batch script
|
|
||||||
run-workflow.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
- ✅ Imports scraper data every 30 minutes
|
|
||||||
- ✅ Processes images every 1 hour
|
|
||||||
- ✅ Monitors bids every 15 minutes
|
|
||||||
- ✅ Checks closing times every 5 minutes
|
|
||||||
|
|
||||||
**Best for:**
|
|
||||||
- Production deployment
|
|
||||||
- Long-running services
|
|
||||||
- Development/testing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. Once Mode (For External Schedulers)
|
|
||||||
|
|
||||||
**Runs complete workflow once and exits.**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Windows
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar once
|
|
||||||
|
|
||||||
# Using batch script
|
|
||||||
run-once.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
1. Imports scraper data
|
|
||||||
2. Processes pending images
|
|
||||||
3. Monitors bids
|
|
||||||
4. Checks closing times
|
|
||||||
5. Exits
|
|
||||||
|
|
||||||
**Best for:**
|
|
||||||
- Windows Task Scheduler
|
|
||||||
- Cron jobs (Linux/Mac)
|
|
||||||
- Manual execution
|
|
||||||
- Testing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3. Legacy Mode
|
|
||||||
|
|
||||||
**Original monitoring approach (backward compatibility).**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar legacy
|
|
||||||
```
|
|
||||||
|
|
||||||
**Best for:**
|
|
||||||
- Maintaining existing deployments
|
|
||||||
- Troubleshooting
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4. Status Mode
|
|
||||||
|
|
||||||
**Shows current status and exits.**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar status
|
|
||||||
|
|
||||||
# Using batch script
|
|
||||||
check-status.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
**Output:**
|
|
||||||
```
|
|
||||||
📊 Workflow Status:
|
|
||||||
Running: No
|
|
||||||
Auctions: 25
|
|
||||||
Lots: 150
|
|
||||||
Images: 300
|
|
||||||
Closing soon (< 30 min): 5
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Workflow Orchestration
|
|
||||||
|
|
||||||
The `WorkflowOrchestrator` coordinates 4 scheduled workflows:
|
|
||||||
|
|
||||||
### Workflow 1: Scraper Data Import
|
|
||||||
**Frequency:** Every 30 minutes
|
|
||||||
**Purpose:** Import new auctions and lots from external scraper
|
|
||||||
|
|
||||||
**Process:**
|
|
||||||
1. Import auctions from scraper database
|
|
||||||
2. Import lots from scraper database
|
|
||||||
3. Import image URLs
|
|
||||||
4. Send notification if significant data imported
|
|
||||||
|
|
||||||
**Code Location:** `WorkflowOrchestrator.java:110`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Workflow 2: Image Processing
|
|
||||||
**Frequency:** Every 1 hour
|
|
||||||
**Purpose:** Download images and run object detection
|
|
||||||
|
|
||||||
**Process:**
|
|
||||||
1. Get unprocessed images from database
|
|
||||||
2. Download each image
|
|
||||||
3. Run YOLO object detection
|
|
||||||
4. Save labels to database
|
|
||||||
5. Send notification for interesting detections (3+ objects)
|
|
||||||
|
|
||||||
**Code Location:** `WorkflowOrchestrator.java:150`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Workflow 3: Bid Monitoring
|
|
||||||
**Frequency:** Every 15 minutes
|
|
||||||
**Purpose:** Check for bid changes and send notifications
|
|
||||||
|
|
||||||
**Process:**
|
|
||||||
1. Get all active lots
|
|
||||||
2. Check for bid changes (via external scraper updates)
|
|
||||||
3. Send notifications for bid increases
|
|
||||||
|
|
||||||
**Code Location:** `WorkflowOrchestrator.java:210`
|
|
||||||
|
|
||||||
**Note:** The external scraper updates bids; this workflow monitors and notifies.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Workflow 4: Closing Alerts
|
|
||||||
**Frequency:** Every 5 minutes
|
|
||||||
**Purpose:** Send alerts for lots closing soon
|
|
||||||
|
|
||||||
**Process:**
|
|
||||||
1. Get all active lots
|
|
||||||
2. Check closing times
|
|
||||||
3. Send high-priority notification for lots closing in < 5 min
|
|
||||||
4. Mark as notified to prevent duplicates
|
|
||||||
|
|
||||||
**Code Location:** `WorkflowOrchestrator.java:240`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Windows Scheduling
|
|
||||||
|
|
||||||
### Option A: Use Built-in Workflow Mode (Recommended)
|
|
||||||
|
|
||||||
**Run as a Windows Service or startup application:**
|
|
||||||
|
|
||||||
1. Create shortcut to `run-workflow.bat`
|
|
||||||
2. Place in: `C:\Users\[YourUser]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup`
|
|
||||||
3. Monitor will start automatically on login
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option B: Windows Task Scheduler (Once Mode)
|
|
||||||
|
|
||||||
**Automated setup:**
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
# Run PowerShell as Administrator
|
|
||||||
.\setup-windows-task.ps1
|
|
||||||
```
|
|
||||||
|
|
||||||
This creates two tasks:
|
|
||||||
- `TroostwijkMonitor-Workflow`: Runs every 30 minutes
|
|
||||||
- `TroostwijkMonitor-StatusCheck`: Runs every 6 hours
|
|
||||||
|
|
||||||
**Manual setup:**
|
|
||||||
|
|
||||||
1. Open Task Scheduler
|
|
||||||
2. Create Basic Task
|
|
||||||
3. Configure:
|
|
||||||
- **Name:** `TroostwijkMonitor`
|
|
||||||
- **Trigger:** Every 30 minutes
|
|
||||||
- **Action:** Start a program
|
|
||||||
- **Program:** `java`
|
|
||||||
- **Arguments:** `-jar "C:\path\to\troostwijk-scraper.jar" once`
|
|
||||||
- **Start in:** `C:\path\to\project`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Option C: Multiple Scheduled Tasks (Fine-grained Control)
|
|
||||||
|
|
||||||
Create separate tasks for each workflow:
|
|
||||||
|
|
||||||
| Task | Frequency | Command |
|
|
||||||
|------|-----------|---------|
|
|
||||||
| Import Data | Every 30 min | `run-once.bat` |
|
|
||||||
| Process Images | Every 1 hour | `run-once.bat` |
|
|
||||||
| Check Bids | Every 15 min | `run-once.bat` |
|
|
||||||
| Closing Alerts | Every 5 min | `run-once.bat` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Event-Driven Triggers
|
|
||||||
|
|
||||||
The orchestrator supports event-driven execution:
|
|
||||||
|
|
||||||
### 1. New Auction Discovered
|
|
||||||
|
|
||||||
```java
|
|
||||||
orchestrator.onNewAuctionDiscovered(auctionInfo);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Triggered when:**
|
|
||||||
- External scraper finds new auction
|
|
||||||
|
|
||||||
**Actions:**
|
|
||||||
- Insert to database
|
|
||||||
- Send notification
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. Bid Change Detected
|
|
||||||
|
|
||||||
```java
|
|
||||||
orchestrator.onBidChange(lot, previousBid, newBid);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Triggered when:**
|
|
||||||
- Bid increases on monitored lot
|
|
||||||
|
|
||||||
**Actions:**
|
|
||||||
- Update database
|
|
||||||
- Send notification: "Nieuw bod op kavel X: €Y (was €Z)"
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3. Objects Detected
|
|
||||||
|
|
||||||
```java
|
|
||||||
orchestrator.onObjectsDetected(lotId, labels);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Triggered when:**
|
|
||||||
- YOLO detects 2+ objects in image
|
|
||||||
|
|
||||||
**Actions:**
|
|
||||||
- Send notification: "Lot X contains: car, truck, machinery"
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Database location
|
|
||||||
set DATABASE_FILE=C:\mnt\okcomputer\output\cache.db
|
|
||||||
|
|
||||||
# Notification configuration
|
|
||||||
set NOTIFICATION_CONFIG=desktop
|
|
||||||
|
|
||||||
# Or for email notifications
|
|
||||||
set NOTIFICATION_CONFIG=smtp:your@gmail.com:app_password:recipient@example.com
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configuration Files
|
|
||||||
|
|
||||||
**YOLO Model Paths** (`Main.java:35-37`):
|
|
||||||
```java
|
|
||||||
String yoloCfg = "models/yolov4.cfg";
|
|
||||||
String yoloWeights = "models/yolov4.weights";
|
|
||||||
String yoloClasses = "models/coco.names";
|
|
||||||
```
|
|
||||||
|
|
||||||
### Customizing Schedules
|
|
||||||
|
|
||||||
Edit `WorkflowOrchestrator.java` to change frequencies:
|
|
||||||
|
|
||||||
```java
|
|
||||||
// Change from 30 minutes to 15 minutes
|
|
||||||
scheduler.scheduleAtFixedRate(() -> {
|
|
||||||
// ... scraper import logic
|
|
||||||
}, 0, 15, TimeUnit.MINUTES); // Changed from 30
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Monitoring & Debugging
|
|
||||||
|
|
||||||
### Check Status
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Quick status check
|
|
||||||
java -jar troostwijk-monitor.jar status
|
|
||||||
|
|
||||||
# Or
|
|
||||||
check-status.bat
|
|
||||||
```
|
|
||||||
|
|
||||||
### View Logs
|
|
||||||
|
|
||||||
Workflows print timestamped logs:
|
|
||||||
|
|
||||||
```
|
|
||||||
📥 [WORKFLOW 1] Importing scraper data...
|
|
||||||
→ Imported 5 auctions
|
|
||||||
→ Imported 25 lots
|
|
||||||
→ Found 50 unprocessed images
|
|
||||||
✓ Scraper import completed in 1250ms
|
|
||||||
|
|
||||||
🖼️ [WORKFLOW 2] Processing pending images...
|
|
||||||
→ Processing 50 images
|
|
||||||
✓ Processed 50 images, detected objects in 12 (15.3s)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
#### 1. No data being imported
|
|
||||||
|
|
||||||
**Problem:** External scraper not running
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Check if scraper is running and populating database
|
|
||||||
sqlite3 C:\mnt\okcomputer\output\cache.db "SELECT COUNT(*) FROM auctions;"
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 2. Images not downloading
|
|
||||||
|
|
||||||
**Problem:** No internet connection or invalid URLs
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
- Check network connectivity
|
|
||||||
- Verify image URLs in database
|
|
||||||
- Check firewall settings
|
|
||||||
|
|
||||||
#### 3. Notifications not showing
|
|
||||||
|
|
||||||
**Problem:** System tray not available
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
- Use email notifications instead
|
|
||||||
- Check notification permissions in Windows
|
|
||||||
|
|
||||||
#### 4. Workflows not running
|
|
||||||
|
|
||||||
**Problem:** Application crashed or was stopped
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
- Check Task Scheduler logs
|
|
||||||
- Review application logs
|
|
||||||
- Restart in workflow mode
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Integration Examples
|
|
||||||
|
|
||||||
### Example 1: Complete Automated Workflow
|
|
||||||
|
|
||||||
**Setup:**
|
|
||||||
1. External scraper runs continuously, populating database
|
|
||||||
2. This monitor runs in workflow mode
|
|
||||||
3. Notifications sent to desktop + email
|
|
||||||
|
|
||||||
**Result:**
|
|
||||||
- New auctions → Notification within 30 min
|
|
||||||
- New images → Processed within 1 hour
|
|
||||||
- Bid changes → Notification within 15 min
|
|
||||||
- Closing alerts → Notification within 5 min
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Example 2: On-Demand Processing
|
|
||||||
|
|
||||||
**Setup:**
|
|
||||||
1. External scraper runs once per day (cron/Task Scheduler)
|
|
||||||
2. This monitor runs in once mode after scraper completes
|
|
||||||
|
|
||||||
**Script:**
|
|
||||||
```bash
|
|
||||||
# run-daily.bat
|
|
||||||
@echo off
|
|
||||||
REM Run scraper first
|
|
||||||
python scraper.py
|
|
||||||
|
|
||||||
REM Wait for completion
|
|
||||||
timeout /t 30
|
|
||||||
|
|
||||||
REM Run monitor once
|
|
||||||
java -jar troostwijk-monitor.jar once
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Example 3: Event-Driven with External Integration
|
|
||||||
|
|
||||||
**Setup:**
|
|
||||||
1. External system calls orchestrator events
|
|
||||||
2. Workflows run on-demand
|
|
||||||
|
|
||||||
**Java code:**
|
|
||||||
```java
|
|
||||||
WorkflowOrchestrator orchestrator = new WorkflowOrchestrator(...);
|
|
||||||
|
|
||||||
// When external scraper finds new auction
|
|
||||||
AuctionInfo newAuction = parseScraperData();
|
|
||||||
orchestrator.onNewAuctionDiscovered(newAuction);
|
|
||||||
|
|
||||||
// When bid detected
|
|
||||||
orchestrator.onBidChange(lot, 100.0, 150.0);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Advanced Topics
|
|
||||||
|
|
||||||
### Custom Workflows
|
|
||||||
|
|
||||||
Add custom workflows to `WorkflowOrchestrator`:
|
|
||||||
|
|
||||||
```java
|
|
||||||
// Workflow 5: Value Estimation (every 2 hours)
|
|
||||||
scheduler.scheduleAtFixedRate(() -> {
|
|
||||||
try {
|
|
||||||
Console.println("💰 [WORKFLOW 5] Estimating values...");
|
|
||||||
|
|
||||||
var lotsWithImages = db.getLotsWithImages();
|
|
||||||
for (var lot : lotsWithImages) {
|
|
||||||
var images = db.getImagesForLot(lot.lotId());
|
|
||||||
double estimatedValue = estimateValue(images);
|
|
||||||
|
|
||||||
// Update database
|
|
||||||
db.updateLotEstimatedValue(lot.lotId(), estimatedValue);
|
|
||||||
|
|
||||||
// Notify if high value
|
|
||||||
if (estimatedValue > 5000) {
|
|
||||||
notifier.sendNotification(
|
|
||||||
String.format("High value lot detected: %d (€%.2f)",
|
|
||||||
lot.lotId(), estimatedValue),
|
|
||||||
"Value Alert", 1
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (Exception e) {
|
|
||||||
Console.println(" ❌ Value estimation failed: " + e.getMessage());
|
|
||||||
}
|
|
||||||
}, 10, 120, TimeUnit.MINUTES);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Webhook Integration
|
|
||||||
|
|
||||||
Trigger workflows via HTTP webhooks:
|
|
||||||
|
|
||||||
```java
|
|
||||||
// In a separate web server (e.g., using Javalin)
|
|
||||||
Javalin app = Javalin.create().start(7070);
|
|
||||||
|
|
||||||
app.post("/webhook/new-auction", ctx -> {
|
|
||||||
AuctionInfo auction = ctx.bodyAsClass(AuctionInfo.class);
|
|
||||||
orchestrator.onNewAuctionDiscovered(auction);
|
|
||||||
ctx.result("OK");
|
|
||||||
});
|
|
||||||
|
|
||||||
app.post("/webhook/bid-change", ctx -> {
|
|
||||||
BidChange change = ctx.bodyAsClass(BidChange.class);
|
|
||||||
orchestrator.onBidChange(change.lot, change.oldBid, change.newBid);
|
|
||||||
ctx.result("OK");
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
| Mode | Use Case | Scheduling | Best For |
|
|
||||||
|------|----------|------------|----------|
|
|
||||||
| **workflow** | Continuous operation | Built-in (Java) | Production, development |
|
|
||||||
| **once** | Single execution | External (Task Scheduler) | Cron jobs, on-demand |
|
|
||||||
| **legacy** | Backward compatibility | Built-in (Java) | Existing deployments |
|
|
||||||
| **status** | Quick check | Manual/External | Health checks, debugging |
|
|
||||||
|
|
||||||
**Recommended Setup for Windows:**
|
|
||||||
1. Install as Windows Service OR
|
|
||||||
2. Add to Startup folder (workflow mode) OR
|
|
||||||
3. Use Task Scheduler (once mode, every 30 min)
|
|
||||||
|
|
||||||
**All workflows automatically:**
|
|
||||||
- Import data from scraper
|
|
||||||
- Process images
|
|
||||||
- Detect objects
|
|
||||||
- Monitor bids
|
|
||||||
- Send notifications
|
|
||||||
- Handle errors gracefully
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Support
|
|
||||||
|
|
||||||
For issues or questions:
|
|
||||||
- Check `TEST_SUITE_SUMMARY.md` for test coverage
|
|
||||||
- Review code in `WorkflowOrchestrator.java`
|
|
||||||
- Run `java -jar troostwijk-monitor.jar status` for diagnostics
|
|
||||||
128
fix-schema.sql
128
fix-schema.sql
@@ -1,128 +0,0 @@
|
|||||||
-- Schema Fix Script for Server Database
|
|
||||||
-- This script migrates auction_id and lot_id from BIGINT to TEXT to match scraper format
|
|
||||||
-- The scraper uses TEXT IDs like "A7-40063-2" but DatabaseService.java was creating BIGINT columns
|
|
||||||
|
|
||||||
-- Step 1: Backup existing data
|
|
||||||
CREATE TABLE IF NOT EXISTS auctions_backup AS SELECT * FROM auctions;
|
|
||||||
CREATE TABLE IF NOT EXISTS lots_backup AS SELECT * FROM lots;
|
|
||||||
CREATE TABLE IF NOT EXISTS images_backup AS SELECT * FROM images;
|
|
||||||
|
|
||||||
-- Step 2: Drop existing tables (CASCADE would drop foreign keys)
|
|
||||||
DROP TABLE IF EXISTS images;
|
|
||||||
DROP TABLE IF EXISTS lots;
|
|
||||||
DROP TABLE IF EXISTS auctions;
|
|
||||||
|
|
||||||
-- Step 3: Recreate auctions table with TEXT primary key (matching scraper format)
|
|
||||||
CREATE TABLE auctions (
|
|
||||||
auction_id TEXT PRIMARY KEY,
|
|
||||||
title TEXT NOT NULL,
|
|
||||||
location TEXT,
|
|
||||||
city TEXT,
|
|
||||||
country TEXT,
|
|
||||||
url TEXT NOT NULL UNIQUE,
|
|
||||||
type TEXT,
|
|
||||||
lot_count INTEGER DEFAULT 0,
|
|
||||||
closing_time TEXT,
|
|
||||||
discovered_at INTEGER
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Step 4: Recreate lots table with TEXT primary key (matching scraper format)
|
|
||||||
CREATE TABLE lots (
|
|
||||||
lot_id TEXT PRIMARY KEY,
|
|
||||||
sale_id TEXT,
|
|
||||||
auction_id TEXT,
|
|
||||||
title TEXT,
|
|
||||||
description TEXT,
|
|
||||||
manufacturer TEXT,
|
|
||||||
type TEXT,
|
|
||||||
year INTEGER,
|
|
||||||
category TEXT,
|
|
||||||
current_bid REAL,
|
|
||||||
currency TEXT DEFAULT 'EUR',
|
|
||||||
url TEXT UNIQUE,
|
|
||||||
closing_time TEXT,
|
|
||||||
closing_notified INTEGER DEFAULT 0,
|
|
||||||
starting_bid REAL,
|
|
||||||
minimum_bid REAL,
|
|
||||||
status TEXT,
|
|
||||||
brand TEXT,
|
|
||||||
model TEXT,
|
|
||||||
attributes_json TEXT,
|
|
||||||
first_bid_time TEXT,
|
|
||||||
last_bid_time TEXT,
|
|
||||||
bid_velocity REAL,
|
|
||||||
bid_increment REAL,
|
|
||||||
year_manufactured INTEGER,
|
|
||||||
condition_score REAL,
|
|
||||||
condition_description TEXT,
|
|
||||||
serial_number TEXT,
|
|
||||||
damage_description TEXT,
|
|
||||||
followers_count INTEGER DEFAULT 0,
|
|
||||||
estimated_min_price REAL,
|
|
||||||
estimated_max_price REAL,
|
|
||||||
lot_condition TEXT,
|
|
||||||
appearance TEXT,
|
|
||||||
estimated_min REAL,
|
|
||||||
estimated_max REAL,
|
|
||||||
next_bid_step_cents INTEGER,
|
|
||||||
condition TEXT,
|
|
||||||
category_path TEXT,
|
|
||||||
city_location TEXT,
|
|
||||||
country_code TEXT,
|
|
||||||
bidding_status TEXT,
|
|
||||||
packaging TEXT,
|
|
||||||
quantity INTEGER,
|
|
||||||
vat REAL,
|
|
||||||
buyer_premium_percentage REAL,
|
|
||||||
remarks TEXT,
|
|
||||||
reserve_price REAL,
|
|
||||||
reserve_met INTEGER,
|
|
||||||
view_count INTEGER,
|
|
||||||
bid_count INTEGER,
|
|
||||||
viewing_time TEXT,
|
|
||||||
pickup_date TEXT,
|
|
||||||
location TEXT,
|
|
||||||
scraped_at TEXT,
|
|
||||||
FOREIGN KEY (auction_id) REFERENCES auctions(auction_id),
|
|
||||||
FOREIGN KEY (sale_id) REFERENCES auctions(auction_id)
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Step 5: Recreate images table
|
|
||||||
CREATE TABLE images (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT,
|
|
||||||
url TEXT,
|
|
||||||
local_path TEXT,
|
|
||||||
labels TEXT,
|
|
||||||
processed_at INTEGER,
|
|
||||||
downloaded INTEGER DEFAULT 0,
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Step 6: Create bid_history table if it doesn't exist
|
|
||||||
CREATE TABLE IF NOT EXISTS bid_history (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT,
|
|
||||||
bid_amount REAL,
|
|
||||||
bid_time TEXT,
|
|
||||||
is_autobid INTEGER DEFAULT 0,
|
|
||||||
bidder_id TEXT,
|
|
||||||
bidder_number INTEGER,
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
);
|
|
||||||
|
|
||||||
-- Step 7: Restore data from backup (converting BIGINT to TEXT if needed)
|
|
||||||
INSERT OR IGNORE INTO auctions SELECT * FROM auctions_backup;
|
|
||||||
INSERT OR IGNORE INTO lots SELECT * FROM lots_backup;
|
|
||||||
INSERT OR IGNORE INTO images SELECT * FROM images_backup;
|
|
||||||
|
|
||||||
-- Step 8: Create indexes for performance
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_auctions_country ON auctions(country);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_lots_sale_id ON lots(sale_id);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_lots_auction_id ON lots(auction_id);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_images_lot_id ON images(lot_id);
|
|
||||||
|
|
||||||
-- Step 9: Clean up backup tables (optional - comment out if you want to keep backups)
|
|
||||||
-- DROP TABLE auctions_backup;
|
|
||||||
-- DROP TABLE lots_backup;
|
|
||||||
-- DROP TABLE images_backup;
|
|
||||||
189
k8s/README.md
189
k8s/README.md
@@ -1,189 +0,0 @@
|
|||||||
# Kubernetes Deployment for Auction Monitor
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### 1. Build and Push Docker Image
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build image
|
|
||||||
docker build -t your-registry/auction-monitor:latest .
|
|
||||||
|
|
||||||
# Push to registry
|
|
||||||
docker push your-registry/auction-monitor:latest
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Update deployment.yaml
|
|
||||||
|
|
||||||
Edit `deployment.yaml` and replace:
|
|
||||||
- `image: auction-monitor:latest` with your image
|
|
||||||
- `auction-monitor.yourdomain.com` with your domain
|
|
||||||
|
|
||||||
### 3. Deploy to Kubernetes
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Apply all resources
|
|
||||||
kubectl apply -f k8s/deployment.yaml
|
|
||||||
|
|
||||||
# Or apply individually
|
|
||||||
kubectl apply -f k8s/namespace.yaml
|
|
||||||
kubectl apply -f k8s/configmap.yaml
|
|
||||||
kubectl apply -f k8s/secret.yaml
|
|
||||||
kubectl apply -f k8s/deployment.yaml
|
|
||||||
kubectl apply -f k8s/service.yaml
|
|
||||||
kubectl apply -f k8s/ingress.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Verify Deployment
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check pods
|
|
||||||
kubectl get pods -n auction-monitor
|
|
||||||
|
|
||||||
# Check services
|
|
||||||
kubectl get svc -n auction-monitor
|
|
||||||
|
|
||||||
# Check ingress
|
|
||||||
kubectl get ingress -n auction-monitor
|
|
||||||
|
|
||||||
# View logs
|
|
||||||
kubectl logs -f deployment/auction-monitor -n auction-monitor
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Access Application
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Port forward for local access
|
|
||||||
kubectl port-forward svc/auction-monitor 8081:8081 -n auction-monitor
|
|
||||||
|
|
||||||
# Access API
|
|
||||||
curl http://localhost:8081/api/monitor/status
|
|
||||||
|
|
||||||
# Access health check
|
|
||||||
curl http://localhost:8081/health/live
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### ConfigMap
|
|
||||||
|
|
||||||
Edit workflow schedules in `configMap`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
data:
|
|
||||||
AUCTION_WORKFLOW_SCRAPER_IMPORT_CRON: "0 */30 * * * ?" # Every 30 min
|
|
||||||
AUCTION_WORKFLOW_IMAGE_PROCESSING_CRON: "0 0 * * * ?" # Every 1 hour
|
|
||||||
AUCTION_WORKFLOW_BID_MONITORING_CRON: "0 */15 * * * ?" # Every 15 min
|
|
||||||
AUCTION_WORKFLOW_CLOSING_ALERTS_CRON: "0 */5 * * * ?" # Every 5 min
|
|
||||||
```
|
|
||||||
|
|
||||||
### Secrets
|
|
||||||
|
|
||||||
Update notification configuration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Create secret
|
|
||||||
kubectl create secret generic auction-secrets \
|
|
||||||
--from-literal=notification-config='smtp:user@gmail.com:password:recipient@example.com' \
|
|
||||||
-n auction-monitor
|
|
||||||
|
|
||||||
# Or edit existing
|
|
||||||
kubectl edit secret auction-secrets -n auction-monitor
|
|
||||||
```
|
|
||||||
|
|
||||||
## Scaling
|
|
||||||
|
|
||||||
### Manual Scaling
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Scale to 3 replicas
|
|
||||||
kubectl scale deployment auction-monitor --replicas=3 -n auction-monitor
|
|
||||||
```
|
|
||||||
|
|
||||||
### Auto Scaling
|
|
||||||
|
|
||||||
HPA is configured in `deployment.yaml`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
spec:
|
|
||||||
minReplicas: 1
|
|
||||||
maxReplicas: 3
|
|
||||||
metrics:
|
|
||||||
- type: Resource
|
|
||||||
resource:
|
|
||||||
name: cpu
|
|
||||||
target:
|
|
||||||
averageUtilization: 80
|
|
||||||
```
|
|
||||||
|
|
||||||
View HPA status:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl get hpa -n auction-monitor
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
### Health Checks
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Liveness
|
|
||||||
kubectl exec -it deployment/auction-monitor -n auction-monitor -- \
|
|
||||||
wget -qO- http://localhost:8081/health/live
|
|
||||||
|
|
||||||
# Readiness
|
|
||||||
kubectl exec -it deployment/auction-monitor -n auction-monitor -- \
|
|
||||||
wget -qO- http://localhost:8081/health/ready
|
|
||||||
```
|
|
||||||
|
|
||||||
### Logs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Follow logs
|
|
||||||
kubectl logs -f deployment/auction-monitor -n auction-monitor
|
|
||||||
|
|
||||||
# Logs from all pods
|
|
||||||
kubectl logs -f -l app=auction-monitor -n auction-monitor
|
|
||||||
|
|
||||||
# Previous pod logs
|
|
||||||
kubectl logs deployment/auction-monitor --previous -n auction-monitor
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Pod not starting
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Describe pod
|
|
||||||
kubectl describe pod -l app=auction-monitor -n auction-monitor
|
|
||||||
|
|
||||||
# Check events
|
|
||||||
kubectl get events -n auction-monitor --sort-by='.lastTimestamp'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Database issues
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check PVC
|
|
||||||
kubectl get pvc -n auction-monitor
|
|
||||||
|
|
||||||
# Check volume mount
|
|
||||||
kubectl exec -it deployment/auction-monitor -n auction-monitor -- ls -la /data
|
|
||||||
```
|
|
||||||
|
|
||||||
### Network issues
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Test service
|
|
||||||
kubectl run -it --rm debug --image=busybox --restart=Never -n auction-monitor -- \
|
|
||||||
wget -qO- http://auction-monitor:8081/health/live
|
|
||||||
```
|
|
||||||
|
|
||||||
## Cleanup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Delete all resources
|
|
||||||
kubectl delete -f k8s/deployment.yaml
|
|
||||||
|
|
||||||
# Or delete namespace (removes everything)
|
|
||||||
kubectl delete namespace auction-monitor
|
|
||||||
```
|
|
||||||
@@ -1,197 +0,0 @@
|
|||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: auction-monitor
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: PersistentVolumeClaim
|
|
||||||
metadata:
|
|
||||||
name: auction-data-pvc
|
|
||||||
namespace: auction-monitor
|
|
||||||
spec:
|
|
||||||
accessModes:
|
|
||||||
- ReadWriteOnce
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
storage: 10Gi
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: ConfigMap
|
|
||||||
metadata:
|
|
||||||
name: auction-config
|
|
||||||
namespace: auction-monitor
|
|
||||||
data:
|
|
||||||
AUCTION_DATABASE_PATH: "/data/cache.db"
|
|
||||||
AUCTION_IMAGES_PATH: "/data/images"
|
|
||||||
AUCTION_NOTIFICATION_CONFIG: "desktop"
|
|
||||||
QUARKUS_HTTP_PORT: "8081"
|
|
||||||
QUARKUS_HTTP_HOST: "0.0.0.0"
|
|
||||||
# Workflow schedules (cron expressions)
|
|
||||||
AUCTION_WORKFLOW_SCRAPER_IMPORT_CRON: "0 */30 * * * ?"
|
|
||||||
AUCTION_WORKFLOW_IMAGE_PROCESSING_CRON: "0 0 * * * ?"
|
|
||||||
AUCTION_WORKFLOW_BID_MONITORING_CRON: "0 */15 * * * ?"
|
|
||||||
AUCTION_WORKFLOW_CLOSING_ALERTS_CRON: "0 */5 * * * ?"
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: auction-secrets
|
|
||||||
namespace: auction-monitor
|
|
||||||
type: Opaque
|
|
||||||
stringData:
|
|
||||||
# Replace with your actual SMTP configuration
|
|
||||||
notification-config: "desktop"
|
|
||||||
# For email: smtp:your@gmail.com:app_password:recipient@example.com
|
|
||||||
---
|
|
||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
metadata:
|
|
||||||
name: auction-monitor
|
|
||||||
namespace: auction-monitor
|
|
||||||
labels:
|
|
||||||
app: auction-monitor
|
|
||||||
version: v1
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
selector:
|
|
||||||
matchLabels:
|
|
||||||
app: auction-monitor
|
|
||||||
template:
|
|
||||||
metadata:
|
|
||||||
labels:
|
|
||||||
app: auction-monitor
|
|
||||||
version: v1
|
|
||||||
annotations:
|
|
||||||
prometheus.io/scrape: "true"
|
|
||||||
prometheus.io/port: "8081"
|
|
||||||
prometheus.io/path: "/q/metrics"
|
|
||||||
spec:
|
|
||||||
containers:
|
|
||||||
- name: auction-monitor
|
|
||||||
image: auction-monitor:latest
|
|
||||||
imagePullPolicy: IfNotPresent
|
|
||||||
ports:
|
|
||||||
- name: http
|
|
||||||
containerPort: 8081
|
|
||||||
protocol: TCP
|
|
||||||
env:
|
|
||||||
- name: JAVA_OPTS
|
|
||||||
value: "-Xmx256m -XX:+UseParallelGC"
|
|
||||||
envFrom:
|
|
||||||
- configMapRef:
|
|
||||||
name: auction-config
|
|
||||||
- secretRef:
|
|
||||||
name: auction-secrets
|
|
||||||
volumeMounts:
|
|
||||||
- name: data
|
|
||||||
mountPath: /data
|
|
||||||
- name: models
|
|
||||||
mountPath: /app/models
|
|
||||||
readOnly: true
|
|
||||||
resources:
|
|
||||||
requests:
|
|
||||||
memory: "256Mi"
|
|
||||||
cpu: "100m"
|
|
||||||
limits:
|
|
||||||
memory: "512Mi"
|
|
||||||
cpu: "500m"
|
|
||||||
livenessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /health/live
|
|
||||||
port: 8081
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
timeoutSeconds: 3
|
|
||||||
failureThreshold: 3
|
|
||||||
readinessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /health/ready
|
|
||||||
port: 8081
|
|
||||||
initialDelaySeconds: 10
|
|
||||||
periodSeconds: 10
|
|
||||||
timeoutSeconds: 3
|
|
||||||
failureThreshold: 3
|
|
||||||
startupProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /health/started
|
|
||||||
port: 8081
|
|
||||||
initialDelaySeconds: 0
|
|
||||||
periodSeconds: 10
|
|
||||||
timeoutSeconds: 3
|
|
||||||
failureThreshold: 30
|
|
||||||
volumes:
|
|
||||||
- name: data
|
|
||||||
persistentVolumeClaim:
|
|
||||||
claimName: auction-data-pvc
|
|
||||||
- name: models
|
|
||||||
emptyDir: {} # Or mount from ConfigMap/PVC if you have YOLO models
|
|
||||||
restartPolicy: Always
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Service
|
|
||||||
metadata:
|
|
||||||
name: auction-monitor
|
|
||||||
namespace: auction-monitor
|
|
||||||
labels:
|
|
||||||
app: auction-monitor
|
|
||||||
spec:
|
|
||||||
type: ClusterIP
|
|
||||||
ports:
|
|
||||||
- port: 8081
|
|
||||||
targetPort: 8081
|
|
||||||
protocol: TCP
|
|
||||||
name: http
|
|
||||||
selector:
|
|
||||||
app: auction-monitor
|
|
||||||
---
|
|
||||||
apiVersion: networking.k8s.io/v1
|
|
||||||
kind: Ingress
|
|
||||||
metadata:
|
|
||||||
name: auction-monitor-ingress
|
|
||||||
namespace: auction-monitor
|
|
||||||
annotations:
|
|
||||||
nginx.ingress.kubernetes.io/rewrite-target: /
|
|
||||||
cert-manager.io/cluster-issuer: "letsencrypt-prod"
|
|
||||||
spec:
|
|
||||||
ingressClassName: nginx
|
|
||||||
tls:
|
|
||||||
- hosts:
|
|
||||||
- auction-monitor.yourdomain.com
|
|
||||||
secretName: auction-monitor-tls
|
|
||||||
rules:
|
|
||||||
- host: auction-monitor.yourdomain.com
|
|
||||||
http:
|
|
||||||
paths:
|
|
||||||
- path: /
|
|
||||||
pathType: Prefix
|
|
||||||
backend:
|
|
||||||
service:
|
|
||||||
name: auction-monitor
|
|
||||||
port:
|
|
||||||
number: 8081
|
|
||||||
---
|
|
||||||
apiVersion: autoscaling/v2
|
|
||||||
kind: HorizontalPodAutoscaler
|
|
||||||
metadata:
|
|
||||||
name: auction-monitor-hpa
|
|
||||||
namespace: auction-monitor
|
|
||||||
spec:
|
|
||||||
scaleTargetRef:
|
|
||||||
apiVersion: apps/v1
|
|
||||||
kind: Deployment
|
|
||||||
name: auction-monitor
|
|
||||||
minReplicas: 1
|
|
||||||
maxReplicas: 3
|
|
||||||
metrics:
|
|
||||||
- type: Resource
|
|
||||||
resource:
|
|
||||||
name: cpu
|
|
||||||
target:
|
|
||||||
type: Utilization
|
|
||||||
averageUtilization: 80
|
|
||||||
- type: Resource
|
|
||||||
resource:
|
|
||||||
name: memory
|
|
||||||
target:
|
|
||||||
type: Utilization
|
|
||||||
averageUtilization: 80
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
person
|
|
||||||
bicycle
|
|
||||||
car
|
|
||||||
motorbike
|
|
||||||
aeroplane
|
|
||||||
bus
|
|
||||||
train
|
|
||||||
truck
|
|
||||||
boat
|
|
||||||
traffic light
|
|
||||||
fire hydrant
|
|
||||||
stop sign
|
|
||||||
parking meter
|
|
||||||
bench
|
|
||||||
bird
|
|
||||||
cat
|
|
||||||
dog
|
|
||||||
horse
|
|
||||||
sheep
|
|
||||||
cow
|
|
||||||
elephant
|
|
||||||
bear
|
|
||||||
zebra
|
|
||||||
giraffe
|
|
||||||
backpack
|
|
||||||
umbrella
|
|
||||||
handbag
|
|
||||||
tie
|
|
||||||
suitcase
|
|
||||||
frisbee
|
|
||||||
skis
|
|
||||||
snowboard
|
|
||||||
sports ball
|
|
||||||
kite
|
|
||||||
baseball bat
|
|
||||||
baseball glove
|
|
||||||
skateboard
|
|
||||||
surfboard
|
|
||||||
tennis racket
|
|
||||||
bottle
|
|
||||||
wine glass
|
|
||||||
cup
|
|
||||||
fork
|
|
||||||
knife
|
|
||||||
spoon
|
|
||||||
bowl
|
|
||||||
banana
|
|
||||||
apple
|
|
||||||
sandwich
|
|
||||||
orange
|
|
||||||
broccoli
|
|
||||||
carrot
|
|
||||||
hot dog
|
|
||||||
pizza
|
|
||||||
donut
|
|
||||||
cake
|
|
||||||
chair
|
|
||||||
sofa
|
|
||||||
pottedplant
|
|
||||||
bed
|
|
||||||
diningtable
|
|
||||||
toilet
|
|
||||||
tvmonitor
|
|
||||||
laptop
|
|
||||||
mouse
|
|
||||||
remote
|
|
||||||
keyboard
|
|
||||||
cell phone
|
|
||||||
microwave
|
|
||||||
oven
|
|
||||||
toaster
|
|
||||||
sink
|
|
||||||
refrigerator
|
|
||||||
book
|
|
||||||
clock
|
|
||||||
vase
|
|
||||||
scissors
|
|
||||||
teddy bear
|
|
||||||
hair drier
|
|
||||||
toothbrush
|
|
||||||
1158
models/yolov4.cfg
1158
models/yolov4.cfg
File diff suppressed because it is too large
Load Diff
Binary file not shown.
25
pom.xml
25
pom.xml
@@ -34,6 +34,7 @@
|
|||||||
<maven-compiler-plugin-version>3.14.0</maven-compiler-plugin-version>
|
<maven-compiler-plugin-version>3.14.0</maven-compiler-plugin-version>
|
||||||
<versions-maven-plugin.version>2.19.0</versions-maven-plugin.version>
|
<versions-maven-plugin.version>2.19.0</versions-maven-plugin.version>
|
||||||
<jandex-maven-plugin-version>3.5.0</jandex-maven-plugin-version>
|
<jandex-maven-plugin-version>3.5.0</jandex-maven-plugin-version>
|
||||||
|
<jdbi.version>3.47.0</jdbi.version>
|
||||||
<maven.compiler.args>
|
<maven.compiler.args>
|
||||||
--enable-native-access=ALL-UNNAMED
|
--enable-native-access=ALL-UNNAMED
|
||||||
--add-opens java.base/sun.misc=ALL-UNNAMED
|
--add-opens java.base/sun.misc=ALL-UNNAMED
|
||||||
@@ -161,11 +162,11 @@
|
|||||||
<artifactId>slf4j-api</artifactId>
|
<artifactId>slf4j-api</artifactId>
|
||||||
<version>2.0.9</version>
|
<version>2.0.9</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
<!-- <dependency>
|
||||||
<groupId>org.slf4j</groupId>
|
<groupId>org.slf4j</groupId>
|
||||||
<artifactId>slf4j-simple</artifactId>
|
<artifactId>slf4j-simple</artifactId>
|
||||||
<version>2.0.9</version>
|
<version>2.0.9</version>
|
||||||
</dependency>
|
</dependency>-->
|
||||||
<!-- JUnit 5 for testing -->
|
<!-- JUnit 5 for testing -->
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.junit.jupiter</groupId>
|
<groupId>org.junit.jupiter</groupId>
|
||||||
@@ -175,6 +176,12 @@
|
|||||||
</dependency>
|
</dependency>
|
||||||
|
|
||||||
<!-- Mockito for mocking in tests -->
|
<!-- Mockito for mocking in tests -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>io.quarkus</groupId>
|
||||||
|
<artifactId>quarkus-junit5-mockito</artifactId>
|
||||||
|
<version>3.30.2</version>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.mockito</groupId>
|
<groupId>org.mockito</groupId>
|
||||||
<artifactId>mockito-core</artifactId>
|
<artifactId>mockito-core</artifactId>
|
||||||
@@ -190,6 +197,18 @@
|
|||||||
<scope>test</scope>
|
<scope>test</scope>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
|
||||||
|
<!-- JDBI3 - Lightweight ORM for SQL -->
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.jdbi</groupId>
|
||||||
|
<artifactId>jdbi3-core</artifactId>
|
||||||
|
<version>${jdbi.version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.jdbi</groupId>
|
||||||
|
<artifactId>jdbi3-sqlobject</artifactId>
|
||||||
|
<version>${jdbi.version}</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
<!-- AssertJ for fluent assertions (optional but recommended) -->
|
<!-- AssertJ for fluent assertions (optional but recommended) -->
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.assertj</groupId>
|
<groupId>org.assertj</groupId>
|
||||||
@@ -421,6 +440,8 @@
|
|||||||
<!-- Enable ByteBuddy experimental mode for Java 25 support -->
|
<!-- Enable ByteBuddy experimental mode for Java 25 support -->
|
||||||
<!-- Mockito requires this for Java 24+ -->
|
<!-- Mockito requires this for Java 24+ -->
|
||||||
<argLine>
|
<argLine>
|
||||||
|
--enable-native-access=ALL-UNNAMED
|
||||||
|
--add-opens java.base/sun.misc=ALL-UNNAMED
|
||||||
-Dnet.bytebuddy.experimental=true
|
-Dnet.bytebuddy.experimental=true
|
||||||
--add-opens java.base/java.lang=ALL-UNNAMED
|
--add-opens java.base/java.lang=ALL-UNNAMED
|
||||||
--add-opens java.base/java.util=ALL-UNNAMED
|
--add-opens java.base/java.util=ALL-UNNAMED
|
||||||
|
|||||||
15
scripts/smb.ps1
Normal file
15
scripts/smb.ps1
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
# PowerShell: map the remote share, copy the folder, then clean up
|
||||||
|
$remote = '\\192.168.1.159\shared-auction-data'
|
||||||
|
$local = 'C:\mnt\okcomputer\output\models'
|
||||||
|
|
||||||
|
# (1) create/verify the PSDrive (prompts for password if needed)
|
||||||
|
if (-not (Get-PSDrive -Name Z -ErrorAction SilentlyContinue)) {
|
||||||
|
$cred = Get-Credential -UserName 'tour' -Message 'SMB password for tour@192.168.1.159'
|
||||||
|
New-PSDrive -Name Z -PSProvider FileSystem -Root $remote -Credential $cred -Persist | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
# (2) copy the local folder into the share
|
||||||
|
Copy-Item -Path $local -Destination 'Z:\' -Recurse -Force
|
||||||
|
|
||||||
|
# (3) optional cleanup
|
||||||
|
Remove-PSDrive -Name Z -Force
|
||||||
@@ -7,13 +7,13 @@ import java.time.LocalDateTime;
|
|||||||
* Data typically populated by the external scraper process
|
* Data typically populated by the external scraper process
|
||||||
*/
|
*/
|
||||||
public record AuctionInfo(
|
public record AuctionInfo(
|
||||||
long auctionId, // Unique auction ID (from URL)
|
long auctionId, // Unique auction ID (from URL)
|
||||||
String title, // Auction title
|
String title, // Auction title
|
||||||
String location, // Location (e.g., "Amsterdam, NL")
|
String location, // Location (e.g., "Amsterdam, NL")
|
||||||
String city, // City name
|
String city, // City name
|
||||||
String country, // Country code (e.g., "NL")
|
String country, // Country code (e.g., "NL")
|
||||||
String url, // Full auction URL
|
String url, // Full auction URL
|
||||||
String typePrefix, // Auction type (A1 or A7)
|
String typePrefix, // Auction type (A1 or A7)
|
||||||
int lotCount, // Number of lots/kavels
|
int lotCount, // Number of lots/kavels
|
||||||
LocalDateTime firstLotClosingTime // Closing time if available
|
LocalDateTime firstLotClosingTime // Closing time if available
|
||||||
) { }
|
) { }
|
||||||
|
|||||||
@@ -11,93 +11,72 @@ import org.eclipse.microprofile.health.Startup;
|
|||||||
import java.nio.file.Files;
|
import java.nio.file.Files;
|
||||||
import java.nio.file.Paths;
|
import java.nio.file.Paths;
|
||||||
|
|
||||||
/**
|
|
||||||
* Health checks for Auction Monitor.
|
|
||||||
* Provides liveness and readiness probes for Kubernetes/Docker deployment.
|
|
||||||
*/
|
|
||||||
@ApplicationScoped
|
@ApplicationScoped
|
||||||
public class AuctionMonitorHealthCheck {
|
public class AuctionMonitorHealthCheck {
|
||||||
|
|
||||||
@Inject
|
@Liveness
|
||||||
DatabaseService db;
|
public static class LivenessCheck
|
||||||
|
implements HealthCheck {
|
||||||
/**
|
|
||||||
* Liveness probe - checks if application is alive
|
@Override public HealthCheckResponse call() {
|
||||||
* GET /health/live
|
return HealthCheckResponse.up("Auction Monitor is alive");
|
||||||
*/
|
}
|
||||||
@Liveness
|
}
|
||||||
public static class LivenessCheck implements HealthCheck {
|
|
||||||
@Override
|
@Readiness
|
||||||
public HealthCheckResponse call() {
|
@ApplicationScoped
|
||||||
return HealthCheckResponse.up("Auction Monitor is alive");
|
public static class ReadinessCheck
|
||||||
}
|
implements HealthCheck {
|
||||||
}
|
|
||||||
|
@Inject DatabaseService db;
|
||||||
/**
|
|
||||||
* Readiness probe - checks if application is ready to serve requests
|
@Override
|
||||||
* GET /health/ready
|
public HealthCheckResponse call() {
|
||||||
*/
|
try {
|
||||||
@Readiness
|
var auctions = db.getAllAuctions();
|
||||||
@ApplicationScoped
|
var dbPath = Paths.get("C:\\mnt\\okcomputer\\output\\cache.db");
|
||||||
public static class ReadinessCheck implements HealthCheck {
|
if (!Files.exists(dbPath.getParent())) {
|
||||||
|
return HealthCheckResponse.down("Database directory does not exist");
|
||||||
@Inject
|
|
||||||
DatabaseService db;
|
|
||||||
|
|
||||||
@Override
|
|
||||||
public HealthCheckResponse call() {
|
|
||||||
try {
|
|
||||||
// Check database connection
|
|
||||||
var auctions = db.getAllAuctions();
|
|
||||||
|
|
||||||
// Check database path exists
|
|
||||||
var dbPath = Paths.get("C:\\mnt\\okcomputer\\output\\cache.db");
|
|
||||||
if (!Files.exists(dbPath.getParent())) {
|
|
||||||
return HealthCheckResponse.down("Database directory does not exist");
|
|
||||||
}
|
|
||||||
|
|
||||||
return HealthCheckResponse.named("database")
|
|
||||||
.up()
|
|
||||||
.withData("auctions", auctions.size())
|
|
||||||
.build();
|
|
||||||
|
|
||||||
} catch (Exception e) {
|
|
||||||
return HealthCheckResponse.named("database")
|
|
||||||
.down()
|
|
||||||
.withData("error", e.getMessage())
|
|
||||||
.build();
|
|
||||||
}
|
}
|
||||||
}
|
|
||||||
}
|
return HealthCheckResponse.named("database")
|
||||||
|
.up()
|
||||||
/**
|
.withData("auctions", auctions.size())
|
||||||
* Startup probe - checks if application has started correctly
|
.build();
|
||||||
* GET /health/started
|
|
||||||
*/
|
} catch (Exception e) {
|
||||||
@Startup
|
return HealthCheckResponse.named("database")
|
||||||
@ApplicationScoped
|
.down()
|
||||||
public static class StartupCheck implements HealthCheck {
|
.withData("error", e.getMessage())
|
||||||
|
.build();
|
||||||
@Inject
|
}
|
||||||
DatabaseService db;
|
}
|
||||||
|
}
|
||||||
@Override
|
|
||||||
public HealthCheckResponse call() {
|
@Startup
|
||||||
try {
|
@ApplicationScoped
|
||||||
// Verify database schema
|
public static class StartupCheck
|
||||||
db.ensureSchema();
|
implements HealthCheck {
|
||||||
|
|
||||||
return HealthCheckResponse.named("startup")
|
@Inject DatabaseService db;
|
||||||
.up()
|
|
||||||
.withData("message", "Database schema initialized")
|
@Override
|
||||||
.build();
|
public HealthCheckResponse call() {
|
||||||
|
try {
|
||||||
} catch (Exception e) {
|
// Verify database schema
|
||||||
return HealthCheckResponse.named("startup")
|
db.ensureSchema();
|
||||||
.down()
|
|
||||||
.withData("error", e.getMessage())
|
return HealthCheckResponse.named("startup")
|
||||||
.build();
|
.up()
|
||||||
}
|
.withData("message", "Database schema initialized")
|
||||||
}
|
.build();
|
||||||
}
|
|
||||||
|
} catch (Exception e) {
|
||||||
|
return HealthCheckResponse.named("startup")
|
||||||
|
.down()
|
||||||
|
.withData("error", e.getMessage())
|
||||||
|
.build();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import jakarta.annotation.PostConstruct;
|
|||||||
import jakarta.enterprise.context.ApplicationScoped;
|
import jakarta.enterprise.context.ApplicationScoped;
|
||||||
import jakarta.enterprise.inject.Produces;
|
import jakarta.enterprise.inject.Produces;
|
||||||
import jakarta.inject.Singleton;
|
import jakarta.inject.Singleton;
|
||||||
|
import nu.pattern.OpenCV;
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
||||||
import org.jboss.logging.Logger;
|
import org.jboss.logging.Logger;
|
||||||
import org.opencv.core.Core;
|
import org.opencv.core.Core;
|
||||||
@@ -19,58 +20,42 @@ import java.sql.SQLException;
|
|||||||
@Startup
|
@Startup
|
||||||
@ApplicationScoped
|
@ApplicationScoped
|
||||||
public class AuctionMonitorProducer {
|
public class AuctionMonitorProducer {
|
||||||
|
|
||||||
private static final Logger LOG = Logger.getLogger(AuctionMonitorProducer.class);
|
private static final Logger LOG = Logger.getLogger(AuctionMonitorProducer.class);
|
||||||
|
|
||||||
@PostConstruct
|
@PostConstruct void init() {
|
||||||
void init() {
|
try {
|
||||||
// Load OpenCV native library at startup
|
OpenCV.loadLocally();
|
||||||
try {
|
LOG.info("✓ OpenCV loaded successfully");
|
||||||
nu.pattern.OpenCV.loadLocally();
|
} catch (Exception e) {
|
||||||
LOG.info("✓ OpenCV loaded successfully");
|
LOG.warn("⚠️ OpenCV not available - image detection will be disabled: " + e.getMessage());
|
||||||
} catch (Exception e) {
|
}
|
||||||
LOG.warn("⚠️ OpenCV not available - image detection will be disabled: " + e.getMessage());
|
}
|
||||||
}
|
|
||||||
}
|
@Produces @Singleton public DatabaseService produceDatabaseService(
|
||||||
|
@ConfigProperty(name = "auction.database.path") String dbPath) throws SQLException {
|
||||||
@Produces
|
var db = new DatabaseService(dbPath);
|
||||||
@Singleton
|
db.ensureSchema();
|
||||||
public DatabaseService produceDatabaseService(
|
return db;
|
||||||
@ConfigProperty(name = "auction.database.path") String dbPath) throws SQLException {
|
}
|
||||||
|
|
||||||
LOG.infof("Initializing DatabaseService with path: %s", dbPath);
|
@Produces @Singleton public NotificationService produceNotificationService(
|
||||||
var db = new DatabaseService(dbPath);
|
@ConfigProperty(name = "auction.notification.config") String config) {
|
||||||
db.ensureSchema();
|
|
||||||
return db;
|
return new NotificationService(config);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Produces
|
@Produces @Singleton public ObjectDetectionService produceObjectDetectionService(
|
||||||
@Singleton
|
@ConfigProperty(name = "auction.yolo.config") String cfgPath,
|
||||||
public NotificationService produceNotificationService(
|
@ConfigProperty(name = "auction.yolo.weights") String weightsPath,
|
||||||
@ConfigProperty(name = "auction.notification.config") String config) {
|
@ConfigProperty(name = "auction.yolo.classes") String classesPath) throws IOException {
|
||||||
|
|
||||||
LOG.infof("Initializing NotificationService with config: %s", config);
|
return new ObjectDetectionService(cfgPath, weightsPath, classesPath);
|
||||||
return new NotificationService(config);
|
}
|
||||||
}
|
|
||||||
|
@Produces @Singleton public ImageProcessingService produceImageProcessingService(
|
||||||
@Produces
|
DatabaseService db,
|
||||||
@Singleton
|
ObjectDetectionService detector) {
|
||||||
public ObjectDetectionService produceObjectDetectionService(
|
return new ImageProcessingService(db, detector);
|
||||||
@ConfigProperty(name = "auction.yolo.config") String cfgPath,
|
}
|
||||||
@ConfigProperty(name = "auction.yolo.weights") String weightsPath,
|
|
||||||
@ConfigProperty(name = "auction.yolo.classes") String classesPath) throws IOException {
|
|
||||||
|
|
||||||
LOG.infof("Initializing ObjectDetectionService");
|
|
||||||
return new ObjectDetectionService(cfgPath, weightsPath, classesPath);
|
|
||||||
}
|
|
||||||
|
|
||||||
@Produces
|
|
||||||
@Singleton
|
|
||||||
public ImageProcessingService produceImageProcessingService(
|
|
||||||
DatabaseService db,
|
|
||||||
ObjectDetectionService detector) {
|
|
||||||
|
|
||||||
LOG.infof("Initializing ImageProcessingService");
|
|
||||||
return new ImageProcessingService(db, detector);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -30,21 +30,12 @@ public class AuctionMonitorResource {
|
|||||||
|
|
||||||
private static final Logger LOG = Logger.getLogger(AuctionMonitorResource.class);
|
private static final Logger LOG = Logger.getLogger(AuctionMonitorResource.class);
|
||||||
|
|
||||||
@Inject
|
@Inject DatabaseService db;
|
||||||
DatabaseService db;
|
@Inject QuarkusWorkflowScheduler scheduler;
|
||||||
|
@Inject NotificationService notifier;
|
||||||
|
@Inject RateLimitedHttpClient httpClient;
|
||||||
|
@Inject LotEnrichmentService enrichmentService;
|
||||||
|
|
||||||
@Inject
|
|
||||||
QuarkusWorkflowScheduler scheduler;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
NotificationService notifier;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
RateLimitedHttpClient httpClient;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
LotEnrichmentService enrichmentService;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/status
|
* GET /api/monitor/status
|
||||||
* Returns current monitoring status
|
* Returns current monitoring status
|
||||||
@@ -99,33 +90,33 @@ public class AuctionMonitorResource {
|
|||||||
stats.put("totalImages", db.getImageCount());
|
stats.put("totalImages", db.getImageCount());
|
||||||
|
|
||||||
// Lot statistics
|
// Lot statistics
|
||||||
var activeLots = 0;
|
var activeLots = 0;
|
||||||
var lotsWithBids = 0;
|
var lotsWithBids = 0;
|
||||||
double totalBids = 0;
|
double totalBids = 0;
|
||||||
var hotLots = 0;
|
var hotLots = 0;
|
||||||
var sleeperLots = 0;
|
var sleeperLots = 0;
|
||||||
var bargainLots = 0;
|
var bargainLots = 0;
|
||||||
var lotsClosing1h = 0;
|
var lotsClosing1h = 0;
|
||||||
var lotsClosing6h = 0;
|
var lotsClosing6h = 0;
|
||||||
double totalBidVelocity = 0;
|
double totalBidVelocity = 0;
|
||||||
int velocityCount = 0;
|
int velocityCount = 0;
|
||||||
|
|
||||||
for (var lot : lots) {
|
for (var lot : lots) {
|
||||||
long minutesLeft = lot.closingTime() != null ? lot.minutesUntilClose() : Long.MAX_VALUE;
|
long minutesLeft = lot.closingTime() != null ? lot.minutesUntilClose() : Long.MAX_VALUE;
|
||||||
|
|
||||||
if (lot.closingTime() != null && minutesLeft > 0) {
|
if (lot.closingTime() != null && minutesLeft > 0) {
|
||||||
activeLots++;
|
activeLots++;
|
||||||
|
|
||||||
// Time-based counts
|
// Time-based counts
|
||||||
if (minutesLeft < 60) lotsClosing1h++;
|
if (minutesLeft < 60) lotsClosing1h++;
|
||||||
if (minutesLeft < 360) lotsClosing6h++;
|
if (minutesLeft < 360) lotsClosing6h++;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (lot.currentBid() > 0) {
|
if (lot.currentBid() > 0) {
|
||||||
lotsWithBids++;
|
lotsWithBids++;
|
||||||
totalBids += lot.currentBid();
|
totalBids += lot.currentBid();
|
||||||
}
|
}
|
||||||
|
|
||||||
// Intelligence metrics (require GraphQL enrichment)
|
// Intelligence metrics (require GraphQL enrichment)
|
||||||
if (lot.followersCount() != null && lot.followersCount() > 20) {
|
if (lot.followersCount() != null && lot.followersCount() > 20) {
|
||||||
hotLots++;
|
hotLots++;
|
||||||
@@ -136,22 +127,22 @@ public class AuctionMonitorResource {
|
|||||||
if (lot.isBelowEstimate()) {
|
if (lot.isBelowEstimate()) {
|
||||||
bargainLots++;
|
bargainLots++;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Bid velocity
|
// Bid velocity
|
||||||
if (lot.bidVelocity() != null && lot.bidVelocity() > 0) {
|
if (lot.bidVelocity() != null && lot.bidVelocity() > 0) {
|
||||||
totalBidVelocity += lot.bidVelocity();
|
totalBidVelocity += lot.bidVelocity();
|
||||||
velocityCount++;
|
velocityCount++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Calculate bids per hour (average velocity across all lots with velocity data)
|
// Calculate bids per hour (average velocity across all lots with velocity data)
|
||||||
double bidsPerHour = velocityCount > 0 ? totalBidVelocity / velocityCount : 0;
|
double bidsPerHour = velocityCount > 0 ? totalBidVelocity / velocityCount : 0;
|
||||||
|
|
||||||
stats.put("activeLots", activeLots);
|
stats.put("activeLots", activeLots);
|
||||||
stats.put("lotsWithBids", lotsWithBids);
|
stats.put("lotsWithBids", lotsWithBids);
|
||||||
stats.put("totalBidValue", String.format("€%.2f", totalBids));
|
stats.put("totalBidValue", String.format("€%.2f", totalBids));
|
||||||
stats.put("averageBid", lotsWithBids > 0 ? String.format("€%.2f", totalBids / lotsWithBids) : "€0.00");
|
stats.put("averageBid", lotsWithBids > 0 ? String.format("€%.2f", totalBids / lotsWithBids) : "€0.00");
|
||||||
|
|
||||||
// Bidding intelligence
|
// Bidding intelligence
|
||||||
stats.put("bidsPerHour", String.format("%.1f", bidsPerHour));
|
stats.put("bidsPerHour", String.format("%.1f", bidsPerHour));
|
||||||
stats.put("hotLots", hotLots);
|
stats.put("hotLots", hotLots);
|
||||||
@@ -159,11 +150,11 @@ public class AuctionMonitorResource {
|
|||||||
stats.put("bargainLots", bargainLots);
|
stats.put("bargainLots", bargainLots);
|
||||||
stats.put("lotsClosing1h", lotsClosing1h);
|
stats.put("lotsClosing1h", lotsClosing1h);
|
||||||
stats.put("lotsClosing6h", lotsClosing6h);
|
stats.put("lotsClosing6h", lotsClosing6h);
|
||||||
|
|
||||||
// Conversion rate
|
// Conversion rate
|
||||||
double conversionRate = activeLots > 0 ? (lotsWithBids * 100.0 / activeLots) : 0;
|
double conversionRate = activeLots > 0 ? (lotsWithBids * 100.0 / activeLots) : 0;
|
||||||
stats.put("conversionRate", String.format("%.1f%%", conversionRate));
|
stats.put("conversionRate", String.format("%.1f%%", conversionRate));
|
||||||
|
|
||||||
return Response.ok(stats).build();
|
return Response.ok(stats).build();
|
||||||
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
@@ -184,12 +175,12 @@ public class AuctionMonitorResource {
|
|||||||
try {
|
try {
|
||||||
var lots = db.getAllLots();
|
var lots = db.getAllLots();
|
||||||
var closingSoon = lots.stream()
|
var closingSoon = lots.stream()
|
||||||
.filter(lot -> lot.closingTime() != null)
|
.filter(lot -> lot.closingTime() != null)
|
||||||
.filter(lot -> lot.minutesUntilClose() > 0 && lot.minutesUntilClose() <= hours * 60)
|
.filter(lot -> lot.minutesUntilClose() > 0 && lot.minutesUntilClose() <= hours * 60)
|
||||||
.sorted((a, b) -> Long.compare(a.minutesUntilClose(), b.minutesUntilClose()))
|
.sorted((a, b) -> Long.compare(a.minutesUntilClose(), b.minutesUntilClose()))
|
||||||
.limit(100)
|
.limit(100)
|
||||||
.toList();
|
.toList();
|
||||||
|
|
||||||
return Response.ok(closingSoon).build();
|
return Response.ok(closingSoon).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get closing soon lots", e);
|
LOG.error("Failed to get closing soon lots", e);
|
||||||
@@ -198,7 +189,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/lots/{lotId}/bid-history
|
* GET /api/monitor/lots/{lotId}/bid-history
|
||||||
* Returns bid history for a specific lot
|
* Returns bid history for a specific lot
|
||||||
@@ -216,7 +207,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* POST /api/monitor/trigger/scraper-import
|
* POST /api/monitor/trigger/scraper-import
|
||||||
* Manually trigger scraper import workflow
|
* Manually trigger scraper import workflow
|
||||||
@@ -288,7 +279,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* POST /api/monitor/trigger/graphql-enrichment
|
* POST /api/monitor/trigger/graphql-enrichment
|
||||||
* Manually trigger GraphQL enrichment for all lots or lots closing soon
|
* Manually trigger GraphQL enrichment for all lots or lots closing soon
|
||||||
@@ -301,15 +292,15 @@ public class AuctionMonitorResource {
|
|||||||
if (hours > 0) {
|
if (hours > 0) {
|
||||||
enriched = enrichmentService.enrichClosingSoonLots(hours);
|
enriched = enrichmentService.enrichClosingSoonLots(hours);
|
||||||
return Response.ok(Map.of(
|
return Response.ok(Map.of(
|
||||||
"message", "GraphQL enrichment triggered for lots closing within " + hours + " hours",
|
"message", "GraphQL enrichment triggered for lots closing within " + hours + " hours",
|
||||||
"enrichedCount", enriched
|
"enrichedCount", enriched
|
||||||
)).build();
|
)).build();
|
||||||
} else {
|
} else {
|
||||||
enriched = enrichmentService.enrichAllActiveLots();
|
enriched = enrichmentService.enrichAllActiveLots();
|
||||||
return Response.ok(Map.of(
|
return Response.ok(Map.of(
|
||||||
"message", "GraphQL enrichment triggered for all lots",
|
"message", "GraphQL enrichment triggered for all lots",
|
||||||
"enrichedCount", enriched
|
"enrichedCount", enriched
|
||||||
)).build();
|
)).build();
|
||||||
}
|
}
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to trigger GraphQL enrichment", e);
|
LOG.error("Failed to trigger GraphQL enrichment", e);
|
||||||
@@ -318,7 +309,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/auctions
|
* GET /api/monitor/auctions
|
||||||
* Returns list of all auctions
|
* Returns list of all auctions
|
||||||
@@ -375,7 +366,7 @@ public class AuctionMonitorResource {
|
|||||||
})
|
})
|
||||||
.sorted((a, b) -> Long.compare(a.minutesUntilClose(), b.minutesUntilClose()))
|
.sorted((a, b) -> Long.compare(a.minutesUntilClose(), b.minutesUntilClose()))
|
||||||
.toList();
|
.toList();
|
||||||
|
|
||||||
return Response.ok(closingSoon).build();
|
return Response.ok(closingSoon).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get closing lots", e);
|
LOG.error("Failed to get closing lots", e);
|
||||||
@@ -530,7 +521,7 @@ public class AuctionMonitorResource {
|
|||||||
public Response getCategoryDistribution() {
|
public Response getCategoryDistribution() {
|
||||||
try {
|
try {
|
||||||
var lots = db.getAllLots();
|
var lots = db.getAllLots();
|
||||||
|
|
||||||
// Category distribution
|
// Category distribution
|
||||||
Map<String, Long> distribution = lots.stream()
|
Map<String, Long> distribution = lots.stream()
|
||||||
.filter(l -> l.category() != null && !l.category().isEmpty())
|
.filter(l -> l.category() != null && !l.category().isEmpty())
|
||||||
@@ -538,34 +529,34 @@ public class AuctionMonitorResource {
|
|||||||
l -> l.category().length() > 20 ? l.category().substring(0, 20) + "..." : l.category(),
|
l -> l.category().length() > 20 ? l.category().substring(0, 20) + "..." : l.category(),
|
||||||
Collectors.counting()
|
Collectors.counting()
|
||||||
));
|
));
|
||||||
|
|
||||||
// Find top category by count
|
// Find top category by count
|
||||||
var topCategory = distribution.entrySet().stream()
|
var topCategory = distribution.entrySet().stream()
|
||||||
.max(Map.Entry.comparingByValue())
|
.max(Map.Entry.comparingByValue())
|
||||||
.map(Map.Entry::getKey)
|
.map(Map.Entry::getKey)
|
||||||
.orElse("N/A");
|
.orElse("N/A");
|
||||||
|
|
||||||
// Calculate average bids per category
|
// Calculate average bids per category
|
||||||
Map<String, Double> avgBidsByCategory = lots.stream()
|
Map<String, Double> avgBidsByCategory = lots.stream()
|
||||||
.filter(l -> l.category() != null && !l.category().isEmpty() && l.currentBid() > 0)
|
.filter(l -> l.category() != null && !l.category().isEmpty() && l.currentBid() > 0)
|
||||||
.collect(Collectors.groupingBy(
|
.collect(Collectors.groupingBy(
|
||||||
l -> l.category().length() > 20 ? l.category().substring(0, 20) + "..." : l.category(),
|
l -> l.category().length() > 20 ? l.category().substring(0, 20) + "..." : l.category(),
|
||||||
Collectors.averagingDouble(Lot::currentBid)
|
Collectors.averagingDouble(Lot::currentBid)
|
||||||
));
|
));
|
||||||
|
|
||||||
double overallAvgBid = lots.stream()
|
double overallAvgBid = lots.stream()
|
||||||
.filter(l -> l.currentBid() > 0)
|
.filter(l -> l.currentBid() > 0)
|
||||||
.mapToDouble(Lot::currentBid)
|
.mapToDouble(Lot::currentBid)
|
||||||
.average()
|
.average()
|
||||||
.orElse(0.0);
|
.orElse(0.0);
|
||||||
|
|
||||||
Map<String, Object> response = new HashMap<>();
|
Map<String, Object> response = new HashMap<>();
|
||||||
response.put("distribution", distribution);
|
response.put("distribution", distribution);
|
||||||
response.put("topCategory", topCategory);
|
response.put("topCategory", topCategory);
|
||||||
response.put("categoryCount", distribution.size());
|
response.put("categoryCount", distribution.size());
|
||||||
response.put("averageBidOverall", String.format("€%.2f", overallAvgBid));
|
response.put("averageBidOverall", String.format("€%.2f", overallAvgBid));
|
||||||
response.put("avgBidsByCategory", avgBidsByCategory);
|
response.put("avgBidsByCategory", avgBidsByCategory);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get category distribution", e);
|
LOG.error("Failed to get category distribution", e);
|
||||||
@@ -663,7 +654,7 @@ public class AuctionMonitorResource {
|
|||||||
.max(Map.Entry.comparingByValue())
|
.max(Map.Entry.comparingByValue())
|
||||||
.map(Map.Entry::getKey)
|
.map(Map.Entry::getKey)
|
||||||
.orElse("N/A");
|
.orElse("N/A");
|
||||||
|
|
||||||
if (!"N/A".equals(topCountry)) {
|
if (!"N/A".equals(topCountry)) {
|
||||||
insights.add(Map.of(
|
insights.add(Map.of(
|
||||||
"icon", "fa-globe",
|
"icon", "fa-globe",
|
||||||
@@ -671,7 +662,7 @@ public class AuctionMonitorResource {
|
|||||||
"description", "Top performing country"
|
"description", "Top performing country"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add sleeper lots insight
|
// Add sleeper lots insight
|
||||||
long sleeperCount = lots.stream().filter(Lot::isSleeperLot).count();
|
long sleeperCount = lots.stream().filter(Lot::isSleeperLot).count();
|
||||||
if (sleeperCount > 0) {
|
if (sleeperCount > 0) {
|
||||||
@@ -681,7 +672,7 @@ public class AuctionMonitorResource {
|
|||||||
"description", "High interest, low bids - opportunity?"
|
"description", "High interest, low bids - opportunity?"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add bargain insight
|
// Add bargain insight
|
||||||
long bargainCount = lots.stream().filter(Lot::isBelowEstimate).count();
|
long bargainCount = lots.stream().filter(Lot::isBelowEstimate).count();
|
||||||
if (bargainCount > 5) {
|
if (bargainCount > 5) {
|
||||||
@@ -691,7 +682,7 @@ public class AuctionMonitorResource {
|
|||||||
"description", "Priced below auction house estimates"
|
"description", "Priced below auction house estimates"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add watch/followers insight
|
// Add watch/followers insight
|
||||||
long highWatchCount = lots.stream()
|
long highWatchCount = lots.stream()
|
||||||
.filter(l -> l.followersCount() != null && l.followersCount() > 20)
|
.filter(l -> l.followersCount() != null && l.followersCount() > 20)
|
||||||
@@ -703,7 +694,7 @@ public class AuctionMonitorResource {
|
|||||||
"description", "High follower count, strong competition"
|
"description", "High follower count, strong competition"
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
return Response.ok(insights).build();
|
return Response.ok(insights).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get insights", e);
|
LOG.error("Failed to get insights", e);
|
||||||
@@ -725,12 +716,12 @@ public class AuctionMonitorResource {
|
|||||||
var sleepers = allLots.stream()
|
var sleepers = allLots.stream()
|
||||||
.filter(Lot::isSleeperLot)
|
.filter(Lot::isSleeperLot)
|
||||||
.toList();
|
.toList();
|
||||||
|
|
||||||
Map<String, Object> response = Map.of(
|
Map<String, Object> response = Map.of(
|
||||||
"count", sleepers.size(),
|
"count", sleepers.size(),
|
||||||
"lots", sleepers
|
"lots", sleepers
|
||||||
);
|
);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get sleeper lots", e);
|
LOG.error("Failed to get sleeper lots", e);
|
||||||
@@ -739,7 +730,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/intelligence/bargains
|
* GET /api/monitor/intelligence/bargains
|
||||||
* Returns lots priced below auction house estimates
|
* Returns lots priced below auction house estimates
|
||||||
@@ -759,12 +750,12 @@ public class AuctionMonitorResource {
|
|||||||
return ratioA.compareTo(ratioB);
|
return ratioA.compareTo(ratioB);
|
||||||
})
|
})
|
||||||
.toList();
|
.toList();
|
||||||
|
|
||||||
Map<String, Object> response = Map.of(
|
Map<String, Object> response = Map.of(
|
||||||
"count", bargains.size(),
|
"count", bargains.size(),
|
||||||
"lots", bargains
|
"lots", bargains
|
||||||
);
|
);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get bargains", e);
|
LOG.error("Failed to get bargains", e);
|
||||||
@@ -773,7 +764,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/intelligence/popular
|
* GET /api/monitor/intelligence/popular
|
||||||
* Returns lots by popularity level
|
* Returns lots by popularity level
|
||||||
@@ -791,13 +782,13 @@ public class AuctionMonitorResource {
|
|||||||
return followersB.compareTo(followersA);
|
return followersB.compareTo(followersA);
|
||||||
})
|
})
|
||||||
.toList();
|
.toList();
|
||||||
|
|
||||||
Map<String, Object> response = Map.of(
|
Map<String, Object> response = Map.of(
|
||||||
"count", popular.size(),
|
"count", popular.size(),
|
||||||
"level", level,
|
"level", level,
|
||||||
"lots", popular
|
"lots", popular
|
||||||
);
|
);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get popular lots", e);
|
LOG.error("Failed to get popular lots", e);
|
||||||
@@ -806,7 +797,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/intelligence/price-analysis
|
* GET /api/monitor/intelligence/price-analysis
|
||||||
* Returns price vs estimate analysis
|
* Returns price vs estimate analysis
|
||||||
@@ -816,28 +807,28 @@ public class AuctionMonitorResource {
|
|||||||
public Response getPriceAnalysis() {
|
public Response getPriceAnalysis() {
|
||||||
try {
|
try {
|
||||||
var allLots = db.getAllLots();
|
var allLots = db.getAllLots();
|
||||||
|
|
||||||
long belowEstimate = allLots.stream().filter(Lot::isBelowEstimate).count();
|
long belowEstimate = allLots.stream().filter(Lot::isBelowEstimate).count();
|
||||||
long aboveEstimate = allLots.stream().filter(Lot::isAboveEstimate).count();
|
long aboveEstimate = allLots.stream().filter(Lot::isAboveEstimate).count();
|
||||||
long withEstimates = allLots.stream()
|
long withEstimates = allLots.stream()
|
||||||
.filter(lot -> lot.estimatedMin() != null && lot.estimatedMax() != null)
|
.filter(lot -> lot.estimatedMin() != null && lot.estimatedMax() != null)
|
||||||
.count();
|
.count();
|
||||||
|
|
||||||
double avgPriceVsEstimate = allLots.stream()
|
double avgPriceVsEstimate = allLots.stream()
|
||||||
.map(Lot::getPriceVsEstimateRatio)
|
.map(Lot::getPriceVsEstimateRatio)
|
||||||
.filter(ratio -> ratio != null)
|
.filter(ratio -> ratio != null)
|
||||||
.mapToDouble(Double::doubleValue)
|
.mapToDouble(Double::doubleValue)
|
||||||
.average()
|
.average()
|
||||||
.orElse(0.0);
|
.orElse(0.0);
|
||||||
|
|
||||||
Map<String, Object> response = Map.of(
|
Map<String, Object> response = Map.of(
|
||||||
"totalLotsWithEstimates", withEstimates,
|
"totalLotsWithEstimates", withEstimates,
|
||||||
"belowEstimate", belowEstimate,
|
"belowEstimate", belowEstimate,
|
||||||
"aboveEstimate", aboveEstimate,
|
"aboveEstimate", aboveEstimate,
|
||||||
"averagePriceVsEstimatePercent", Math.round(avgPriceVsEstimate),
|
"averagePriceVsEstimatePercent", Math.round(avgPriceVsEstimate),
|
||||||
"bargainOpportunities", belowEstimate
|
"bargainOpportunities", belowEstimate
|
||||||
);
|
);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get price analysis", e);
|
LOG.error("Failed to get price analysis", e);
|
||||||
@@ -846,7 +837,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/lots/{lotId}/intelligence
|
* GET /api/monitor/lots/{lotId}/intelligence
|
||||||
* Returns detailed intelligence for a specific lot
|
* Returns detailed intelligence for a specific lot
|
||||||
@@ -859,13 +850,13 @@ public class AuctionMonitorResource {
|
|||||||
.filter(l -> l.lotId() == lotId)
|
.filter(l -> l.lotId() == lotId)
|
||||||
.findFirst()
|
.findFirst()
|
||||||
.orElse(null);
|
.orElse(null);
|
||||||
|
|
||||||
if (lot == null) {
|
if (lot == null) {
|
||||||
return Response.status(Response.Status.NOT_FOUND)
|
return Response.status(Response.Status.NOT_FOUND)
|
||||||
.entity(Map.of("error", "Lot not found"))
|
.entity(Map.of("error", "Lot not found"))
|
||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
|
|
||||||
Map<String, Object> intelligence = new HashMap<>();
|
Map<String, Object> intelligence = new HashMap<>();
|
||||||
intelligence.put("lotId", lot.lotId());
|
intelligence.put("lotId", lot.lotId());
|
||||||
intelligence.put("followersCount", lot.followersCount());
|
intelligence.put("followersCount", lot.followersCount());
|
||||||
@@ -882,7 +873,7 @@ public class AuctionMonitorResource {
|
|||||||
intelligence.put("condition", lot.condition());
|
intelligence.put("condition", lot.condition());
|
||||||
intelligence.put("vat", lot.vat());
|
intelligence.put("vat", lot.vat());
|
||||||
intelligence.put("buyerPremium", lot.buyerPremiumPercentage());
|
intelligence.put("buyerPremium", lot.buyerPremiumPercentage());
|
||||||
|
|
||||||
return Response.ok(intelligence).build();
|
return Response.ok(intelligence).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get lot intelligence", e);
|
LOG.error("Failed to get lot intelligence", e);
|
||||||
@@ -891,7 +882,7 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* GET /api/monitor/charts/watch-distribution
|
* GET /api/monitor/charts/watch-distribution
|
||||||
* Returns follower/watch count distribution
|
* Returns follower/watch count distribution
|
||||||
@@ -901,14 +892,14 @@ public class AuctionMonitorResource {
|
|||||||
public Response getWatchDistribution() {
|
public Response getWatchDistribution() {
|
||||||
try {
|
try {
|
||||||
var lots = db.getAllLots();
|
var lots = db.getAllLots();
|
||||||
|
|
||||||
Map<String, Long> distribution = new HashMap<>();
|
Map<String, Long> distribution = new HashMap<>();
|
||||||
distribution.put("0 watchers", lots.stream().filter(l -> l.followersCount() == null || l.followersCount() == 0).count());
|
distribution.put("0 watchers", lots.stream().filter(l -> l.followersCount() == null || l.followersCount() == 0).count());
|
||||||
distribution.put("1-5 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 1 && l.followersCount() <= 5).count());
|
distribution.put("1-5 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 1 && l.followersCount() <= 5).count());
|
||||||
distribution.put("6-20 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 6 && l.followersCount() <= 20).count());
|
distribution.put("6-20 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 6 && l.followersCount() <= 20).count());
|
||||||
distribution.put("21-50 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 21 && l.followersCount() <= 50).count());
|
distribution.put("21-50 watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() >= 21 && l.followersCount() <= 50).count());
|
||||||
distribution.put("50+ watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() > 50).count());
|
distribution.put("50+ watchers", lots.stream().filter(l -> l.followersCount() != null && l.followersCount() > 50).count());
|
||||||
|
|
||||||
return Response.ok(distribution).build();
|
return Response.ok(distribution).build();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
LOG.error("Failed to get watch distribution", e);
|
LOG.error("Failed to get watch distribution", e);
|
||||||
@@ -917,14 +908,14 @@ public class AuctionMonitorResource {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper class for trend data
|
// Helper class for trend data
|
||||||
public static class TrendHour {
|
public static class TrendHour {
|
||||||
|
|
||||||
public int hour;
|
public int hour;
|
||||||
public int lots;
|
public int lots;
|
||||||
public int bids;
|
public int bids;
|
||||||
|
|
||||||
public TrendHour(int hour, int lots, int bids) {
|
public TrendHour(int hour, int lots, int bids) {
|
||||||
this.hour = hour;
|
this.hour = hour;
|
||||||
this.lots = lots;
|
this.lots = lots;
|
||||||
|
|||||||
@@ -1,795 +1,218 @@
|
|||||||
package auctiora;
|
package auctiora;
|
||||||
|
|
||||||
import jakarta.enterprise.context.ApplicationScoped;
|
import auctiora.db.*;
|
||||||
import jakarta.inject.Inject;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
import org.jdbi.v3.core.Jdbi;
|
||||||
|
|
||||||
import java.io.Console;
|
|
||||||
import java.sql.DriverManager;
|
|
||||||
import java.sql.SQLException;
|
|
||||||
import java.time.Instant;
|
|
||||||
import java.time.LocalDateTime;
|
|
||||||
import java.util.ArrayList;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Service for persisting auctions, lots, and images into a SQLite database.
|
* Refactored database service using repository pattern and JDBI3.
|
||||||
* Data is typically populated by an external scraper process;
|
* Delegates operations to specialized repositories for better separation of concerns.
|
||||||
* this service enriches it with image processing and monitoring.
|
*
|
||||||
|
* @deprecated Legacy methods maintained for backward compatibility.
|
||||||
|
* New code should use repositories directly via dependency injection.
|
||||||
*/
|
*/
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class DatabaseService {
|
public class DatabaseService {
|
||||||
|
|
||||||
private final String url;
|
private final Jdbi jdbi;
|
||||||
|
private final LotRepository lotRepository;
|
||||||
|
private final AuctionRepository auctionRepository;
|
||||||
|
private final ImageRepository imageRepository;
|
||||||
|
|
||||||
DatabaseService(String dbPath) {
|
/**
|
||||||
// Enable WAL mode and busy timeout for concurrent access
|
* Constructor for programmatic instantiation (tests, CLI tools).
|
||||||
this.url = "jdbc:sqlite:" + dbPath + "?journal_mode=WAL&busy_timeout=10000";
|
*/
|
||||||
}
|
public DatabaseService(String dbPath) {
|
||||||
|
String url = "jdbc:sqlite:" + dbPath + "?journal_mode=WAL&busy_timeout=10000";
|
||||||
/**
|
this.jdbi = Jdbi.create(url);
|
||||||
* Creates tables if they do not already exist.
|
|
||||||
* Schema supports data from external scraper and adds image processing results.
|
|
||||||
*/
|
|
||||||
void ensureSchema() throws SQLException {
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
|
||||||
// Enable WAL mode for better concurrent access
|
|
||||||
stmt.execute("PRAGMA journal_mode=WAL");
|
|
||||||
stmt.execute("PRAGMA busy_timeout=10000");
|
|
||||||
stmt.execute("PRAGMA synchronous=NORMAL");
|
|
||||||
|
|
||||||
// Cache table (for HTTP caching)
|
// Initialize schema
|
||||||
stmt.execute("""
|
DatabaseSchema.ensureSchema(jdbi);
|
||||||
CREATE TABLE IF NOT EXISTS cache (
|
|
||||||
url TEXT PRIMARY KEY,
|
|
||||||
content BLOB,
|
|
||||||
timestamp REAL,
|
|
||||||
status_code INTEGER
|
|
||||||
)""");
|
|
||||||
|
|
||||||
// Auctions table (populated by external scraper)
|
// Create repositories
|
||||||
stmt.execute("""
|
this.lotRepository = new LotRepository(jdbi);
|
||||||
CREATE TABLE IF NOT EXISTS auctions (
|
this.auctionRepository = new AuctionRepository(jdbi);
|
||||||
auction_id TEXT PRIMARY KEY,
|
this.imageRepository = new ImageRepository(jdbi);
|
||||||
url TEXT UNIQUE,
|
}
|
||||||
title TEXT,
|
|
||||||
location TEXT,
|
|
||||||
lots_count INTEGER,
|
|
||||||
first_lot_closing_time TEXT,
|
|
||||||
scraped_at TEXT,
|
|
||||||
city TEXT,
|
|
||||||
country TEXT,
|
|
||||||
type TEXT,
|
|
||||||
lot_count INTEGER DEFAULT 0,
|
|
||||||
closing_time TEXT,
|
|
||||||
discovered_at INTEGER
|
|
||||||
)""");
|
|
||||||
|
|
||||||
// Lots table (populated by external scraper)
|
/**
|
||||||
stmt.execute("""
|
* Constructor with JDBI instance (for dependency injection).
|
||||||
CREATE TABLE IF NOT EXISTS lots (
|
*/
|
||||||
lot_id TEXT PRIMARY KEY,
|
public DatabaseService(Jdbi jdbi) {
|
||||||
auction_id TEXT,
|
this.jdbi = jdbi;
|
||||||
url TEXT UNIQUE,
|
DatabaseSchema.ensureSchema(jdbi);
|
||||||
title TEXT,
|
|
||||||
current_bid TEXT,
|
|
||||||
bid_count INTEGER,
|
|
||||||
closing_time TEXT,
|
|
||||||
viewing_time TEXT,
|
|
||||||
pickup_date TEXT,
|
|
||||||
location TEXT,
|
|
||||||
description TEXT,
|
|
||||||
category TEXT,
|
|
||||||
scraped_at TEXT,
|
|
||||||
sale_id INTEGER,
|
|
||||||
manufacturer TEXT,
|
|
||||||
type TEXT,
|
|
||||||
year INTEGER,
|
|
||||||
currency TEXT DEFAULT 'EUR',
|
|
||||||
closing_notified INTEGER DEFAULT 0,
|
|
||||||
starting_bid TEXT,
|
|
||||||
minimum_bid TEXT,
|
|
||||||
status TEXT,
|
|
||||||
brand TEXT,
|
|
||||||
model TEXT,
|
|
||||||
attributes_json TEXT,
|
|
||||||
first_bid_time TEXT,
|
|
||||||
last_bid_time TEXT,
|
|
||||||
bid_velocity REAL,
|
|
||||||
bid_increment REAL,
|
|
||||||
year_manufactured INTEGER,
|
|
||||||
condition_score REAL,
|
|
||||||
condition_description TEXT,
|
|
||||||
serial_number TEXT,
|
|
||||||
damage_description TEXT,
|
|
||||||
followers_count INTEGER DEFAULT 0,
|
|
||||||
estimated_min_price REAL,
|
|
||||||
estimated_max_price REAL,
|
|
||||||
lot_condition TEXT,
|
|
||||||
appearance TEXT,
|
|
||||||
estimated_min REAL,
|
|
||||||
estimated_max REAL,
|
|
||||||
next_bid_step_cents INTEGER,
|
|
||||||
condition TEXT,
|
|
||||||
category_path TEXT,
|
|
||||||
city_location TEXT,
|
|
||||||
country_code TEXT,
|
|
||||||
bidding_status TEXT,
|
|
||||||
packaging TEXT,
|
|
||||||
quantity INTEGER,
|
|
||||||
vat REAL,
|
|
||||||
buyer_premium_percentage REAL,
|
|
||||||
remarks TEXT,
|
|
||||||
reserve_price REAL,
|
|
||||||
reserve_met INTEGER,
|
|
||||||
view_count INTEGER,
|
|
||||||
FOREIGN KEY (auction_id) REFERENCES auctions(auction_id)
|
|
||||||
)""");
|
|
||||||
|
|
||||||
// Images table (populated by external scraper with URLs and local_path)
|
this.lotRepository = new LotRepository(jdbi);
|
||||||
stmt.execute("""
|
this.auctionRepository = new AuctionRepository(jdbi);
|
||||||
CREATE TABLE IF NOT EXISTS images (
|
this.imageRepository = new ImageRepository(jdbi);
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
}
|
||||||
lot_id TEXT,
|
|
||||||
url TEXT,
|
|
||||||
local_path TEXT,
|
|
||||||
downloaded INTEGER DEFAULT 0,
|
|
||||||
labels TEXT,
|
|
||||||
processed_at INTEGER,
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
)""");
|
|
||||||
|
|
||||||
// Bid history table
|
// ==================== LEGACY COMPATIBILITY METHODS ====================
|
||||||
stmt.execute("""
|
// These methods delegate to repositories for backward compatibility
|
||||||
CREATE TABLE IF NOT EXISTS bid_history (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
lot_id TEXT NOT NULL,
|
|
||||||
bid_amount REAL NOT NULL,
|
|
||||||
bid_time TEXT NOT NULL,
|
|
||||||
is_autobid INTEGER DEFAULT 0,
|
|
||||||
bidder_id TEXT,
|
|
||||||
bidder_number INTEGER,
|
|
||||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
|
||||||
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
|
||||||
)""");
|
|
||||||
|
|
||||||
// Indexes for performance
|
void ensureSchema() {
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_timestamp ON cache(timestamp)");
|
DatabaseSchema.ensureSchema(jdbi);
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_auctions_country ON auctions(country)");
|
}
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_lots_sale_id ON lots(sale_id)");
|
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_images_lot_id ON images(lot_id)");
|
|
||||||
stmt.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_lot_url ON images(lot_id, url)");
|
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_bid_history_lot_time ON bid_history(lot_id, bid_time)");
|
|
||||||
stmt.execute("CREATE INDEX IF NOT EXISTS idx_bid_history_bidder ON bid_history(bidder_id)");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
|
synchronized void upsertAuction(AuctionInfo auction) {
|
||||||
|
auctionRepository.upsert(auction);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
synchronized List<AuctionInfo> getAllAuctions() {
|
||||||
* Inserts or updates an auction record (typically called by external scraper)
|
return auctionRepository.getAll();
|
||||||
* Handles both auction_id conflicts and url uniqueness constraints
|
}
|
||||||
*/
|
|
||||||
synchronized void upsertAuction(AuctionInfo auction) throws SQLException {
|
|
||||||
// First try to INSERT with ON CONFLICT on auction_id
|
|
||||||
var insertSql = """
|
|
||||||
INSERT INTO auctions (auction_id, title, location, city, country, url, type, lot_count, closing_time, discovered_at)
|
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
||||||
ON CONFLICT(auction_id) DO UPDATE SET
|
|
||||||
title = excluded.title,
|
|
||||||
location = excluded.location,
|
|
||||||
city = excluded.city,
|
|
||||||
country = excluded.country,
|
|
||||||
url = excluded.url,
|
|
||||||
type = excluded.type,
|
|
||||||
lot_count = excluded.lot_count,
|
|
||||||
closing_time = excluded.closing_time
|
|
||||||
""";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url)) {
|
synchronized List<AuctionInfo> getAuctionsByCountry(String countryCode) {
|
||||||
try (var ps = conn.prepareStatement(insertSql)) {
|
return auctionRepository.getByCountry(countryCode);
|
||||||
ps.setLong(1, auction.auctionId());
|
}
|
||||||
ps.setString(2, auction.title());
|
|
||||||
ps.setString(3, auction.location());
|
|
||||||
ps.setString(4, auction.city());
|
|
||||||
ps.setString(5, auction.country());
|
|
||||||
ps.setString(6, auction.url());
|
|
||||||
ps.setString(7, auction.typePrefix());
|
|
||||||
ps.setInt(8, auction.lotCount());
|
|
||||||
ps.setString(9, auction.firstLotClosingTime() != null ? auction.firstLotClosingTime().toString() : null);
|
|
||||||
ps.setLong(10, Instant.now().getEpochSecond());
|
|
||||||
ps.executeUpdate();
|
|
||||||
} catch (SQLException e) {
|
|
||||||
// Handle both PRIMARY KEY and URL constraint failures
|
|
||||||
String errMsg = e.getMessage();
|
|
||||||
if (errMsg.contains("UNIQUE constraint failed: auctions.auction_id") ||
|
|
||||||
errMsg.contains("UNIQUE constraint failed: auctions.url") ||
|
|
||||||
errMsg.contains("PRIMARY KEY constraint failed")) {
|
|
||||||
|
|
||||||
// Try updating by URL as fallback (most reliable unique identifier)
|
synchronized void upsertLot(Lot lot) {
|
||||||
var updateByUrlSql = """
|
lotRepository.upsert(lot);
|
||||||
UPDATE auctions SET
|
}
|
||||||
auction_id = ?,
|
|
||||||
title = ?,
|
|
||||||
location = ?,
|
|
||||||
city = ?,
|
|
||||||
country = ?,
|
|
||||||
type = ?,
|
|
||||||
lot_count = ?,
|
|
||||||
closing_time = ?
|
|
||||||
WHERE url = ?
|
|
||||||
""";
|
|
||||||
try (var ps = conn.prepareStatement(updateByUrlSql)) {
|
|
||||||
ps.setLong(1, auction.auctionId());
|
|
||||||
ps.setString(2, auction.title());
|
|
||||||
ps.setString(3, auction.location());
|
|
||||||
ps.setString(4, auction.city());
|
|
||||||
ps.setString(5, auction.country());
|
|
||||||
ps.setString(6, auction.typePrefix());
|
|
||||||
ps.setInt(7, auction.lotCount());
|
|
||||||
ps.setString(8, auction.firstLotClosingTime() != null ? auction.firstLotClosingTime().toString() : null);
|
|
||||||
ps.setString(9, auction.url());
|
|
||||||
|
|
||||||
int updated = ps.executeUpdate();
|
synchronized void upsertLotWithIntelligence(Lot lot) {
|
||||||
if (updated == 0) {
|
lotRepository.upsertWithIntelligence(lot);
|
||||||
// Auction doesn't exist by URL either - this is unexpected
|
}
|
||||||
log.warn("Could not insert or update auction with url={}, auction_id={} - constraint violation but no existing record found",
|
|
||||||
auction.url(), auction.auctionId());
|
|
||||||
} else {
|
|
||||||
log.debug("Updated existing auction by URL: {}", auction.url());
|
|
||||||
}
|
|
||||||
} catch (SQLException updateEx) {
|
|
||||||
// UPDATE also failed - log and swallow the exception
|
|
||||||
log.warn("Failed to update auction by URL ({}): {}", auction.url(), updateEx.getMessage());
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
throw e;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Retrieves all auctions from the database
|
|
||||||
*/
|
|
||||||
synchronized List<AuctionInfo> getAllAuctions() throws SQLException {
|
|
||||||
List<AuctionInfo> auctions = new ArrayList<>();
|
|
||||||
var sql = "SELECT auction_id, title, location, city, country, url, type, lot_count, closing_time FROM auctions";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
synchronized void updateLotCurrentBid(Lot lot) {
|
||||||
var rs = stmt.executeQuery(sql);
|
lotRepository.updateCurrentBid(lot);
|
||||||
while (rs.next()) {
|
}
|
||||||
var closingStr = rs.getString("closing_time");
|
|
||||||
LocalDateTime closing = null;
|
|
||||||
if (closingStr != null && !closingStr.isBlank()) {
|
|
||||||
try {
|
|
||||||
closing = LocalDateTime.parse(closingStr);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.debug("Invalid closing_time format for auction {}: {}", rs.getLong("auction_id"), closingStr);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
auctions.add(new AuctionInfo(
|
synchronized void updateLotNotificationFlags(Lot lot) {
|
||||||
rs.getLong("auction_id"),
|
lotRepository.updateNotificationFlags(lot);
|
||||||
rs.getString("title"),
|
}
|
||||||
rs.getString("location"),
|
|
||||||
rs.getString("city"),
|
|
||||||
rs.getString("country"),
|
|
||||||
rs.getString("url"),
|
|
||||||
rs.getString("type"),
|
|
||||||
rs.getInt("lot_count"),
|
|
||||||
closing
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return auctions;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Retrieves auctions by country code
|
|
||||||
*/
|
|
||||||
synchronized List<AuctionInfo> getAuctionsByCountry(String countryCode) throws SQLException {
|
|
||||||
List<AuctionInfo> auctions = new ArrayList<>();
|
|
||||||
var sql = "SELECT auction_id, title, location, city, country, url, type, lot_count, closing_time "
|
|
||||||
+ "FROM auctions WHERE country = ?";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var ps = conn.prepareStatement(sql)) {
|
synchronized List<Lot> getActiveLots() {
|
||||||
ps.setString(1, countryCode);
|
return lotRepository.getActiveLots();
|
||||||
var rs = ps.executeQuery();
|
}
|
||||||
while (rs.next()) {
|
|
||||||
var closingStr = rs.getString("closing_time");
|
|
||||||
LocalDateTime closing = null;
|
|
||||||
if (closingStr != null && !closingStr.isBlank()) {
|
|
||||||
try {
|
|
||||||
closing = LocalDateTime.parse(closingStr);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.debug("Invalid closing_time format for auction {}: {}", rs.getLong("auction_id"), closingStr);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
auctions.add(new AuctionInfo(
|
synchronized List<Lot> getAllLots() {
|
||||||
rs.getLong("auction_id"),
|
return lotRepository.getAllLots();
|
||||||
rs.getString("title"),
|
}
|
||||||
rs.getString("location"),
|
|
||||||
rs.getString("city"),
|
|
||||||
rs.getString("country"),
|
|
||||||
rs.getString("url"),
|
|
||||||
rs.getString("type"),
|
|
||||||
rs.getInt("lot_count"),
|
|
||||||
closing
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return auctions;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Inserts or updates a lot record (typically called by external scraper)
|
|
||||||
*/
|
|
||||||
synchronized void upsertLot(Lot lot) throws SQLException {
|
|
||||||
// First try to update existing lot by lot_id
|
|
||||||
var updateSql = """
|
|
||||||
UPDATE lots SET
|
|
||||||
sale_id = ?,
|
|
||||||
title = ?,
|
|
||||||
description = ?,
|
|
||||||
manufacturer = ?,
|
|
||||||
type = ?,
|
|
||||||
year = ?,
|
|
||||||
category = ?,
|
|
||||||
current_bid = ?,
|
|
||||||
currency = ?,
|
|
||||||
url = ?,
|
|
||||||
closing_time = ?
|
|
||||||
WHERE lot_id = ?
|
|
||||||
""";
|
|
||||||
|
|
||||||
var insertSql = """
|
synchronized List<BidHistory> getBidHistory(String lotId) {
|
||||||
INSERT OR IGNORE INTO lots (lot_id, sale_id, title, description, manufacturer, type, year, category, current_bid, currency, url, closing_time, closing_notified)
|
return lotRepository.getBidHistory(lotId);
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
}
|
||||||
""";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url)) {
|
synchronized void insertBidHistory(List<BidHistory> bidHistory) {
|
||||||
// Try UPDATE first
|
lotRepository.insertBidHistory(bidHistory);
|
||||||
try (var ps = conn.prepareStatement(updateSql)) {
|
}
|
||||||
ps.setLong(1, lot.saleId());
|
|
||||||
ps.setString(2, lot.title());
|
|
||||||
ps.setString(3, lot.description());
|
|
||||||
ps.setString(4, lot.manufacturer());
|
|
||||||
ps.setString(5, lot.type());
|
|
||||||
ps.setInt(6, lot.year());
|
|
||||||
ps.setString(7, lot.category());
|
|
||||||
ps.setDouble(8, lot.currentBid());
|
|
||||||
ps.setString(9, lot.currency());
|
|
||||||
ps.setString(10, lot.url());
|
|
||||||
ps.setString(11, lot.closingTime() != null ? lot.closingTime().toString() : null);
|
|
||||||
ps.setLong(12, lot.lotId());
|
|
||||||
|
|
||||||
int updated = ps.executeUpdate();
|
synchronized void insertImage(long lotId, String url, String filePath, List<String> labels) {
|
||||||
if (updated > 0) {
|
imageRepository.insert(lotId, url, filePath, labels);
|
||||||
return; // Successfully updated existing record
|
}
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// If no rows updated, try INSERT (ignore if conflicts with UNIQUE constraints)
|
synchronized void updateImageLabels(int imageId, List<String> labels) {
|
||||||
try (var ps = conn.prepareStatement(insertSql)) {
|
imageRepository.updateLabels(imageId, labels);
|
||||||
ps.setLong(1, lot.lotId());
|
}
|
||||||
ps.setLong(2, lot.saleId());
|
|
||||||
ps.setString(3, lot.title());
|
|
||||||
ps.setString(4, lot.description());
|
|
||||||
ps.setString(5, lot.manufacturer());
|
|
||||||
ps.setString(6, lot.type());
|
|
||||||
ps.setInt(7, lot.year());
|
|
||||||
ps.setString(8, lot.category());
|
|
||||||
ps.setDouble(9, lot.currentBid());
|
|
||||||
ps.setString(10, lot.currency());
|
|
||||||
ps.setString(11, lot.url());
|
|
||||||
ps.setString(12, lot.closingTime() != null ? lot.closingTime().toString() : null);
|
|
||||||
ps.setInt(13, lot.closingNotified() ? 1 : 0);
|
|
||||||
ps.executeUpdate();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
synchronized List<String> getImageLabels(int imageId) {
|
||||||
* Updates a lot with full intelligence data from GraphQL enrichment.
|
return imageRepository.getLabels(imageId);
|
||||||
* This is a comprehensive update that includes all 24 intelligence fields.
|
}
|
||||||
*/
|
|
||||||
synchronized void upsertLotWithIntelligence(Lot lot) throws SQLException {
|
|
||||||
var sql = """
|
|
||||||
UPDATE lots SET
|
|
||||||
sale_id = ?,
|
|
||||||
title = ?,
|
|
||||||
description = ?,
|
|
||||||
manufacturer = ?,
|
|
||||||
type = ?,
|
|
||||||
year = ?,
|
|
||||||
category = ?,
|
|
||||||
current_bid = ?,
|
|
||||||
currency = ?,
|
|
||||||
url = ?,
|
|
||||||
closing_time = ?,
|
|
||||||
followers_count = ?,
|
|
||||||
estimated_min = ?,
|
|
||||||
estimated_max = ?,
|
|
||||||
next_bid_step_in_cents = ?,
|
|
||||||
condition = ?,
|
|
||||||
category_path = ?,
|
|
||||||
city_location = ?,
|
|
||||||
country_code = ?,
|
|
||||||
bidding_status = ?,
|
|
||||||
appearance = ?,
|
|
||||||
packaging = ?,
|
|
||||||
quantity = ?,
|
|
||||||
vat = ?,
|
|
||||||
buyer_premium_percentage = ?,
|
|
||||||
remarks = ?,
|
|
||||||
starting_bid = ?,
|
|
||||||
reserve_price = ?,
|
|
||||||
reserve_met = ?,
|
|
||||||
bid_increment = ?,
|
|
||||||
view_count = ?,
|
|
||||||
first_bid_time = ?,
|
|
||||||
last_bid_time = ?,
|
|
||||||
bid_velocity = ?
|
|
||||||
WHERE lot_id = ?
|
|
||||||
""";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var ps = conn.prepareStatement(sql)) {
|
synchronized List<ImageRecord> getImagesForLot(long lotId) {
|
||||||
ps.setLong(1, lot.saleId());
|
return imageRepository.getImagesForLot(lotId)
|
||||||
ps.setString(2, lot.title());
|
.stream()
|
||||||
ps.setString(3, lot.description());
|
.map(img -> new ImageRecord(img.id(), img.lotId(), img.url(), img.filePath(), img.labels()))
|
||||||
ps.setString(4, lot.manufacturer());
|
.toList();
|
||||||
ps.setString(5, lot.type());
|
}
|
||||||
ps.setInt(6, lot.year());
|
|
||||||
ps.setString(7, lot.category());
|
|
||||||
ps.setDouble(8, lot.currentBid());
|
|
||||||
ps.setString(9, lot.currency());
|
|
||||||
ps.setString(10, lot.url());
|
|
||||||
ps.setString(11, lot.closingTime() != null ? lot.closingTime().toString() : null);
|
|
||||||
|
|
||||||
// Intelligence fields
|
synchronized List<ImageDetectionRecord> getImagesNeedingDetection() {
|
||||||
if (lot.followersCount() != null) ps.setInt(12, lot.followersCount()); else ps.setNull(12, java.sql.Types.INTEGER);
|
return imageRepository.getImagesNeedingDetection()
|
||||||
if (lot.estimatedMin() != null) ps.setDouble(13, lot.estimatedMin()); else ps.setNull(13, java.sql.Types.REAL);
|
.stream()
|
||||||
if (lot.estimatedMax() != null) ps.setDouble(14, lot.estimatedMax()); else ps.setNull(14, java.sql.Types.REAL);
|
.map(img -> new ImageDetectionRecord(img.id(), img.lotId(), img.filePath()))
|
||||||
if (lot.nextBidStepInCents() != null) ps.setLong(15, lot.nextBidStepInCents()); else ps.setNull(15, java.sql.Types.BIGINT);
|
.toList();
|
||||||
ps.setString(16, lot.condition());
|
}
|
||||||
ps.setString(17, lot.categoryPath());
|
|
||||||
ps.setString(18, lot.cityLocation());
|
|
||||||
ps.setString(19, lot.countryCode());
|
|
||||||
ps.setString(20, lot.biddingStatus());
|
|
||||||
ps.setString(21, lot.appearance());
|
|
||||||
ps.setString(22, lot.packaging());
|
|
||||||
if (lot.quantity() != null) ps.setLong(23, lot.quantity()); else ps.setNull(23, java.sql.Types.BIGINT);
|
|
||||||
if (lot.vat() != null) ps.setDouble(24, lot.vat()); else ps.setNull(24, java.sql.Types.REAL);
|
|
||||||
if (lot.buyerPremiumPercentage() != null) ps.setDouble(25, lot.buyerPremiumPercentage()); else ps.setNull(25, java.sql.Types.REAL);
|
|
||||||
ps.setString(26, lot.remarks());
|
|
||||||
if (lot.startingBid() != null) ps.setDouble(27, lot.startingBid()); else ps.setNull(27, java.sql.Types.REAL);
|
|
||||||
if (lot.reservePrice() != null) ps.setDouble(28, lot.reservePrice()); else ps.setNull(28, java.sql.Types.REAL);
|
|
||||||
if (lot.reserveMet() != null) ps.setInt(29, lot.reserveMet() ? 1 : 0); else ps.setNull(29, java.sql.Types.INTEGER);
|
|
||||||
if (lot.bidIncrement() != null) ps.setDouble(30, lot.bidIncrement()); else ps.setNull(30, java.sql.Types.REAL);
|
|
||||||
if (lot.viewCount() != null) ps.setInt(31, lot.viewCount()); else ps.setNull(31, java.sql.Types.INTEGER);
|
|
||||||
ps.setString(32, lot.firstBidTime() != null ? lot.firstBidTime().toString() : null);
|
|
||||||
ps.setString(33, lot.lastBidTime() != null ? lot.lastBidTime().toString() : null);
|
|
||||||
if (lot.bidVelocity() != null) ps.setDouble(34, lot.bidVelocity()); else ps.setNull(34, java.sql.Types.REAL);
|
|
||||||
|
|
||||||
ps.setLong(35, lot.lotId());
|
synchronized int getImageCount() {
|
||||||
|
return imageRepository.getImageCount();
|
||||||
|
}
|
||||||
|
|
||||||
int updated = ps.executeUpdate();
|
synchronized List<AuctionInfo> importAuctionsFromScraper() {
|
||||||
if (updated == 0) {
|
return jdbi.withHandle(handle -> {
|
||||||
log.warn("Failed to update lot {} - lot not found in database", lot.lotId());
|
var sql = """
|
||||||
}
|
SELECT
|
||||||
}
|
l.auction_id,
|
||||||
}
|
MIN(l.title) as title,
|
||||||
|
MIN(l.location) as location,
|
||||||
|
MIN(l.url) as url,
|
||||||
|
COUNT(*) as lots_count,
|
||||||
|
MIN(l.closing_time) as first_lot_closing_time,
|
||||||
|
MIN(l.scraped_at) as scraped_at
|
||||||
|
FROM lots l
|
||||||
|
WHERE l.auction_id IS NOT NULL
|
||||||
|
GROUP BY l.auction_id
|
||||||
|
""";
|
||||||
|
|
||||||
/**
|
return handle.createQuery(sql)
|
||||||
* Inserts a complete image record (for testing/legacy compatibility).
|
.map((rs, ctx) -> {
|
||||||
* In production, scraper inserts with local_path, monitor updates labels via updateImageLabels.
|
try {
|
||||||
*/
|
var auction = ScraperDataAdapter.fromScraperAuction(rs);
|
||||||
synchronized void insertImage(long lotId, String url, String filePath, List<String> labels) throws SQLException {
|
if (auction.auctionId() != 0L) {
|
||||||
var sql = "INSERT INTO images (lot_id, url, local_path, labels, processed_at, downloaded) VALUES (?, ?, ?, ?, ?, 1)";
|
auctionRepository.upsert(auction);
|
||||||
try (var conn = DriverManager.getConnection(this.url); var ps = conn.prepareStatement(sql)) {
|
return auction;
|
||||||
ps.setLong(1, lotId);
|
}
|
||||||
ps.setString(2, url);
|
} catch (Exception e) {
|
||||||
ps.setString(3, filePath);
|
log.warn("Failed to import auction: {}", e.getMessage());
|
||||||
ps.setString(4, String.join(",", labels));
|
}
|
||||||
ps.setLong(5, Instant.now().getEpochSecond());
|
return null;
|
||||||
ps.executeUpdate();
|
})
|
||||||
}
|
.list()
|
||||||
}
|
.stream()
|
||||||
|
.filter(a -> a != null)
|
||||||
|
.toList();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
synchronized List<Lot> importLotsFromScraper() {
|
||||||
* Updates the labels field for an image after object detection
|
return jdbi.withHandle(handle -> {
|
||||||
*/
|
var sql = "SELECT * FROM lots";
|
||||||
synchronized void updateImageLabels(int imageId, List<String> labels) throws SQLException {
|
|
||||||
var sql = "UPDATE images SET labels = ?, processed_at = ? WHERE id = ?";
|
|
||||||
try (var conn = DriverManager.getConnection(this.url); var ps = conn.prepareStatement(sql)) {
|
|
||||||
ps.setString(1, String.join(",", labels));
|
|
||||||
ps.setLong(2, Instant.now().getEpochSecond());
|
|
||||||
ps.setInt(3, imageId);
|
|
||||||
ps.executeUpdate();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
return handle.createQuery(sql)
|
||||||
* Gets the labels for a specific image
|
.map((rs, ctx) -> {
|
||||||
*/
|
try {
|
||||||
synchronized List<String> getImageLabels(int imageId) throws SQLException {
|
var lot = ScraperDataAdapter.fromScraperLot(rs);
|
||||||
var sql = "SELECT labels FROM images WHERE id = ?";
|
if (lot.lotId() != 0L && lot.saleId() != 0L) {
|
||||||
try (var conn = DriverManager.getConnection(this.url); var ps = conn.prepareStatement(sql)) {
|
lotRepository.upsert(lot);
|
||||||
ps.setInt(1, imageId);
|
return lot;
|
||||||
var rs = ps.executeQuery();
|
}
|
||||||
if (rs.next()) {
|
} catch (Exception e) {
|
||||||
var labelsStr = rs.getString("labels");
|
log.warn("Failed to import lot: {}", e.getMessage());
|
||||||
if (labelsStr != null && !labelsStr.isEmpty()) {
|
}
|
||||||
return List.of(labelsStr.split(","));
|
return null;
|
||||||
}
|
})
|
||||||
}
|
.list()
|
||||||
}
|
.stream()
|
||||||
return List.of();
|
.filter(l -> l != null)
|
||||||
}
|
.toList();
|
||||||
|
});
|
||||||
/**
|
}
|
||||||
* Retrieves images for a specific lot
|
|
||||||
*/
|
|
||||||
synchronized List<ImageRecord> getImagesForLot(long lotId) throws SQLException {
|
|
||||||
List<ImageRecord> images = new ArrayList<>();
|
|
||||||
var sql = "SELECT id, lot_id, url, local_path, labels FROM images WHERE lot_id = ?";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var ps = conn.prepareStatement(sql)) {
|
|
||||||
ps.setLong(1, lotId);
|
|
||||||
var rs = ps.executeQuery();
|
|
||||||
while (rs.next()) {
|
|
||||||
images.add(new ImageRecord(
|
|
||||||
rs.getInt("id"),
|
|
||||||
rs.getLong("lot_id"),
|
|
||||||
rs.getString("url"),
|
|
||||||
rs.getString("local_path"),
|
|
||||||
rs.getString("labels")
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return images;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Retrieves all lots that are active and need monitoring
|
|
||||||
*/
|
|
||||||
synchronized List<Lot> getActiveLots() throws SQLException {
|
|
||||||
List<Lot> list = new ArrayList<>();
|
|
||||||
var sql = "SELECT lot_id, sale_id as auction_id, title, description, manufacturer, type, year, category, " +
|
|
||||||
"current_bid, currency, url, closing_time, closing_notified FROM lots";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
// ==================== DIRECT REPOSITORY ACCESS ====================
|
||||||
var rs = stmt.executeQuery(sql);
|
// Expose repositories for modern usage patterns
|
||||||
while (rs.next()) {
|
|
||||||
try {
|
|
||||||
// Use ScraperDataAdapter to handle TEXT parsing from legacy database
|
|
||||||
var lot = ScraperDataAdapter.fromScraperLot(rs);
|
|
||||||
list.add(lot);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.warn("Failed to parse lot {}: {}", rs.getString("lot_id"), e.getMessage());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return list;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Retrieves all lots from the database
|
|
||||||
*/
|
|
||||||
synchronized List<Lot> getAllLots() throws SQLException {
|
|
||||||
return getActiveLots();
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Gets the total number of images in the database
|
|
||||||
*/
|
|
||||||
synchronized int getImageCount() throws SQLException {
|
|
||||||
var sql = "SELECT COUNT(*) as count FROM images";
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
|
||||||
var rs = stmt.executeQuery(sql);
|
|
||||||
if (rs.next()) {
|
|
||||||
return rs.getInt("count");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Updates the current bid of a lot (used by monitoring service)
|
|
||||||
*/
|
|
||||||
synchronized void updateLotCurrentBid(Lot lot) throws SQLException {
|
|
||||||
try (var conn = DriverManager.getConnection(url);
|
|
||||||
var ps = conn.prepareStatement("UPDATE lots SET current_bid = ? WHERE lot_id = ?")) {
|
|
||||||
ps.setDouble(1, lot.currentBid());
|
|
||||||
ps.setLong(2, lot.lotId());
|
|
||||||
ps.executeUpdate();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Updates the closingNotified flag of a lot
|
|
||||||
*/
|
|
||||||
synchronized void updateLotNotificationFlags(Lot lot) throws SQLException {
|
|
||||||
try (var conn = DriverManager.getConnection(url);
|
|
||||||
var ps = conn.prepareStatement("UPDATE lots SET closing_notified = ? WHERE lot_id = ?")) {
|
|
||||||
ps.setInt(1, lot.closingNotified() ? 1 : 0);
|
|
||||||
ps.setLong(2, lot.lotId());
|
|
||||||
ps.executeUpdate();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
public LotRepository lots() {
|
||||||
* Retrieves bid history for a specific lot
|
return lotRepository;
|
||||||
*/
|
}
|
||||||
synchronized List<BidHistory> getBidHistory(String lotId) throws SQLException {
|
|
||||||
List<BidHistory> history = new ArrayList<>();
|
|
||||||
var sql = "SELECT id, lot_id, bid_amount, bid_time, is_autobid, bidder_id, bidder_number " +
|
|
||||||
"FROM bid_history WHERE lot_id = ? ORDER BY bid_time DESC LIMIT 100";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url);
|
public AuctionRepository auctions() {
|
||||||
var ps = conn.prepareStatement(sql)) {
|
return auctionRepository;
|
||||||
ps.setString(1, lotId);
|
}
|
||||||
var rs = ps.executeQuery();
|
|
||||||
|
|
||||||
while (rs.next()) {
|
public ImageRepository images() {
|
||||||
LocalDateTime bidTime = null;
|
return imageRepository;
|
||||||
var bidTimeStr = rs.getString("bid_time");
|
}
|
||||||
if (bidTimeStr != null && !bidTimeStr.isBlank()) {
|
|
||||||
try {
|
|
||||||
bidTime = LocalDateTime.parse(bidTimeStr);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.debug("Invalid bid_time format: {}", bidTimeStr);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
history.add(new BidHistory(
|
public Jdbi getJdbi() {
|
||||||
rs.getInt("id"),
|
return jdbi;
|
||||||
rs.getString("lot_id"),
|
}
|
||||||
rs.getDouble("bid_amount"),
|
|
||||||
bidTime,
|
|
||||||
rs.getInt("is_autobid") != 0,
|
|
||||||
rs.getString("bidder_id"),
|
|
||||||
rs.getInt("bidder_number")
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return history;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Imports auctions from scraper's schema format.
|
|
||||||
* Since the scraper doesn't populate a separate auctions table,
|
|
||||||
* we derive auction metadata by aggregating lots data.
|
|
||||||
*
|
|
||||||
* @return List of imported auctions
|
|
||||||
*/
|
|
||||||
synchronized List<AuctionInfo> importAuctionsFromScraper() throws SQLException {
|
|
||||||
List<AuctionInfo> imported = new ArrayList<>();
|
|
||||||
|
|
||||||
// Derive auctions from lots table (scraper doesn't populate auctions table)
|
// ==================== LEGACY RECORDS ====================
|
||||||
var sql = """
|
// Keep records for backward compatibility with existing code
|
||||||
SELECT
|
|
||||||
l.auction_id,
|
|
||||||
MIN(l.title) as title,
|
|
||||||
MIN(l.location) as location,
|
|
||||||
MIN(l.url) as url,
|
|
||||||
COUNT(*) as lots_count,
|
|
||||||
MIN(l.closing_time) as first_lot_closing_time,
|
|
||||||
MIN(l.scraped_at) as scraped_at
|
|
||||||
FROM lots l
|
|
||||||
WHERE l.auction_id IS NOT NULL
|
|
||||||
GROUP BY l.auction_id
|
|
||||||
""";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
public record ImageRecord(int id, long lotId, String url, String filePath, String labels) {}
|
||||||
var rs = stmt.executeQuery(sql);
|
|
||||||
while (rs.next()) {
|
|
||||||
try {
|
|
||||||
var auction = ScraperDataAdapter.fromScraperAuction(rs);
|
|
||||||
// Skip auctions with invalid IDs (0 indicates parsing failed)
|
|
||||||
if (auction.auctionId() == 0L) {
|
|
||||||
log.debug("Skipping auction with invalid ID: auction_id={}", auction.auctionId());
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
upsertAuction(auction);
|
|
||||||
imported.add(auction);
|
|
||||||
} catch (SQLException e) {
|
|
||||||
// SQLException should be handled by upsertAuction, but if it propagates here, log it
|
|
||||||
log.warn("Failed to import auction (SQL error): {}", e.getMessage());
|
|
||||||
} catch (Exception e) {
|
|
||||||
// Other exceptions (parsing errors, etc)
|
|
||||||
log.warn("Failed to import auction (parsing error): {}", e.getMessage());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (SQLException e) {
|
|
||||||
// Table might not exist in scraper format - that's ok
|
|
||||||
log.info("ℹ️ Scraper lots table not found or incompatible schema: {}", e.getMessage());
|
|
||||||
}
|
|
||||||
|
|
||||||
return imported;
|
public record ImageDetectionRecord(int id, long lotId, String filePath) {}
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Imports lots from scraper's schema format.
|
|
||||||
* Reads from scraper's tables and converts to monitor format using adapter.
|
|
||||||
*
|
|
||||||
* @return List of imported lots
|
|
||||||
*/
|
|
||||||
synchronized List<Lot> importLotsFromScraper() throws SQLException {
|
|
||||||
List<Lot> imported = new ArrayList<>();
|
|
||||||
var sql = "SELECT lot_id, auction_id, title, description, category, " +
|
|
||||||
"current_bid, closing_time, url " +
|
|
||||||
"FROM lots";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
|
||||||
var rs = stmt.executeQuery(sql);
|
|
||||||
while (rs.next()) {
|
|
||||||
try {
|
|
||||||
var lot = ScraperDataAdapter.fromScraperLot(rs);
|
|
||||||
// Skip lots with invalid IDs (0 indicates parsing failed)
|
|
||||||
if (lot.lotId() == 0L || lot.saleId() == 0L) {
|
|
||||||
log.debug("Skipping lot with invalid ID: lot_id={}, sale_id={}", lot.lotId(), lot.saleId());
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
upsertLot(lot);
|
|
||||||
imported.add(lot);
|
|
||||||
} catch (Exception e) {
|
|
||||||
System.err.println("Failed to import lot: " + e.getMessage());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (SQLException e) {
|
|
||||||
// Table might not exist in scraper format - that's ok
|
|
||||||
log.info("ℹ️ Scraper lots table not found or incompatible schema");
|
|
||||||
}
|
|
||||||
|
|
||||||
return imported;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Gets images that have been downloaded by the scraper but need object detection.
|
|
||||||
* Only returns images that have local_path set but no labels yet.
|
|
||||||
*
|
|
||||||
* @return List of images needing object detection
|
|
||||||
*/
|
|
||||||
synchronized List<ImageDetectionRecord> getImagesNeedingDetection() throws SQLException {
|
|
||||||
List<ImageDetectionRecord> images = new ArrayList<>();
|
|
||||||
var sql = """
|
|
||||||
SELECT i.id, i.lot_id, i.local_path
|
|
||||||
FROM images i
|
|
||||||
WHERE i.local_path IS NOT NULL
|
|
||||||
AND i.local_path != ''
|
|
||||||
AND (i.labels IS NULL OR i.labels = '')
|
|
||||||
""";
|
|
||||||
|
|
||||||
try (var conn = DriverManager.getConnection(url); var stmt = conn.createStatement()) {
|
|
||||||
var rs = stmt.executeQuery(sql);
|
|
||||||
while (rs.next()) {
|
|
||||||
// Extract numeric lot ID from TEXT field (e.g., "A1-34732-49" -> 3473249)
|
|
||||||
String lotIdStr = rs.getString("lot_id");
|
|
||||||
long lotId = ScraperDataAdapter.extractNumericId(lotIdStr);
|
|
||||||
|
|
||||||
images.add(new ImageDetectionRecord(
|
|
||||||
rs.getInt("id"),
|
|
||||||
lotId,
|
|
||||||
rs.getString("local_path")
|
|
||||||
));
|
|
||||||
}
|
|
||||||
} catch (SQLException e) {
|
|
||||||
log.info("ℹ️ No images needing detection found");
|
|
||||||
}
|
|
||||||
|
|
||||||
return images;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Simple record for image data from database
|
|
||||||
*/
|
|
||||||
record ImageRecord(int id, long lotId, String url, String filePath, String labels) { }
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Record for images that need object detection processing
|
|
||||||
*/
|
|
||||||
record ImageDetectionRecord(int id, long lotId, String filePath) { }
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,106 +1,55 @@
|
|||||||
package auctiora;
|
package auctiora;
|
||||||
|
|
||||||
import jakarta.enterprise.context.ApplicationScoped;
|
|
||||||
import jakarta.inject.Inject;
|
|
||||||
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import java.io.IOException;
|
|
||||||
import java.nio.file.Files;
|
|
||||||
import java.nio.file.Paths;
|
|
||||||
import java.sql.SQLException;
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Service responsible for processing images from the IMAGES table.
|
|
||||||
* Performs object detection on already-downloaded images and updates the database.
|
|
||||||
*
|
|
||||||
* NOTE: Image downloading is handled by the external scraper process.
|
|
||||||
* This service only performs object detection on images that already have local_path set.
|
|
||||||
*/
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
class ImageProcessingService {
|
public record ImageProcessingService(DatabaseService db, ObjectDetectionService detector) {
|
||||||
|
|
||||||
private final DatabaseService db;
|
boolean processImage(int id, String path, long lot) {
|
||||||
private final ObjectDetectionService detector;
|
|
||||||
|
|
||||||
ImageProcessingService(DatabaseService db, ObjectDetectionService detector) {
|
|
||||||
this.db = db;
|
|
||||||
this.detector = detector;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Processes a single image: runs object detection and updates labels in database.
|
|
||||||
*
|
|
||||||
* @param imageId database ID of the image record
|
|
||||||
* @param localPath local file path to the downloaded image
|
|
||||||
* @param lotId lot identifier (for logging)
|
|
||||||
* @return true if processing succeeded
|
|
||||||
*/
|
|
||||||
boolean processImage(int imageId, String localPath, long lotId) {
|
|
||||||
try {
|
try {
|
||||||
// Normalize path separators (convert Windows backslashes to forward slashes)
|
path = path.replace('\\', '/');
|
||||||
localPath = localPath.replace('\\', '/');
|
var f = new java.io.File(path);
|
||||||
|
if (!f.exists() || !f.canRead()) {
|
||||||
// Check if file exists before processing
|
log.warn("Image not accessible: {}", path);
|
||||||
var file = new java.io.File(localPath);
|
|
||||||
if (!file.exists() || !file.canRead()) {
|
|
||||||
log.warn(" Image file not accessible: {}", localPath);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (f.length() > 50L * 1024 * 1024) {
|
||||||
// Check file size (skip very large files that might cause issues)
|
log.warn("Image too large: {}", path);
|
||||||
long fileSizeBytes = file.length();
|
|
||||||
if (fileSizeBytes > 50 * 1024 * 1024) { // 50 MB limit
|
|
||||||
log.warn(" Image file too large ({}MB): {}", fileSizeBytes / (1024 * 1024), localPath);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run object detection on the local file
|
var labels = detector.detectObjects(path);
|
||||||
var labels = detector.detectObjects(localPath);
|
db.updateImageLabels(id, labels);
|
||||||
|
|
||||||
// Update the database with detected labels
|
if (!labels.isEmpty())
|
||||||
db.updateImageLabels(imageId, labels);
|
log.info("Lot {}: {}", lot, String.join(", ", labels));
|
||||||
|
|
||||||
if (!labels.isEmpty()) {
|
|
||||||
log.info(" Lot {}: Detected {}", lotId, String.join(", ", labels));
|
|
||||||
}
|
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
log.error(" Failed to process image {}: {}", imageId, e.getMessage());
|
log.error("Process fail {}: {}", id, e.getMessage());
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Batch processes all pending images in the database.
|
|
||||||
* Only processes images that have been downloaded by the scraper but haven't had object detection run yet.
|
|
||||||
*/
|
|
||||||
void processPendingImages() {
|
void processPendingImages() {
|
||||||
log.info("Processing pending images...");
|
|
||||||
|
|
||||||
try {
|
try {
|
||||||
var pendingImages = db.getImagesNeedingDetection();
|
var images = db.getImagesNeedingDetection();
|
||||||
log.info("Found {} images needing object detection", pendingImages.size());
|
log.info("Pending {}", images.size());
|
||||||
|
|
||||||
var processed = 0;
|
int processed = 0, detected = 0;
|
||||||
var detected = 0;
|
|
||||||
|
for (var i : images) {
|
||||||
for (var image : pendingImages) {
|
if (processImage(i.id(), i.filePath(), i.lotId())) {
|
||||||
if (processImage(image.id(), image.filePath(), image.lotId())) {
|
|
||||||
processed++;
|
processed++;
|
||||||
// Re-fetch to check if labels were found
|
var lbl = db.getImageLabels(i.id());
|
||||||
var labels = db.getImageLabels(image.id());
|
if (lbl != null && !lbl.isEmpty()) detected++;
|
||||||
if (labels != null && !labels.isEmpty()) {
|
|
||||||
detected++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
log.info("Processed {} images, detected objects in {}", processed, detected);
|
log.info("Processed {}, detected {}", processed, detected);
|
||||||
|
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
log.error("Error processing pending images: {}", e.getMessage());
|
log.error("Batch fail: {}", e.getMessage());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ import java.time.LocalDateTime;
|
|||||||
/// Data typically populated by the external scraper process.
|
/// Data typically populated by the external scraper process.
|
||||||
/// This project enriches the data with image analysis and monitoring.
|
/// This project enriches the data with image analysis and monitoring.
|
||||||
@With
|
@With
|
||||||
record Lot(
|
public record Lot(
|
||||||
long saleId,
|
long saleId,
|
||||||
long lotId,
|
long lotId,
|
||||||
String displayId, // Full lot ID string (e.g., "A1-34732-49") for GraphQL queries
|
String displayId, // Full lot ID string (e.g., "A1-34732-49") for GraphQL queries
|
||||||
|
|||||||
@@ -16,73 +16,66 @@ import lombok.extern.slf4j.Slf4j;
|
|||||||
@Slf4j
|
@Slf4j
|
||||||
@ApplicationScoped
|
@ApplicationScoped
|
||||||
public class LotEnrichmentScheduler {
|
public class LotEnrichmentScheduler {
|
||||||
|
|
||||||
@Inject
|
@Inject LotEnrichmentService enrichmentService;
|
||||||
LotEnrichmentService enrichmentService;
|
|
||||||
|
/**
|
||||||
/**
|
* Enriches lots closing within 1 hour - HIGH PRIORITY
|
||||||
* Enriches lots closing within 1 hour - HIGH PRIORITY
|
* Runs every 5 minutes
|
||||||
* Runs every 5 minutes
|
*/
|
||||||
*/
|
@Scheduled(cron = "0 */5 * * * ?")
|
||||||
@Scheduled(cron = "0 */5 * * * ?")
|
public void enrichCriticalLots() {
|
||||||
public void enrichCriticalLots() {
|
try {
|
||||||
try {
|
log.debug("Enriching critical lots (closing < 1 hour)");
|
||||||
log.debug("Enriching critical lots (closing < 1 hour)");
|
int enriched = enrichmentService.enrichClosingSoonLots(1);
|
||||||
int enriched = enrichmentService.enrichClosingSoonLots(1);
|
if (enriched > 0) log.info("Enriched {} critical lots", enriched);
|
||||||
if (enriched > 0) {
|
} catch (Exception e) {
|
||||||
log.info("Enriched {} critical lots", enriched);
|
log.error("Failed to enrich critical lots", e);
|
||||||
}
|
}
|
||||||
} catch (Exception e) {
|
}
|
||||||
log.error("Failed to enrich critical lots", e);
|
|
||||||
}
|
/**
|
||||||
}
|
* Enriches lots closing within 6 hours - MEDIUM PRIORITY
|
||||||
|
* Runs every 30 minutes
|
||||||
/**
|
*/
|
||||||
* Enriches lots closing within 6 hours - MEDIUM PRIORITY
|
@Scheduled(cron = "0 */30 * * * ?")
|
||||||
* Runs every 30 minutes
|
public void enrichUrgentLots() {
|
||||||
*/
|
try {
|
||||||
@Scheduled(cron = "0 */30 * * * ?")
|
log.debug("Enriching urgent lots (closing < 6 hours)");
|
||||||
public void enrichUrgentLots() {
|
int enriched = enrichmentService.enrichClosingSoonLots(6);
|
||||||
try {
|
if (enriched > 0) log.info("Enriched {} urgent lots", enriched);
|
||||||
log.debug("Enriching urgent lots (closing < 6 hours)");
|
} catch (Exception e) {
|
||||||
int enriched = enrichmentService.enrichClosingSoonLots(6);
|
log.error("Failed to enrich urgent lots", e);
|
||||||
if (enriched > 0) {
|
}
|
||||||
log.info("Enriched {} urgent lots", enriched);
|
}
|
||||||
}
|
|
||||||
} catch (Exception e) {
|
/**
|
||||||
log.error("Failed to enrich urgent lots", e);
|
* Enriches lots closing within 24 hours - NORMAL PRIORITY
|
||||||
}
|
* Runs every 2 hours
|
||||||
}
|
*/
|
||||||
|
@Scheduled(cron = "0 0 */2 * * ?")
|
||||||
/**
|
public void enrichDailyLots() {
|
||||||
* Enriches lots closing within 24 hours - NORMAL PRIORITY
|
try {
|
||||||
* Runs every 2 hours
|
log.debug("Enriching daily lots (closing < 24 hours)");
|
||||||
*/
|
int enriched = enrichmentService.enrichClosingSoonLots(24);
|
||||||
@Scheduled(cron = "0 0 */2 * * ?")
|
if (enriched > 0) log.info("Enriched {} daily lots", enriched);
|
||||||
public void enrichDailyLots() {
|
} catch (Exception e) {
|
||||||
try {
|
log.error("Failed to enrich daily lots", e);
|
||||||
log.debug("Enriching daily lots (closing < 24 hours)");
|
}
|
||||||
int enriched = enrichmentService.enrichClosingSoonLots(24);
|
}
|
||||||
if (enriched > 0) {
|
|
||||||
log.info("Enriched {} daily lots", enriched);
|
/**
|
||||||
}
|
* Enriches all active lots - LOW PRIORITY
|
||||||
} catch (Exception e) {
|
* Runs every 6 hours to keep all data fresh
|
||||||
log.error("Failed to enrich daily lots", e);
|
*/
|
||||||
}
|
@Scheduled(cron = "0 0 */6 * * ?")
|
||||||
}
|
public void enrichAllLots() {
|
||||||
|
try {
|
||||||
/**
|
log.info("Starting full enrichment of all lots");
|
||||||
* Enriches all active lots - LOW PRIORITY
|
int enriched = enrichmentService.enrichAllActiveLots();
|
||||||
* Runs every 6 hours to keep all data fresh
|
log.info("Full enrichment complete: {} lots updated", enriched);
|
||||||
*/
|
} catch (Exception e) {
|
||||||
@Scheduled(cron = "0 0 */6 * * ?")
|
log.error("Failed to enrich all lots", e);
|
||||||
public void enrichAllLots() {
|
}
|
||||||
try {
|
}
|
||||||
log.info("Starting full enrichment of all lots");
|
|
||||||
int enriched = enrichmentService.enrichAllActiveLots();
|
|
||||||
log.info("Full enrichment complete: {} lots updated", enriched);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("Failed to enrich all lots", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,190 +16,186 @@ import java.util.stream.Collectors;
|
|||||||
@Slf4j
|
@Slf4j
|
||||||
@ApplicationScoped
|
@ApplicationScoped
|
||||||
public class LotEnrichmentService {
|
public class LotEnrichmentService {
|
||||||
|
|
||||||
@Inject
|
@Inject TroostwijkGraphQLClient graphQLClient;
|
||||||
TroostwijkGraphQLClient graphQLClient;
|
@Inject DatabaseService db;
|
||||||
|
/**
|
||||||
@Inject
|
* Enriches a single lot with GraphQL intelligence data
|
||||||
DatabaseService db;
|
*/
|
||||||
|
public boolean enrichLot(Lot lot) {
|
||||||
/**
|
if (lot.displayId() == null || lot.displayId().isBlank()) {
|
||||||
* Enriches a single lot with GraphQL intelligence data
|
log.debug("Cannot enrich lot {} - missing displayId", lot.lotId());
|
||||||
*/
|
return false;
|
||||||
public boolean enrichLot(Lot lot) {
|
}
|
||||||
if (lot.displayId() == null || lot.displayId().isBlank()) {
|
|
||||||
log.debug("Cannot enrich lot {} - missing displayId", lot.lotId());
|
try {
|
||||||
|
var intelligence = graphQLClient.fetchLotIntelligence(lot.displayId(), lot.lotId());
|
||||||
|
if (intelligence == null) {
|
||||||
|
log.debug("No intelligence data for lot {}", lot.displayId());
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
// Merge intelligence with existing lot data
|
||||||
|
var enrichedLot = mergeLotWithIntelligence(lot, intelligence);
|
||||||
|
db.upsertLotWithIntelligence(enrichedLot);
|
||||||
|
|
||||||
|
log.debug("Enriched lot {} with GraphQL data", lot.lotId());
|
||||||
|
return true;
|
||||||
|
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("Failed to enrich lot {}: {}", lot.lotId(), e.getMessage());
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Enriches multiple lots sequentially
|
||||||
|
* @param lots List of lots to enrich
|
||||||
|
* @return Number of successfully enriched lots
|
||||||
|
*/
|
||||||
|
public int enrichLotsBatch(List<Lot> lots) {
|
||||||
|
if (lots.isEmpty()) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info("Enriching {} lots via GraphQL", lots.size());
|
||||||
|
int enrichedCount = 0;
|
||||||
|
|
||||||
|
for (var lot : lots) {
|
||||||
|
if (lot.displayId() == null || lot.displayId().isBlank()) {
|
||||||
|
log.debug("Skipping lot {} - missing displayId", lot.lotId());
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
var intelligence = graphQLClient.fetchLotIntelligence(lot.displayId(), lot.lotId());
|
var intelligence = graphQLClient.fetchLotIntelligence(lot.displayId(), lot.lotId());
|
||||||
if (intelligence == null) {
|
if (intelligence != null) {
|
||||||
log.debug("No intelligence data for lot {}", lot.displayId());
|
var enrichedLot = mergeLotWithIntelligence(lot, intelligence);
|
||||||
return false;
|
db.upsertLotWithIntelligence(enrichedLot);
|
||||||
|
enrichedCount++;
|
||||||
|
} else {
|
||||||
|
log.debug("No intelligence data for lot {}", lot.displayId());
|
||||||
}
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
// Merge intelligence with existing lot data
|
log.warn("Failed to enrich lot {}: {}", lot.displayId(), e.getMessage());
|
||||||
var enrichedLot = mergeLotWithIntelligence(lot, intelligence);
|
}
|
||||||
db.upsertLotWithIntelligence(enrichedLot);
|
|
||||||
|
// Small delay to respect rate limits (handled by RateLimitedHttpClient)
|
||||||
log.debug("Enriched lot {} with GraphQL data", lot.lotId());
|
}
|
||||||
return true;
|
|
||||||
|
log.info("Successfully enriched {}/{} lots", enrichedCount, lots.size());
|
||||||
} catch (Exception e) {
|
return enrichedCount;
|
||||||
log.warn("Failed to enrich lot {}: {}", lot.lotId(), e.getMessage());
|
}
|
||||||
return false;
|
|
||||||
}
|
/**
|
||||||
}
|
* Enriches lots closing soon (within specified hours) with higher priority
|
||||||
|
*/
|
||||||
/**
|
public int enrichClosingSoonLots(int hoursUntilClose) {
|
||||||
* Enriches multiple lots sequentially
|
try {
|
||||||
* @param lots List of lots to enrich
|
var allLots = db.getAllLots();
|
||||||
* @return Number of successfully enriched lots
|
var closingSoon = allLots.stream()
|
||||||
*/
|
.filter(lot -> lot.closingTime() != null)
|
||||||
public int enrichLotsBatch(List<Lot> lots) {
|
.filter(lot -> {
|
||||||
if (lots.isEmpty()) {
|
long minutes = lot.minutesUntilClose();
|
||||||
|
return minutes > 0 && minutes <= hoursUntilClose * 60;
|
||||||
|
})
|
||||||
|
.toList();
|
||||||
|
|
||||||
|
if (closingSoon.isEmpty()) {
|
||||||
|
log.debug("No lots closing within {} hours", hoursUntilClose);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
log.info("Enriching {} lots via GraphQL", lots.size());
|
log.info("Enriching {} lots closing within {} hours", closingSoon.size(), hoursUntilClose);
|
||||||
int enrichedCount = 0;
|
return enrichLotsBatch(closingSoon);
|
||||||
|
|
||||||
for (var lot : lots) {
|
} catch (Exception e) {
|
||||||
if (lot.displayId() == null || lot.displayId().isBlank()) {
|
log.error("Failed to enrich closing soon lots: {}", e.getMessage());
|
||||||
log.debug("Skipping lot {} - missing displayId", lot.lotId());
|
return 0;
|
||||||
continue;
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Enriches all active lots (can be slow for large datasets)
|
||||||
|
*/
|
||||||
|
public int enrichAllActiveLots() {
|
||||||
|
try {
|
||||||
|
var allLots = db.getAllLots();
|
||||||
|
log.info("Enriching all {} active lots", allLots.size());
|
||||||
|
|
||||||
|
// Process in batches to avoid overwhelming the API
|
||||||
|
int batchSize = 50;
|
||||||
|
int totalEnriched = 0;
|
||||||
|
|
||||||
|
for (int i = 0; i < allLots.size(); i += batchSize) {
|
||||||
|
int end = Math.min(i + batchSize, allLots.size());
|
||||||
|
List<Lot> batch = allLots.subList(i, end);
|
||||||
|
|
||||||
|
int enriched = enrichLotsBatch(batch);
|
||||||
|
totalEnriched += enriched;
|
||||||
|
|
||||||
|
// Small delay between batches to respect rate limits
|
||||||
|
if (end < allLots.size()) {
|
||||||
|
Thread.sleep(1000);
|
||||||
}
|
}
|
||||||
|
}
|
||||||
try {
|
|
||||||
var intelligence = graphQLClient.fetchLotIntelligence(lot.displayId(), lot.lotId());
|
log.info("Finished enriching all lots. Total enriched: {}/{}", totalEnriched, allLots.size());
|
||||||
if (intelligence != null) {
|
return totalEnriched;
|
||||||
var enrichedLot = mergeLotWithIntelligence(lot, intelligence);
|
|
||||||
db.upsertLotWithIntelligence(enrichedLot);
|
} catch (Exception e) {
|
||||||
enrichedCount++;
|
log.error("Failed to enrich all lots: {}", e.getMessage());
|
||||||
} else {
|
return 0;
|
||||||
log.debug("No intelligence data for lot {}", lot.displayId());
|
}
|
||||||
}
|
}
|
||||||
} catch (Exception e) {
|
|
||||||
log.warn("Failed to enrich lot {}: {}", lot.displayId(), e.getMessage());
|
/**
|
||||||
}
|
* Merges existing lot data with GraphQL intelligence
|
||||||
|
*/
|
||||||
// Small delay to respect rate limits (handled by RateLimitedHttpClient)
|
private Lot mergeLotWithIntelligence(Lot lot, LotIntelligence intel) {
|
||||||
}
|
return new Lot(
|
||||||
|
lot.saleId(),
|
||||||
log.info("Successfully enriched {}/{} lots", enrichedCount, lots.size());
|
lot.lotId(),
|
||||||
return enrichedCount;
|
lot.displayId(), // Preserve displayId
|
||||||
}
|
lot.title(),
|
||||||
|
lot.description(),
|
||||||
/**
|
lot.manufacturer(),
|
||||||
* Enriches lots closing soon (within specified hours) with higher priority
|
lot.type(),
|
||||||
*/
|
lot.year(),
|
||||||
public int enrichClosingSoonLots(int hoursUntilClose) {
|
lot.category(),
|
||||||
try {
|
lot.currentBid(),
|
||||||
var allLots = db.getAllLots();
|
lot.currency(),
|
||||||
var closingSoon = allLots.stream()
|
lot.url(),
|
||||||
.filter(lot -> lot.closingTime() != null)
|
lot.closingTime(),
|
||||||
.filter(lot -> {
|
lot.closingNotified(),
|
||||||
long minutes = lot.minutesUntilClose();
|
// HIGH PRIORITY FIELDS from GraphQL
|
||||||
return minutes > 0 && minutes <= hoursUntilClose * 60;
|
intel.followersCount(),
|
||||||
})
|
intel.estimatedMin(),
|
||||||
.toList();
|
intel.estimatedMax(),
|
||||||
|
intel.nextBidStepInCents(),
|
||||||
if (closingSoon.isEmpty()) {
|
intel.condition(),
|
||||||
log.debug("No lots closing within {} hours", hoursUntilClose);
|
intel.categoryPath(),
|
||||||
return 0;
|
intel.cityLocation(),
|
||||||
}
|
intel.countryCode(),
|
||||||
|
// MEDIUM PRIORITY FIELDS
|
||||||
log.info("Enriching {} lots closing within {} hours", closingSoon.size(), hoursUntilClose);
|
intel.biddingStatus(),
|
||||||
return enrichLotsBatch(closingSoon);
|
intel.appearance(),
|
||||||
|
intel.packaging(),
|
||||||
} catch (Exception e) {
|
intel.quantity(),
|
||||||
log.error("Failed to enrich closing soon lots: {}", e.getMessage());
|
intel.vat(),
|
||||||
return 0;
|
intel.buyerPremiumPercentage(),
|
||||||
}
|
intel.remarks(),
|
||||||
}
|
// BID INTELLIGENCE FIELDS
|
||||||
|
intel.startingBid(),
|
||||||
/**
|
intel.reservePrice(),
|
||||||
* Enriches all active lots (can be slow for large datasets)
|
intel.reserveMet(),
|
||||||
*/
|
intel.bidIncrement(),
|
||||||
public int enrichAllActiveLots() {
|
intel.viewCount(),
|
||||||
try {
|
intel.firstBidTime(),
|
||||||
var allLots = db.getAllLots();
|
intel.lastBidTime(),
|
||||||
log.info("Enriching all {} active lots", allLots.size());
|
intel.bidVelocity(),
|
||||||
|
null, // condition_score (computed separately)
|
||||||
// Process in batches to avoid overwhelming the API
|
null // provenance_docs (computed separately)
|
||||||
int batchSize = 50;
|
);
|
||||||
int totalEnriched = 0;
|
}
|
||||||
|
|
||||||
for (int i = 0; i < allLots.size(); i += batchSize) {
|
|
||||||
int end = Math.min(i + batchSize, allLots.size());
|
|
||||||
List<Lot> batch = allLots.subList(i, end);
|
|
||||||
|
|
||||||
int enriched = enrichLotsBatch(batch);
|
|
||||||
totalEnriched += enriched;
|
|
||||||
|
|
||||||
// Small delay between batches to respect rate limits
|
|
||||||
if (end < allLots.size()) {
|
|
||||||
Thread.sleep(1000);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
log.info("Finished enriching all lots. Total enriched: {}/{}", totalEnriched, allLots.size());
|
|
||||||
return totalEnriched;
|
|
||||||
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("Failed to enrich all lots: {}", e.getMessage());
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Merges existing lot data with GraphQL intelligence
|
|
||||||
*/
|
|
||||||
private Lot mergeLotWithIntelligence(Lot lot, LotIntelligence intel) {
|
|
||||||
return new Lot(
|
|
||||||
lot.saleId(),
|
|
||||||
lot.lotId(),
|
|
||||||
lot.displayId(), // Preserve displayId
|
|
||||||
lot.title(),
|
|
||||||
lot.description(),
|
|
||||||
lot.manufacturer(),
|
|
||||||
lot.type(),
|
|
||||||
lot.year(),
|
|
||||||
lot.category(),
|
|
||||||
lot.currentBid(),
|
|
||||||
lot.currency(),
|
|
||||||
lot.url(),
|
|
||||||
lot.closingTime(),
|
|
||||||
lot.closingNotified(),
|
|
||||||
// HIGH PRIORITY FIELDS from GraphQL
|
|
||||||
intel.followersCount(),
|
|
||||||
intel.estimatedMin(),
|
|
||||||
intel.estimatedMax(),
|
|
||||||
intel.nextBidStepInCents(),
|
|
||||||
intel.condition(),
|
|
||||||
intel.categoryPath(),
|
|
||||||
intel.cityLocation(),
|
|
||||||
intel.countryCode(),
|
|
||||||
// MEDIUM PRIORITY FIELDS
|
|
||||||
intel.biddingStatus(),
|
|
||||||
intel.appearance(),
|
|
||||||
intel.packaging(),
|
|
||||||
intel.quantity(),
|
|
||||||
intel.vat(),
|
|
||||||
intel.buyerPremiumPercentage(),
|
|
||||||
intel.remarks(),
|
|
||||||
// BID INTELLIGENCE FIELDS
|
|
||||||
intel.startingBid(),
|
|
||||||
intel.reservePrice(),
|
|
||||||
intel.reserveMet(),
|
|
||||||
intel.bidIncrement(),
|
|
||||||
intel.viewCount(),
|
|
||||||
intel.firstBidTime(),
|
|
||||||
intel.lastBidTime(),
|
|
||||||
intel.bidVelocity(),
|
|
||||||
null, // condition_score (computed separately)
|
|
||||||
null // provenance_docs (computed separately)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,231 +0,0 @@
|
|||||||
package auctiora;
|
|
||||||
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
|
||||||
import org.opencv.core.Core;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Main entry point for Troostwijk Auction Monitor.
|
|
||||||
*
|
|
||||||
* ARCHITECTURE:
|
|
||||||
* This project focuses on:
|
|
||||||
* 1. Image processing and object detection
|
|
||||||
* 2. Bid monitoring and notifications
|
|
||||||
* 3. Data enrichment
|
|
||||||
*
|
|
||||||
* Auction/Lot scraping is handled by the external ARCHITECTURE-TROOSTWIJK-SCRAPER process.
|
|
||||||
* That process populates the auctions and lots tables in the shared database.
|
|
||||||
* This process reads from those tables and enriches them with:
|
|
||||||
* - Downloaded images
|
|
||||||
* - Object detection labels
|
|
||||||
* - Bid monitoring
|
|
||||||
* - Notifications
|
|
||||||
*/
|
|
||||||
@Slf4j
|
|
||||||
public class Main {
|
|
||||||
|
|
||||||
@SuppressWarnings("restricted")
|
|
||||||
private static Object loadOpenCV() {
|
|
||||||
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
public static void main(String[] args) throws Exception {
|
|
||||||
log.info("=== Troostwijk Auction Monitor ===\n");
|
|
||||||
|
|
||||||
// Parse command line arguments
|
|
||||||
var mode = args.length > 0 ? args[0] : "workflow";
|
|
||||||
|
|
||||||
// Configuration - Windows paths
|
|
||||||
var databaseFile = System.getenv().getOrDefault("DATABASE_FILE", "C:\\mnt\\okcomputer\\output\\cache.db");
|
|
||||||
var notificationConfig = System.getenv().getOrDefault("NOTIFICATION_CONFIG", "desktop");
|
|
||||||
|
|
||||||
// YOLO model paths (optional - monitor works without object detection)
|
|
||||||
var yoloCfg = "models/yolov4.cfg";
|
|
||||||
var yoloWeights = "models/yolov4.weights";
|
|
||||||
var yoloClasses = "models/coco.names";
|
|
||||||
|
|
||||||
// Load native OpenCV library (only if models exist)
|
|
||||||
try {
|
|
||||||
loadOpenCV();
|
|
||||||
log.info("✓ OpenCV loaded");
|
|
||||||
} catch (UnsatisfiedLinkError e) {
|
|
||||||
log.info("⚠️ OpenCV not available - image detection disabled");
|
|
||||||
}
|
|
||||||
|
|
||||||
switch (mode.toLowerCase()) {
|
|
||||||
case "workflow":
|
|
||||||
runWorkflowMode(databaseFile, notificationConfig, yoloCfg, yoloWeights, yoloClasses);
|
|
||||||
break;
|
|
||||||
|
|
||||||
case "once":
|
|
||||||
runOnceMode(databaseFile, notificationConfig, yoloCfg, yoloWeights, yoloClasses);
|
|
||||||
break;
|
|
||||||
|
|
||||||
case "legacy":
|
|
||||||
runLegacyMode(databaseFile, notificationConfig, yoloCfg, yoloWeights, yoloClasses);
|
|
||||||
break;
|
|
||||||
|
|
||||||
case "status":
|
|
||||||
showStatus(databaseFile, notificationConfig, yoloCfg, yoloWeights, yoloClasses);
|
|
||||||
break;
|
|
||||||
|
|
||||||
default:
|
|
||||||
showUsage();
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* WORKFLOW MODE: Run orchestrated scheduled workflows (default)
|
|
||||||
* This is the recommended mode for production use.
|
|
||||||
*/
|
|
||||||
private static void runWorkflowMode(String dbPath, String notifConfig,
|
|
||||||
String yoloCfg, String yoloWeights, String yoloClasses)
|
|
||||||
throws Exception {
|
|
||||||
|
|
||||||
log.info("🚀 Starting in WORKFLOW MODE (Orchestrated Scheduling)\n");
|
|
||||||
|
|
||||||
var orchestrator = new WorkflowOrchestrator(
|
|
||||||
dbPath, notifConfig, yoloCfg, yoloWeights, yoloClasses
|
|
||||||
);
|
|
||||||
|
|
||||||
// Show initial status
|
|
||||||
orchestrator.printStatus();
|
|
||||||
|
|
||||||
// Start all scheduled workflows
|
|
||||||
orchestrator.startScheduledWorkflows();
|
|
||||||
|
|
||||||
log.info("✓ All workflows are running");
|
|
||||||
log.info(" - Scraper import: every 30 min");
|
|
||||||
log.info(" - Image processing: every 1 hour");
|
|
||||||
log.info(" - Bid monitoring: every 15 min");
|
|
||||||
log.info(" - Closing alerts: every 5 min");
|
|
||||||
log.info("\nPress Ctrl+C to stop.\n");
|
|
||||||
|
|
||||||
// Add shutdown hook
|
|
||||||
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
|
|
||||||
log.info("\n🛑 Shutdown signal received...");
|
|
||||||
orchestrator.shutdown();
|
|
||||||
}));
|
|
||||||
|
|
||||||
// Keep application alive
|
|
||||||
try {
|
|
||||||
Thread.sleep(Long.MAX_VALUE);
|
|
||||||
} catch (InterruptedException e) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
orchestrator.shutdown();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* ONCE MODE: Run complete workflow once and exit
|
|
||||||
* Useful for cron jobs or scheduled tasks.
|
|
||||||
*/
|
|
||||||
private static void runOnceMode(String dbPath, String notifConfig,
|
|
||||||
String yoloCfg, String yoloWeights, String yoloClasses)
|
|
||||||
throws Exception {
|
|
||||||
|
|
||||||
log.info("🔄 Starting in ONCE MODE (Single Execution)\n");
|
|
||||||
|
|
||||||
var orchestrator = new WorkflowOrchestrator(
|
|
||||||
dbPath, notifConfig, yoloCfg, yoloWeights, yoloClasses
|
|
||||||
);
|
|
||||||
|
|
||||||
orchestrator.runCompleteWorkflowOnce();
|
|
||||||
|
|
||||||
log.info("✓ Workflow execution completed. Exiting.\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* LEGACY MODE: Original monitoring approach
|
|
||||||
* Kept for backward compatibility.
|
|
||||||
*/
|
|
||||||
private static void runLegacyMode(String dbPath, String notifConfig,
|
|
||||||
String yoloCfg, String yoloWeights, String yoloClasses)
|
|
||||||
throws Exception {
|
|
||||||
|
|
||||||
log.info("⚙️ Starting in LEGACY MODE\n");
|
|
||||||
|
|
||||||
var monitor = new TroostwijkMonitor(dbPath, notifConfig,
|
|
||||||
yoloCfg, yoloWeights, yoloClasses);
|
|
||||||
|
|
||||||
log.info("\n📊 Current Database State:");
|
|
||||||
monitor.printDatabaseStats();
|
|
||||||
|
|
||||||
log.info("\n[1/2] Processing images...");
|
|
||||||
monitor.processPendingImages();
|
|
||||||
|
|
||||||
log.info("\n[2/2] Starting bid monitoring...");
|
|
||||||
monitor.scheduleMonitoring();
|
|
||||||
|
|
||||||
log.info("\n✓ Monitor is running. Press Ctrl+C to stop.\n");
|
|
||||||
log.info("NOTE: This process expects auction/lot data from the external scraper.");
|
|
||||||
log.info(" Make sure ARCHITECTURE-TROOSTWIJK-SCRAPER is running and populating the database.\n");
|
|
||||||
|
|
||||||
try {
|
|
||||||
Thread.sleep(Long.MAX_VALUE);
|
|
||||||
} catch (InterruptedException e) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
log.info("Monitor interrupted, exiting.");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* STATUS MODE: Show current status and exit
|
|
||||||
*/
|
|
||||||
private static void showStatus(String dbPath, String notifConfig,
|
|
||||||
String yoloCfg, String yoloWeights, String yoloClasses)
|
|
||||||
throws Exception {
|
|
||||||
|
|
||||||
log.info("📊 Checking Status...\n");
|
|
||||||
|
|
||||||
var orchestrator = new WorkflowOrchestrator(
|
|
||||||
dbPath, notifConfig, yoloCfg, yoloWeights, yoloClasses
|
|
||||||
);
|
|
||||||
|
|
||||||
orchestrator.printStatus();
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Show usage information
|
|
||||||
*/
|
|
||||||
private static void showUsage() {
|
|
||||||
log.info("Usage: java -jar troostwijk-monitor.jar [mode]\n");
|
|
||||||
log.info("Modes:");
|
|
||||||
log.info(" workflow - Run orchestrated scheduled workflows (default)");
|
|
||||||
log.info(" once - Run complete workflow once and exit (for cron)");
|
|
||||||
log.info(" legacy - Run original monitoring approach");
|
|
||||||
log.info(" status - Show current status and exit");
|
|
||||||
log.info("\nEnvironment Variables:");
|
|
||||||
log.info(" DATABASE_FILE - Path to SQLite database");
|
|
||||||
log.info(" (default: C:\\mnt\\okcomputer\\output\\cache.db)");
|
|
||||||
log.info(" NOTIFICATION_CONFIG - 'desktop' or 'smtp:user:pass:email'");
|
|
||||||
log.info(" (default: desktop)");
|
|
||||||
log.info("\nExamples:");
|
|
||||||
log.info(" java -jar troostwijk-monitor.jar workflow");
|
|
||||||
log.info(" java -jar troostwijk-monitor.jar once");
|
|
||||||
log.info(" java -jar troostwijk-monitor.jar status");
|
|
||||||
IO.println();
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Alternative entry point for container environments.
|
|
||||||
* Simply keeps the container alive for manual commands.
|
|
||||||
*/
|
|
||||||
public static void main2(String[] args) {
|
|
||||||
if (args.length > 0) {
|
|
||||||
log.info("Command mode - exiting to allow shell commands");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
log.info("Troostwijk Monitor container is running and healthy.");
|
|
||||||
log.info("Use 'docker exec' or 'dokku run' to execute commands.");
|
|
||||||
|
|
||||||
try {
|
|
||||||
Thread.sleep(Long.MAX_VALUE);
|
|
||||||
} catch (InterruptedException e) {
|
|
||||||
Thread.currentThread().interrupt();
|
|
||||||
log.info("Container interrupted, exiting.");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,48 +1,49 @@
|
|||||||
package auctiora;
|
package auctiora;
|
||||||
|
|
||||||
import jakarta.enterprise.context.ApplicationScoped;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import jakarta.inject.Inject;
|
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
|
||||||
|
|
||||||
import javax.mail.*;
|
import javax.mail.*;
|
||||||
import javax.mail.internet.*;
|
import javax.mail.internet.*;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
|
||||||
import java.awt.*;
|
import java.awt.*;
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
import java.util.Properties;
|
import java.util.Properties;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class NotificationService {
|
public record NotificationService(Config cfg) {
|
||||||
|
|
||||||
private final Config config;
|
// Extra convenience constructor: raw string → Config
|
||||||
|
public NotificationService(String raw) {
|
||||||
public NotificationService(String cfg) {
|
this(Config.parse(raw));
|
||||||
this.config = Config.parse(cfg);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public void sendNotification(String message, String title, int priority) {
|
public void sendNotification(String msg, String title, int prio) {
|
||||||
if (config.useDesktop()) sendDesktop(title, message, priority);
|
if (cfg.useDesktop()) sendDesktop(title, msg, prio);
|
||||||
if (config.useEmail()) sendEmail(title, message, priority);
|
if (cfg.useEmail()) sendEmail(title, msg, prio);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendDesktop(String title, String msg, int prio) {
|
private void sendDesktop(String title, String msg, int prio) {
|
||||||
try {
|
try {
|
||||||
if (!SystemTray.isSupported()) {
|
if (!SystemTray.isSupported()) {
|
||||||
log.info("Desktop notifications not supported — " + title + " / " + msg);
|
log.info("Desktop not supported: {}", title);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
var tray = SystemTray.getSystemTray();
|
|
||||||
var image = Toolkit.getDefaultToolkit().createImage(new byte[0]);
|
var tray = SystemTray.getSystemTray();
|
||||||
var trayIcon = new TrayIcon(image, "NotificationService");
|
var icon = new TrayIcon(
|
||||||
trayIcon.setImageAutoSize(true);
|
Toolkit.getDefaultToolkit().createImage(new byte[0]),
|
||||||
|
"notify"
|
||||||
|
);
|
||||||
|
icon.setImageAutoSize(true);
|
||||||
|
tray.add(icon);
|
||||||
|
|
||||||
var type = prio > 0 ? TrayIcon.MessageType.WARNING : TrayIcon.MessageType.INFO;
|
var type = prio > 0 ? TrayIcon.MessageType.WARNING : TrayIcon.MessageType.INFO;
|
||||||
tray.add(trayIcon);
|
icon.displayMessage(title, msg, type);
|
||||||
trayIcon.displayMessage(title, msg, type);
|
|
||||||
Thread.sleep(2000);
|
Thread.sleep(2000);
|
||||||
tray.remove(trayIcon);
|
tray.remove(icon);
|
||||||
log.info("Desktop notification sent: " + title);
|
log.info("Desktop notification: {}", title);
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
System.err.println("Desktop notification failed: " + e);
|
log.warn("Desktop failed: {}", e.getMessage());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,29 +58,32 @@ public class NotificationService {
|
|||||||
|
|
||||||
var session = Session.getInstance(props, new Authenticator() {
|
var session = Session.getInstance(props, new Authenticator() {
|
||||||
|
|
||||||
|
@Override
|
||||||
protected PasswordAuthentication getPasswordAuthentication() {
|
protected PasswordAuthentication getPasswordAuthentication() {
|
||||||
return new PasswordAuthentication(config.smtpUsername(), config.smtpPassword());
|
return new PasswordAuthentication(cfg.smtpUsername(), cfg.smtpPassword());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
var m = new MimeMessage(session);
|
var m = new MimeMessage(session);
|
||||||
m.setFrom(new InternetAddress(config.smtpUsername()));
|
m.setFrom(new InternetAddress(cfg.smtpUsername()));
|
||||||
m.setRecipients(Message.RecipientType.TO, InternetAddress.parse(config.toEmail()));
|
m.setRecipients(Message.RecipientType.TO, InternetAddress.parse(cfg.toEmail()));
|
||||||
m.setSubject("[Troostwijk] " + title);
|
m.setSubject("[Troostwijk] " + title);
|
||||||
m.setText(msg);
|
m.setText(msg);
|
||||||
m.setSentDate(new Date());
|
m.setSentDate(new Date());
|
||||||
|
|
||||||
if (prio > 0) {
|
if (prio > 0) {
|
||||||
m.setHeader("X-Priority", "1");
|
m.setHeader("X-Priority", "1");
|
||||||
m.setHeader("Importance", "High");
|
m.setHeader("Importance", "High");
|
||||||
}
|
}
|
||||||
|
|
||||||
Transport.send(m);
|
Transport.send(m);
|
||||||
log.info("Email notification sent: " + title);
|
log.info("Email notification: {}", title);
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
log.info("Email notification failed: " + e);
|
log.warn("Email failed: {}", e.getMessage());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private record Config(
|
public record Config(
|
||||||
boolean useDesktop,
|
boolean useDesktop,
|
||||||
boolean useEmail,
|
boolean useEmail,
|
||||||
String smtpUsername,
|
String smtpUsername,
|
||||||
@@ -87,16 +91,20 @@ public class NotificationService {
|
|||||||
String toEmail
|
String toEmail
|
||||||
) {
|
) {
|
||||||
|
|
||||||
static Config parse(String cfg) {
|
public static Config parse(String raw) {
|
||||||
if ("desktop".equalsIgnoreCase(cfg)) {
|
if ("desktop".equalsIgnoreCase(raw)) {
|
||||||
return new Config(true, false, null, null, null);
|
return new Config(true, false, null, null, null);
|
||||||
} else if (cfg.startsWith("smtp:")) {
|
|
||||||
var parts = cfg.split(":", -1); // Use -1 to include trailing empty strings
|
|
||||||
if (parts.length < 4)
|
|
||||||
throw new IllegalArgumentException("Email config must be 'smtp:username:password:toEmail'");
|
|
||||||
return new Config(true, true, parts[1], parts[2], parts[3]);
|
|
||||||
}
|
}
|
||||||
throw new IllegalArgumentException("Config must be 'desktop' or 'smtp:username:password:toEmail'");
|
|
||||||
|
if (raw != null && raw.startsWith("smtp:")) {
|
||||||
|
var p = raw.split(":", -1);
|
||||||
|
if (p.length < 4) {
|
||||||
|
throw new IllegalArgumentException("Format: smtp:username:password:toEmail");
|
||||||
|
}
|
||||||
|
return new Config(true, true, p[1], p[2], p[3]);
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new IllegalArgumentException("Use 'desktop' or 'smtp:username:password:toEmail'");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,10 +1,13 @@
|
|||||||
package auctiora;
|
package auctiora;
|
||||||
|
|
||||||
|
import jakarta.annotation.PostConstruct;
|
||||||
import jakarta.enterprise.context.ApplicationScoped;
|
import jakarta.enterprise.context.ApplicationScoped;
|
||||||
import jakarta.inject.Inject;
|
import jakarta.inject.Inject;
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
||||||
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import nu.pattern.OpenCV;
|
||||||
|
import org.opencv.core.Core;
|
||||||
import org.opencv.core.Mat;
|
import org.opencv.core.Mat;
|
||||||
import org.opencv.core.Scalar;
|
import org.opencv.core.Scalar;
|
||||||
import org.opencv.core.Size;
|
import org.opencv.core.Size;
|
||||||
@@ -23,7 +26,7 @@ import static org.opencv.dnn.Dnn.DNN_TARGET_CPU;
|
|||||||
/**
|
/**
|
||||||
* Service for performing object detection on images using OpenCV's DNN
|
* Service for performing object detection on images using OpenCV's DNN
|
||||||
* module. The DNN module can load pre‑trained models from several
|
* module. The DNN module can load pre‑trained models from several
|
||||||
* frameworks (Darknet, TensorFlow, ONNX, etc.)【784097309529506†L209-L233】. Here
|
* frameworks (Darknet, TensorFlow, ONNX, etc.). Here
|
||||||
* we load a YOLO model (Darknet) by specifying the configuration and
|
* we load a YOLO model (Darknet) by specifying the configuration and
|
||||||
* weights files. For each image we run a forward pass and return a
|
* weights files. For each image we run a forward pass and return a
|
||||||
* list of detected class labels.
|
* list of detected class labels.
|
||||||
@@ -34,13 +37,44 @@ import static org.opencv.dnn.Dnn.DNN_TARGET_CPU;
|
|||||||
@Slf4j
|
@Slf4j
|
||||||
public class ObjectDetectionService {
|
public class ObjectDetectionService {
|
||||||
|
|
||||||
private final Net net;
|
private Net net;
|
||||||
private final List<String> classNames;
|
private List<String> classNames;
|
||||||
private final boolean enabled;
|
private boolean enabled;
|
||||||
private int warnCount = 0;
|
private int warnCount = 0;
|
||||||
private static final int MAX_WARNINGS = 5;
|
private static final int MAX_WARNINGS = 5;
|
||||||
|
private static boolean openCvLoaded = false;
|
||||||
|
|
||||||
|
private final String cfgPath;
|
||||||
|
private final String weightsPath;
|
||||||
|
private final String classNamesPath;
|
||||||
|
|
||||||
ObjectDetectionService(String cfgPath, String weightsPath, String classNamesPath) throws IOException {
|
ObjectDetectionService(String cfgPath, String weightsPath, String classNamesPath) throws IOException {
|
||||||
|
this.cfgPath = cfgPath;
|
||||||
|
this.weightsPath = weightsPath;
|
||||||
|
this.classNamesPath = classNamesPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
@PostConstruct
|
||||||
|
void init() {
|
||||||
|
// Load OpenCV native libraries first
|
||||||
|
if (!openCvLoaded) {
|
||||||
|
try {
|
||||||
|
OpenCV.loadLocally();
|
||||||
|
openCvLoaded = true;
|
||||||
|
log.info("✓ OpenCV {} loaded successfully", Core.VERSION);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("⚠️ Object detection disabled: OpenCV native libraries not loaded");
|
||||||
|
enabled = false;
|
||||||
|
net = null;
|
||||||
|
classNames = new ArrayList<>();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
initializeModel();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void initializeModel() {
|
||||||
// Check if model files exist
|
// Check if model files exist
|
||||||
var cfgFile = Paths.get(cfgPath);
|
var cfgFile = Paths.get(cfgPath);
|
||||||
var weightsFile = Paths.get(weightsPath);
|
var weightsFile = Paths.get(weightsPath);
|
||||||
@@ -53,44 +87,48 @@ public class ObjectDetectionService {
|
|||||||
log.info(" - {}", weightsPath);
|
log.info(" - {}", weightsPath);
|
||||||
log.info(" - {}", classNamesPath);
|
log.info(" - {}", classNamesPath);
|
||||||
log.info(" Scraper will continue without image analysis.");
|
log.info(" Scraper will continue without image analysis.");
|
||||||
this.enabled = false;
|
enabled = false;
|
||||||
this.net = null;
|
net = null;
|
||||||
this.classNames = new ArrayList<>();
|
classNames = new ArrayList<>();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Load network
|
// Load network
|
||||||
this.net = Dnn.readNetFromDarknet(cfgPath, weightsPath);
|
net = Dnn.readNetFromDarknet(cfgPath, weightsPath);
|
||||||
|
|
||||||
// Try to use GPU/CUDA if available, fallback to CPU
|
// Try to use GPU/CUDA if available, fallback to CPU
|
||||||
try {
|
try {
|
||||||
this.net.setPreferableBackend(Dnn.DNN_BACKEND_CUDA);
|
net.setPreferableBackend(Dnn.DNN_BACKEND_CUDA);
|
||||||
this.net.setPreferableTarget(Dnn.DNN_TARGET_CUDA);
|
net.setPreferableTarget(Dnn.DNN_TARGET_CUDA);
|
||||||
log.info("✓ Object detection enabled with YOLO (CUDA/GPU acceleration)");
|
log.info("✓ Object detection enabled with YOLO (CUDA/GPU acceleration)");
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
// CUDA not available, try Vulkan for AMD GPUs
|
// CUDA not available, try Vulkan for AMD GPUs
|
||||||
try {
|
try {
|
||||||
this.net.setPreferableBackend(Dnn.DNN_BACKEND_VKCOM);
|
net.setPreferableBackend(Dnn.DNN_BACKEND_VKCOM);
|
||||||
this.net.setPreferableTarget(Dnn.DNN_TARGET_VULKAN);
|
net.setPreferableTarget(Dnn.DNN_TARGET_VULKAN);
|
||||||
log.info("✓ Object detection enabled with YOLO (Vulkan/GPU acceleration)");
|
log.info("✓ Object detection enabled with YOLO (Vulkan/GPU acceleration)");
|
||||||
} catch (Exception e2) {
|
} catch (Exception e2) {
|
||||||
// GPU not available, fallback to CPU
|
// GPU not available, fallback to CPU
|
||||||
this.net.setPreferableBackend(DNN_BACKEND_OPENCV);
|
net.setPreferableBackend(DNN_BACKEND_OPENCV);
|
||||||
this.net.setPreferableTarget(DNN_TARGET_CPU);
|
net.setPreferableTarget(DNN_TARGET_CPU);
|
||||||
log.info("✓ Object detection enabled with YOLO (CPU only)");
|
log.info("✓ Object detection enabled with YOLO (CPU only)");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load class names (one per line)
|
// Load class names (one per line)
|
||||||
this.classNames = Files.readAllLines(classNamesFile);
|
classNames = Files.readAllLines(classNamesFile);
|
||||||
this.enabled = true;
|
enabled = true;
|
||||||
} catch (UnsatisfiedLinkError e) {
|
} catch (UnsatisfiedLinkError e) {
|
||||||
System.err.println("⚠️ Object detection disabled: OpenCV native libraries not loaded");
|
log.error("⚠️ Object detection disabled: OpenCV native libraries not loaded", e);
|
||||||
throw new IOException("Failed to initialize object detection: OpenCV native libraries not loaded", e);
|
enabled = false;
|
||||||
|
net = null;
|
||||||
|
classNames = new ArrayList<>();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
System.err.println("⚠️ Object detection disabled: " + e.getMessage());
|
log.error("⚠️ Object detection disabled: " + e.getMessage(), e);
|
||||||
throw new IOException("Failed to initialize object detection", e);
|
enabled = false;
|
||||||
|
net = null;
|
||||||
|
classNames = new ArrayList<>();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
@@ -121,15 +159,15 @@ public class ObjectDetectionService {
|
|||||||
var confThreshold = 0.5f;
|
var confThreshold = 0.5f;
|
||||||
for (var out : outs) {
|
for (var out : outs) {
|
||||||
// YOLO output shape: [num_detections, 85] where 85 = 4 (bbox) + 1 (objectness) + 80 (classes)
|
// YOLO output shape: [num_detections, 85] where 85 = 4 (bbox) + 1 (objectness) + 80 (classes)
|
||||||
int numDetections = out.rows();
|
int numDetections = out.rows();
|
||||||
int numElements = out.cols();
|
int numElements = out.cols();
|
||||||
int expectedLength = 5 + classNames.size();
|
int expectedLength = 5 + classNames.size();
|
||||||
|
|
||||||
if (numElements < expectedLength) {
|
if (numElements < expectedLength) {
|
||||||
// Rate-limit warnings to prevent thread blocking from excessive logging
|
// Rate-limit warnings to prevent thread blocking from excessive logging
|
||||||
if (warnCount < MAX_WARNINGS) {
|
if (warnCount < MAX_WARNINGS) {
|
||||||
log.warn("Output matrix has wrong dimensions: expected {} columns, got {}. Output shape: [{}, {}]",
|
log.warn("Output matrix has wrong dimensions: expected {} columns, got {}. Output shape: [{}, {}]",
|
||||||
expectedLength, numElements, numDetections, numElements);
|
expectedLength, numElements, numDetections, numElements);
|
||||||
warnCount++;
|
warnCount++;
|
||||||
if (warnCount == MAX_WARNINGS) {
|
if (warnCount == MAX_WARNINGS) {
|
||||||
log.warn("Suppressing further dimension warnings (reached {} warnings)", MAX_WARNINGS);
|
log.warn("Suppressing further dimension warnings (reached {} warnings)", MAX_WARNINGS);
|
||||||
@@ -137,27 +175,27 @@ public class ObjectDetectionService {
|
|||||||
}
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
for (var i = 0; i < numDetections; i++) {
|
for (var i = 0; i < numDetections; i++) {
|
||||||
// Get entire row (all 85 elements)
|
// Get entire row (all 85 elements)
|
||||||
var data = new double[numElements];
|
var data = new double[numElements];
|
||||||
for (int j = 0; j < numElements; j++) {
|
for (int j = 0; j < numElements; j++) {
|
||||||
data[j] = out.get(i, j)[0];
|
data[j] = out.get(i, j)[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
// Extract objectness score (index 4) and class scores (index 5+)
|
// Extract objectness score (index 4) and class scores (index 5+)
|
||||||
double objectness = data[4];
|
double objectness = data[4];
|
||||||
if (objectness < confThreshold) {
|
if (objectness < confThreshold) {
|
||||||
continue; // Skip low-confidence detections
|
continue; // Skip low-confidence detections
|
||||||
}
|
}
|
||||||
|
|
||||||
// Extract class scores
|
// Extract class scores
|
||||||
var scores = new double[classNames.size()];
|
var scores = new double[classNames.size()];
|
||||||
System.arraycopy(data, 5, scores, 0, Math.min(scores.length, data.length - 5));
|
System.arraycopy(data, 5, scores, 0, Math.min(scores.length, data.length - 5));
|
||||||
|
|
||||||
var classId = argMax(scores);
|
var classId = argMax(scores);
|
||||||
var confidence = scores[classId] * objectness; // Combine objectness with class confidence
|
var confidence = scores[classId] * objectness; // Combine objectness with class confidence
|
||||||
|
|
||||||
if (confidence > confThreshold) {
|
if (confidence > confThreshold) {
|
||||||
var label = classNames.get(classId);
|
var label = classNames.get(classId);
|
||||||
if (!labels.contains(label)) {
|
if (!labels.contains(label)) {
|
||||||
@@ -166,7 +204,7 @@ public class ObjectDetectionService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Release resources
|
// Release resources
|
||||||
image.release();
|
image.release();
|
||||||
blob.release();
|
blob.release();
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import io.quarkus.scheduler.Scheduled;
|
|||||||
import jakarta.enterprise.context.ApplicationScoped;
|
import jakarta.enterprise.context.ApplicationScoped;
|
||||||
import jakarta.enterprise.event.Observes;
|
import jakarta.enterprise.event.Observes;
|
||||||
import jakarta.inject.Inject;
|
import jakarta.inject.Inject;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
||||||
import org.jboss.logging.Logger;
|
import org.jboss.logging.Logger;
|
||||||
|
|
||||||
@@ -22,24 +23,14 @@ public class QuarkusWorkflowScheduler {
|
|||||||
|
|
||||||
private static final Logger LOG = Logger.getLogger(QuarkusWorkflowScheduler.class);
|
private static final Logger LOG = Logger.getLogger(QuarkusWorkflowScheduler.class);
|
||||||
|
|
||||||
@Inject
|
@Inject DatabaseService db;
|
||||||
DatabaseService db;
|
@Inject NotificationService notifier;
|
||||||
|
@Inject ObjectDetectionService detector;
|
||||||
|
@Inject ImageProcessingService imageProcessor;
|
||||||
|
@Inject LotEnrichmentService enrichmentService;
|
||||||
|
|
||||||
@Inject
|
@ConfigProperty(name = "auction.database.path") String databasePath;
|
||||||
NotificationService notifier;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
ObjectDetectionService detector;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
ImageProcessingService imageProcessor;
|
|
||||||
|
|
||||||
@Inject
|
|
||||||
LotEnrichmentService enrichmentService;
|
|
||||||
|
|
||||||
@ConfigProperty(name = "auction.database.path")
|
|
||||||
String databasePath;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Triggered on application startup to enrich existing lots with bid intelligence
|
* Triggered on application startup to enrich existing lots with bid intelligence
|
||||||
*/
|
*/
|
||||||
@@ -108,41 +99,41 @@ public class QuarkusWorkflowScheduler {
|
|||||||
try {
|
try {
|
||||||
LOG.info("🖼️ [WORKFLOW 2] Processing pending images...");
|
LOG.info("🖼️ [WORKFLOW 2] Processing pending images...");
|
||||||
var start = System.currentTimeMillis();
|
var start = System.currentTimeMillis();
|
||||||
|
|
||||||
// Get images that have been downloaded but need object detection
|
// Get images that have been downloaded but need object detection
|
||||||
var pendingImages = db.getImagesNeedingDetection();
|
var pendingImages = db.getImagesNeedingDetection();
|
||||||
|
|
||||||
if (pendingImages.isEmpty()) {
|
if (pendingImages.isEmpty()) {
|
||||||
LOG.info(" → No pending images to process");
|
LOG.info(" → No pending images to process");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Limit batch size to prevent thread blocking (max 100 images per run)
|
// Limit batch size to prevent thread blocking (max 100 images per run)
|
||||||
final int MAX_BATCH_SIZE = 100;
|
final int MAX_BATCH_SIZE = 100;
|
||||||
int totalPending = pendingImages.size();
|
int totalPending = pendingImages.size();
|
||||||
if (totalPending > MAX_BATCH_SIZE) {
|
if (totalPending > MAX_BATCH_SIZE) {
|
||||||
LOG.infof(" → Found %d pending images, processing first %d (batch limit)",
|
LOG.infof(" → Found %d pending images, processing first %d (batch limit)",
|
||||||
totalPending, MAX_BATCH_SIZE);
|
totalPending, MAX_BATCH_SIZE);
|
||||||
pendingImages = pendingImages.subList(0, MAX_BATCH_SIZE);
|
pendingImages = pendingImages.subList(0, MAX_BATCH_SIZE);
|
||||||
} else {
|
} else {
|
||||||
LOG.infof(" → Processing %d images", totalPending);
|
LOG.infof(" → Processing %d images", totalPending);
|
||||||
}
|
}
|
||||||
|
|
||||||
var processed = 0;
|
var processed = 0;
|
||||||
var detected = 0;
|
var detected = 0;
|
||||||
var failed = 0;
|
var failed = 0;
|
||||||
|
|
||||||
for (var image : pendingImages) {
|
for (var image : pendingImages) {
|
||||||
try {
|
try {
|
||||||
// Run object detection on already-downloaded image
|
// Run object detection on already-downloaded image
|
||||||
if (imageProcessor.processImage(image.id(), image.filePath(), image.lotId())) {
|
if (imageProcessor.processImage(image.id(), image.filePath(), image.lotId())) {
|
||||||
processed++;
|
processed++;
|
||||||
|
|
||||||
// Check if objects were detected
|
// Check if objects were detected
|
||||||
var labels = db.getImageLabels(image.id());
|
var labels = db.getImageLabels(image.id());
|
||||||
if (labels != null && !labels.isEmpty()) {
|
if (labels != null && !labels.isEmpty()) {
|
||||||
detected++;
|
detected++;
|
||||||
|
|
||||||
// Send notification for interesting detections
|
// Send notification for interesting detections
|
||||||
if (labels.size() >= 3) {
|
if (labels.size() >= 3) {
|
||||||
notifier.sendNotification(
|
notifier.sendNotification(
|
||||||
@@ -151,16 +142,16 @@ public class QuarkusWorkflowScheduler {
|
|||||||
String.join(", ", labels)),
|
String.join(", ", labels)),
|
||||||
"Objects Detected",
|
"Objects Detected",
|
||||||
0
|
0
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
failed++;
|
failed++;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Rate limiting (lighter since no network I/O)
|
// Rate limiting (lighter since no network I/O)
|
||||||
Thread.sleep(100);
|
Thread.sleep(100);
|
||||||
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
failed++;
|
failed++;
|
||||||
LOG.warnf(" ⚠️ Failed to process image: %s", e.getMessage());
|
LOG.warnf(" ⚠️ Failed to process image: %s", e.getMessage());
|
||||||
@@ -170,7 +161,7 @@ public class QuarkusWorkflowScheduler {
|
|||||||
var duration = System.currentTimeMillis() - start;
|
var duration = System.currentTimeMillis() - start;
|
||||||
LOG.infof(" ✓ Processed %d/%d images, detected objects in %d, failed %d (%.1fs)",
|
LOG.infof(" ✓ Processed %d/%d images, detected objects in %d, failed %d (%.1fs)",
|
||||||
processed, totalPending, detected, failed, duration / 1000.0);
|
processed, totalPending, detected, failed, duration / 1000.0);
|
||||||
|
|
||||||
if (totalPending > MAX_BATCH_SIZE) {
|
if (totalPending > MAX_BATCH_SIZE) {
|
||||||
LOG.infof(" → %d images remaining for next run", totalPending - MAX_BATCH_SIZE);
|
LOG.infof(" → %d images remaining for next run", totalPending - MAX_BATCH_SIZE);
|
||||||
}
|
}
|
||||||
@@ -238,7 +229,7 @@ public class QuarkusWorkflowScheduler {
|
|||||||
lot.manufacturer(), lot.type(), lot.year(), lot.category(),
|
lot.manufacturer(), lot.type(), lot.year(), lot.category(),
|
||||||
lot.currentBid(), lot.currency(), lot.url(),
|
lot.currentBid(), lot.currency(), lot.url(),
|
||||||
lot.closingTime(), true
|
lot.closingTime(), true
|
||||||
);
|
);
|
||||||
db.updateLotNotificationFlags(updated);
|
db.updateLotNotificationFlags(updated);
|
||||||
|
|
||||||
alertsSent++;
|
alertsSent++;
|
||||||
|
|||||||
@@ -142,29 +142,15 @@ public class RateLimitedHttpClient {
|
|||||||
* Determines max requests per second for a given host.
|
* Determines max requests per second for a given host.
|
||||||
*/
|
*/
|
||||||
private int getMaxRequestsPerSecond(String host) {
|
private int getMaxRequestsPerSecond(String host) {
|
||||||
if (host.contains("troostwijk")) {
|
return host.contains("troostwijk") ? troostwijkMaxRequestsPerSecond : defaultMaxRequestsPerSecond;
|
||||||
return troostwijkMaxRequestsPerSecond;
|
|
||||||
}
|
|
||||||
return defaultMaxRequestsPerSecond;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Extracts host from URI (e.g., "api.troostwijkauctions.com").
|
|
||||||
*/
|
|
||||||
private String extractHost(URI uri) {
|
private String extractHost(URI uri) {
|
||||||
return uri.getHost() != null ? uri.getHost() : uri.toString();
|
return uri.getHost() != null ? uri.getHost() : uri.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Gets statistics for all hosts.
|
|
||||||
*/
|
|
||||||
public Map<String, RequestStats> getAllStats() {
|
public Map<String, RequestStats> getAllStats() {
|
||||||
return Map.copyOf(requestStats);
|
return Map.copyOf(requestStats);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Gets statistics for a specific host.
|
|
||||||
*/
|
|
||||||
public RequestStats getStats(String host) {
|
public RequestStats getStats(String host) {
|
||||||
return requestStats.get(host);
|
return requestStats.get(host);
|
||||||
}
|
}
|
||||||
@@ -218,10 +204,7 @@ public class RateLimitedHttpClient {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
public static final class RequestStats {
|
||||||
* Statistics tracker for HTTP requests per host.
|
|
||||||
*/
|
|
||||||
public static class RequestStats {
|
|
||||||
|
|
||||||
private final String host;
|
private final String host;
|
||||||
private final AtomicLong totalRequests = new AtomicLong(0);
|
private final AtomicLong totalRequests = new AtomicLong(0);
|
||||||
@@ -234,25 +217,16 @@ public class RateLimitedHttpClient {
|
|||||||
this.host = host;
|
this.host = host;
|
||||||
}
|
}
|
||||||
|
|
||||||
void incrementTotal() {
|
void incrementTotal() { totalRequests.incrementAndGet(); }
|
||||||
totalRequests.incrementAndGet();
|
|
||||||
}
|
|
||||||
|
|
||||||
void recordSuccess(long durationMs) {
|
void recordSuccess(long durationMs) {
|
||||||
successfulRequests.incrementAndGet();
|
successfulRequests.incrementAndGet();
|
||||||
totalDurationMs.addAndGet(durationMs);
|
totalDurationMs.addAndGet(durationMs);
|
||||||
}
|
}
|
||||||
|
|
||||||
void incrementFailed() {
|
void incrementFailed() { failedRequests.incrementAndGet(); }
|
||||||
failedRequests.incrementAndGet();
|
void incrementRateLimited() { rateLimitedRequests.incrementAndGet(); }
|
||||||
}
|
public String getHost() { return host; }
|
||||||
|
|
||||||
void incrementRateLimited() {
|
|
||||||
rateLimitedRequests.incrementAndGet();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Getters
|
|
||||||
public String getHost() { return host; }
|
|
||||||
public long getTotalRequests() { return totalRequests.get(); }
|
public long getTotalRequests() { return totalRequests.get(); }
|
||||||
public long getSuccessfulRequests() { return successfulRequests.get(); }
|
public long getSuccessfulRequests() { return successfulRequests.get(); }
|
||||||
public long getFailedRequests() { return failedRequests.get(); }
|
public long getFailedRequests() { return failedRequests.get(); }
|
||||||
|
|||||||
@@ -63,13 +63,15 @@ public class ScraperDataAdapter {
|
|||||||
lotIdStr, // Store full displayId for GraphQL queries
|
lotIdStr, // Store full displayId for GraphQL queries
|
||||||
rs.getString("title"),
|
rs.getString("title"),
|
||||||
getStringOrDefault(rs, "description", ""),
|
getStringOrDefault(rs, "description", ""),
|
||||||
"", "", 0,
|
getStringOrDefault(rs, "manufacturer", ""),
|
||||||
|
getStringOrDefault(rs, "type", ""),
|
||||||
|
getIntOrDefault(rs, "year", 0),
|
||||||
getStringOrDefault(rs, "category", ""),
|
getStringOrDefault(rs, "category", ""),
|
||||||
bid,
|
bid,
|
||||||
currency,
|
currency,
|
||||||
rs.getString("url"),
|
rs.getString("url"),
|
||||||
closing,
|
closing,
|
||||||
false,
|
getBooleanOrDefault(rs, "closing_notified", false),
|
||||||
// New intelligence fields - set to null for now
|
// New intelligence fields - set to null for now
|
||||||
null, null, null, null, null, null, null, null,
|
null, null, null, null, null, null, null, null,
|
||||||
null, null, null, null, null, null, null,
|
null, null, null, null, null, null, null,
|
||||||
@@ -166,4 +168,9 @@ public class ScraperDataAdapter {
|
|||||||
var v = rs.getInt(col);
|
var v = rs.getInt(col);
|
||||||
return rs.wasNull() ? def : v;
|
return rs.wasNull() ? def : v;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static boolean getBooleanOrDefault(ResultSet rs, String col, boolean def) throws SQLException {
|
||||||
|
var v = rs.getInt(col);
|
||||||
|
return rs.wasNull() ? def : v != 0;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ import jakarta.ws.rs.Path;
|
|||||||
import jakarta.ws.rs.Produces;
|
import jakarta.ws.rs.Produces;
|
||||||
import jakarta.ws.rs.core.MediaType;
|
import jakarta.ws.rs.core.MediaType;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import nu.pattern.OpenCV;
|
||||||
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
import org.eclipse.microprofile.config.inject.ConfigProperty;
|
||||||
|
import org.opencv.core.Core;
|
||||||
import java.time.Instant;
|
import java.time.Instant;
|
||||||
import java.time.ZoneId;
|
import java.time.ZoneId;
|
||||||
import java.time.format.DateTimeFormatter;
|
import java.time.format.DateTimeFormatter;
|
||||||
@@ -19,18 +21,11 @@ public class StatusResource {
|
|||||||
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z")
|
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z")
|
||||||
.withZone(ZoneId.systemDefault());
|
.withZone(ZoneId.systemDefault());
|
||||||
|
|
||||||
@ConfigProperty(name = "application.version", defaultValue = "1.0-SNAPSHOT")
|
@ConfigProperty(name = "application.version", defaultValue = "1.0-SNAPSHOT") String appVersion;
|
||||||
String appVersion;
|
@ConfigProperty(name = "application.groupId") String groupId;
|
||||||
@ConfigProperty(name = "application.groupId")
|
@ConfigProperty(name = "application.artifactId") String artifactId;
|
||||||
String groupId;
|
@ConfigProperty(name = "application.version") String version;
|
||||||
|
|
||||||
@ConfigProperty(name = "application.artifactId")
|
|
||||||
String artifactId;
|
|
||||||
|
|
||||||
@ConfigProperty(name = "application.version")
|
|
||||||
String version;
|
|
||||||
|
|
||||||
// Java 16+ Record for structured response
|
|
||||||
public record StatusResponse(
|
public record StatusResponse(
|
||||||
String groupId,
|
String groupId,
|
||||||
String artifactId,
|
String artifactId,
|
||||||
@@ -47,8 +42,6 @@ public class StatusResource {
|
|||||||
@Path("/status")
|
@Path("/status")
|
||||||
@Produces(MediaType.APPLICATION_JSON)
|
@Produces(MediaType.APPLICATION_JSON)
|
||||||
public StatusResponse getStatus() {
|
public StatusResponse getStatus() {
|
||||||
log.info("Status endpoint called");
|
|
||||||
|
|
||||||
return new StatusResponse(groupId, artifactId, version,
|
return new StatusResponse(groupId, artifactId, version,
|
||||||
"running",
|
"running",
|
||||||
FORMATTER.format(Instant.now()),
|
FORMATTER.format(Instant.now()),
|
||||||
@@ -63,8 +56,6 @@ public class StatusResource {
|
|||||||
@Path("/hello")
|
@Path("/hello")
|
||||||
@Produces(MediaType.APPLICATION_JSON)
|
@Produces(MediaType.APPLICATION_JSON)
|
||||||
public Map<String, String> sayHello() {
|
public Map<String, String> sayHello() {
|
||||||
log.info("hello endpoint called");
|
|
||||||
|
|
||||||
return Map.of(
|
return Map.of(
|
||||||
"message", "Hello from Scrape-UI!",
|
"message", "Hello from Scrape-UI!",
|
||||||
"timestamp", FORMATTER.format(Instant.now()),
|
"timestamp", FORMATTER.format(Instant.now()),
|
||||||
@@ -74,11 +65,10 @@ public class StatusResource {
|
|||||||
|
|
||||||
private String getOpenCvVersion() {
|
private String getOpenCvVersion() {
|
||||||
try {
|
try {
|
||||||
// Load OpenCV if not already loaded (safe to call multiple times)
|
// OpenCV is already loaded by AuctionMonitorProducer
|
||||||
nu.pattern.OpenCV.loadLocally();
|
return Core.VERSION;
|
||||||
return org.opencv.core.Core.VERSION;
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
return "4.9.0 (default)";
|
return "Not loaded";
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -44,15 +44,15 @@ public class TroostwijkGraphQLClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
String query = buildLotQuery();
|
var query = buildLotQuery();
|
||||||
String variables = buildVariables(displayId);
|
var variables = buildVariables(displayId);
|
||||||
|
|
||||||
// Proper GraphQL request format with query and variables
|
// Proper GraphQL request format with query and variables
|
||||||
String requestBody = String.format(
|
var requestBody = String.format(
|
||||||
"{\"query\":\"%s\",\"variables\":%s}",
|
"{\"query\":\"%s\",\"variables\":%s}",
|
||||||
escapeJson(query),
|
escapeJson(query),
|
||||||
variables
|
variables
|
||||||
);
|
);
|
||||||
|
|
||||||
var request = java.net.http.HttpRequest.newBuilder()
|
var request = java.net.http.HttpRequest.newBuilder()
|
||||||
.uri(java.net.URI.create(GRAPHQL_ENDPOINT))
|
.uri(java.net.URI.create(GRAPHQL_ENDPOINT))
|
||||||
@@ -87,15 +87,15 @@ public class TroostwijkGraphQLClient {
|
|||||||
List<LotIntelligence> results = new ArrayList<>();
|
List<LotIntelligence> results = new ArrayList<>();
|
||||||
|
|
||||||
// Split into batches of 50 to avoid query size limits
|
// Split into batches of 50 to avoid query size limits
|
||||||
int batchSize = 50;
|
var batchSize = 50;
|
||||||
for (int i = 0; i < lotIds.size(); i += batchSize) {
|
for (var i = 0; i < lotIds.size(); i += batchSize) {
|
||||||
int end = Math.min(i + batchSize, lotIds.size());
|
var end = Math.min(i + batchSize, lotIds.size());
|
||||||
List<Long> batch = lotIds.subList(i, end);
|
var batch = lotIds.subList(i, end);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
String query = buildBatchLotQuery(batch);
|
var query = buildBatchLotQuery(batch);
|
||||||
String requestBody = String.format("{\"query\":\"%s\"}",
|
var requestBody = String.format("{\"query\":\"%s\"}",
|
||||||
escapeJson(query));
|
escapeJson(query));
|
||||||
|
|
||||||
var request = java.net.http.HttpRequest.newBuilder()
|
var request = java.net.http.HttpRequest.newBuilder()
|
||||||
.uri(java.net.URI.create(GRAPHQL_ENDPOINT))
|
.uri(java.net.URI.create(GRAPHQL_ENDPOINT))
|
||||||
@@ -162,9 +162,9 @@ public class TroostwijkGraphQLClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private String buildBatchLotQuery(List<Long> lotIds) {
|
private String buildBatchLotQuery(List<Long> lotIds) {
|
||||||
StringBuilder query = new StringBuilder("query {");
|
var query = new StringBuilder("query {");
|
||||||
|
|
||||||
for (int i = 0; i < lotIds.size(); i++) {
|
for (var i = 0; i < lotIds.size(); i++) {
|
||||||
query.append(String.format("""
|
query.append(String.format("""
|
||||||
lot%d: lot(id: %d) {
|
lot%d: lot(id: %d) {
|
||||||
id
|
id
|
||||||
@@ -196,9 +196,9 @@ public class TroostwijkGraphQLClient {
|
|||||||
log.debug("GraphQL API returned HTML instead of JSON - likely auth required or wrong endpoint");
|
log.debug("GraphQL API returned HTML instead of JSON - likely auth required or wrong endpoint");
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
JsonNode root = objectMapper.readTree(json);
|
var root = objectMapper.readTree(json);
|
||||||
JsonNode lotNode = root.path("data").path("lotDetails");
|
var lotNode = root.path("data").path("lotDetails");
|
||||||
|
|
||||||
if (lotNode.isMissingNode()) {
|
if (lotNode.isMissingNode()) {
|
||||||
log.debug("No lotDetails in GraphQL response");
|
log.debug("No lotDetails in GraphQL response");
|
||||||
@@ -206,19 +206,19 @@ public class TroostwijkGraphQLClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Extract location from nested object
|
// Extract location from nested object
|
||||||
JsonNode locationNode = lotNode.path("location");
|
var locationNode = lotNode.path("location");
|
||||||
String city = locationNode.isMissingNode() ? null : getStringOrNull(locationNode, "city");
|
var city = locationNode.isMissingNode() ? null : getStringOrNull(locationNode, "city");
|
||||||
String countryCode = locationNode.isMissingNode() ? null : getStringOrNull(locationNode, "country");
|
var countryCode = locationNode.isMissingNode() ? null : getStringOrNull(locationNode, "country");
|
||||||
|
|
||||||
// Extract bids count from nested biddingStatistics
|
// Extract bids count from nested biddingStatistics
|
||||||
JsonNode statsNode = lotNode.path("biddingStatistics");
|
var statsNode = lotNode.path("biddingStatistics");
|
||||||
Integer bidsCount = statsNode.isMissingNode() ? null : getIntOrNull(statsNode, "numberOfBids");
|
var bidsCount = statsNode.isMissingNode() ? null : getIntOrNull(statsNode, "numberOfBids");
|
||||||
|
|
||||||
// Convert cents to euros for estimates
|
// Convert cents to euros for estimates
|
||||||
Long estimatedMinCents = getLongOrNull(lotNode, "estimatedValueInCentsMin");
|
var estimatedMinCents = getLongOrNull(lotNode, "estimatedValueInCentsMin");
|
||||||
Long estimatedMaxCents = getLongOrNull(lotNode, "estimatedValueInCentsMax");
|
var estimatedMaxCents = getLongOrNull(lotNode, "estimatedValueInCentsMax");
|
||||||
Double estimatedMin = estimatedMinCents != null ? estimatedMinCents.doubleValue() : null;
|
var estimatedMin = estimatedMinCents != null ? estimatedMinCents.doubleValue() : null;
|
||||||
Double estimatedMax = estimatedMaxCents != null ? estimatedMaxCents.doubleValue() : null;
|
var estimatedMax = estimatedMaxCents != null ? estimatedMaxCents.doubleValue() : null;
|
||||||
|
|
||||||
return new LotIntelligence(
|
return new LotIntelligence(
|
||||||
lotId,
|
lotId,
|
||||||
@@ -257,11 +257,11 @@ public class TroostwijkGraphQLClient {
|
|||||||
List<LotIntelligence> results = new ArrayList<>();
|
List<LotIntelligence> results = new ArrayList<>();
|
||||||
|
|
||||||
try {
|
try {
|
||||||
JsonNode root = objectMapper.readTree(json);
|
var root = objectMapper.readTree(json);
|
||||||
JsonNode data = root.path("data");
|
var data = root.path("data");
|
||||||
|
|
||||||
for (int i = 0; i < lotIds.size(); i++) {
|
for (var i = 0; i < lotIds.size(); i++) {
|
||||||
JsonNode lotNode = data.path("lot" + i);
|
var lotNode = data.path("lot" + i);
|
||||||
if (!lotNode.isMissingNode()) {
|
if (!lotNode.isMissingNode()) {
|
||||||
var intelligence = parseLotIntelligenceFromNode(lotNode, lotIds.get(i));
|
var intelligence = parseLotIntelligenceFromNode(lotNode, lotIds.get(i));
|
||||||
if (intelligence != null) {
|
if (intelligence != null) {
|
||||||
@@ -313,17 +313,17 @@ public class TroostwijkGraphQLClient {
|
|||||||
|
|
||||||
private Double calculateBidVelocity(JsonNode lotNode) {
|
private Double calculateBidVelocity(JsonNode lotNode) {
|
||||||
try {
|
try {
|
||||||
Integer bidsCount = getIntOrNull(lotNode, "bidsCount");
|
var bidsCount = getIntOrNull(lotNode, "bidsCount");
|
||||||
String firstBidStr = getStringOrNull(lotNode, "firstBidTime");
|
var firstBidStr = getStringOrNull(lotNode, "firstBidTime");
|
||||||
|
|
||||||
if (bidsCount == null || firstBidStr == null || bidsCount == 0) {
|
if (bidsCount == null || firstBidStr == null || bidsCount == 0) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
LocalDateTime firstBid = parseDateTime(firstBidStr);
|
var firstBid = parseDateTime(firstBidStr);
|
||||||
if (firstBid == null) return null;
|
if (firstBid == null) return null;
|
||||||
|
|
||||||
long hoursElapsed = java.time.Duration.between(firstBid, LocalDateTime.now()).toHours();
|
var hoursElapsed = java.time.Duration.between(firstBid, LocalDateTime.now()).toHours();
|
||||||
if (hoursElapsed == 0) return (double) bidsCount;
|
if (hoursElapsed == 0) return (double) bidsCount;
|
||||||
|
|
||||||
return (double) bidsCount / hoursElapsed;
|
return (double) bidsCount / hoursElapsed;
|
||||||
@@ -352,27 +352,27 @@ public class TroostwijkGraphQLClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private Integer getIntOrNull(JsonNode node, String field) {
|
private Integer getIntOrNull(JsonNode node, String field) {
|
||||||
JsonNode fieldNode = node.path(field);
|
var fieldNode = node.path(field);
|
||||||
return fieldNode.isNumber() ? fieldNode.asInt() : null;
|
return fieldNode.isNumber() ? fieldNode.asInt() : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private Long getLongOrNull(JsonNode node, String field) {
|
private Long getLongOrNull(JsonNode node, String field) {
|
||||||
JsonNode fieldNode = node.path(field);
|
var fieldNode = node.path(field);
|
||||||
return fieldNode.isNumber() ? fieldNode.asLong() : null;
|
return fieldNode.isNumber() ? fieldNode.asLong() : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private Double getDoubleOrNull(JsonNode node, String field) {
|
private Double getDoubleOrNull(JsonNode node, String field) {
|
||||||
JsonNode fieldNode = node.path(field);
|
var fieldNode = node.path(field);
|
||||||
return fieldNode.isNumber() ? fieldNode.asDouble() : null;
|
return fieldNode.isNumber() ? fieldNode.asDouble() : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private String getStringOrNull(JsonNode node, String field) {
|
private String getStringOrNull(JsonNode node, String field) {
|
||||||
JsonNode fieldNode = node.path(field);
|
var fieldNode = node.path(field);
|
||||||
return fieldNode.isTextual() ? fieldNode.asText() : null;
|
return fieldNode.isTextual() ? fieldNode.asText() : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private Boolean getBooleanOrNull(JsonNode node, String field) {
|
private Boolean getBooleanOrNull(JsonNode node, String field) {
|
||||||
JsonNode fieldNode = node.path(field);
|
var fieldNode = node.path(field);
|
||||||
return fieldNode.isBoolean() ? fieldNode.asBoolean() : null;
|
return fieldNode.isBoolean() ? fieldNode.asBoolean() : null;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -60,25 +60,21 @@ public class TroostwijkMonitor {
|
|||||||
for (var lot : activeLots) {
|
for (var lot : activeLots) {
|
||||||
checkAndUpdateLot(lot);
|
checkAndUpdateLot(lot);
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
log.error("Error during scheduled monitoring", e);
|
log.error("Error during scheduled monitoring", e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private void checkAndUpdateLot(Lot lot) {
|
private void checkAndUpdateLot(Lot lot) {
|
||||||
refreshLotBid(lot);
|
refreshLotBid(lot);
|
||||||
|
|
||||||
var minutesLeft = lot.minutesUntilClose();
|
var minutesLeft = lot.minutesUntilClose();
|
||||||
if (minutesLeft < 30) {
|
if (minutesLeft < 30) {
|
||||||
if (minutesLeft <= 5 && !lot.closingNotified()) {
|
if (minutesLeft <= 5 && !lot.closingNotified()) {
|
||||||
notifier.sendNotification(
|
notifier.sendNotification(
|
||||||
"Kavel " + lot.lotId() + " sluit binnen " + minutesLeft + " min.",
|
"Kavel " + lot.lotId() + " sluit binnen " + minutesLeft + " min.",
|
||||||
"Lot nearing closure", 1);
|
"Lot nearing closure", 1);
|
||||||
try {
|
db.updateLotNotificationFlags(lot.withClosingNotified(true));
|
||||||
db.updateLotNotificationFlags(lot.withClosingNotified(true));
|
|
||||||
} catch (SQLException e) {
|
|
||||||
throw new RuntimeException(e);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
scheduler.schedule(() -> refreshLotBid(lot), 5, TimeUnit.MINUTES);
|
scheduler.schedule(() -> refreshLotBid(lot), 5, TimeUnit.MINUTES);
|
||||||
}
|
}
|
||||||
@@ -109,12 +105,12 @@ public class TroostwijkMonitor {
|
|||||||
notifier.sendNotification(msg, "Kavel bieding update", 0);
|
notifier.sendNotification(msg, "Kavel bieding update", 0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} catch (IOException | InterruptedException | SQLException e) {
|
} catch (IOException | InterruptedException e) {
|
||||||
log.warn("Failed to refresh bid for lot {}", lot.lotId(), e);
|
log.warn("Failed to refresh bid for lot {}", lot.lotId(), e);
|
||||||
if (e instanceof InterruptedException) Thread.currentThread().interrupt();
|
if (e instanceof InterruptedException) Thread.currentThread().interrupt();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public void printDatabaseStats() {
|
public void printDatabaseStats() {
|
||||||
try {
|
try {
|
||||||
var allLots = db.getAllLots();
|
var allLots = db.getAllLots();
|
||||||
@@ -123,9 +119,9 @@ public class TroostwijkMonitor {
|
|||||||
allLots.size(), imageCount);
|
allLots.size(), imageCount);
|
||||||
if (!allLots.isEmpty()) {
|
if (!allLots.isEmpty()) {
|
||||||
var sum = allLots.stream().mapToDouble(Lot::currentBid).sum();
|
var sum = allLots.stream().mapToDouble(Lot::currentBid).sum();
|
||||||
log.info("Total current bids: €{:.2f}", sum);
|
log.info("Total current bids: €{}", String.format("%.2f", sum));
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
log.warn("Could not retrieve database stats", e);
|
log.warn("Could not retrieve database stats", e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -50,25 +50,25 @@ public class ValuationAnalyticsResource {
|
|||||||
public Response calculateValuation(ValuationRequest request) {
|
public Response calculateValuation(ValuationRequest request) {
|
||||||
try {
|
try {
|
||||||
LOG.infof("Valuation request for lot: %s", request.lotId);
|
LOG.infof("Valuation request for lot: %s", request.lotId);
|
||||||
long startTime = System.currentTimeMillis();
|
var startTime = System.currentTimeMillis();
|
||||||
|
|
||||||
// Step 1: Fetch comparable sales from database
|
// Step 1: Fetch comparable sales from database
|
||||||
List<ComparableLot> comparables = fetchComparables(request);
|
var comparables = fetchComparables(request);
|
||||||
|
|
||||||
// Step 2: Calculate Fair Market Value (FMV)
|
// Step 2: Calculate Fair Market Value (FMV)
|
||||||
FairMarketValue fmv = calculateFairMarketValue(request, comparables);
|
var fmv = calculateFairMarketValue(request, comparables);
|
||||||
|
|
||||||
// Step 3: Calculate undervaluation score
|
// Step 3: Calculate undervaluation score
|
||||||
double undervaluationScore = calculateUndervaluationScore(request, fmv.value);
|
var undervaluationScore = calculateUndervaluationScore(request, fmv.value);
|
||||||
|
|
||||||
// Step 4: Predict final price
|
// Step 4: Predict final price
|
||||||
PricePrediction prediction = calculateFinalPrice(request, fmv.value);
|
var prediction = calculateFinalPrice(request, fmv.value);
|
||||||
|
|
||||||
// Step 5: Generate bidding strategy
|
// Step 5: Generate bidding strategy
|
||||||
BiddingStrategy strategy = generateBiddingStrategy(request, fmv, prediction);
|
var strategy = generateBiddingStrategy(request, fmv, prediction);
|
||||||
|
|
||||||
// Step 6: Compile response
|
// Step 6: Compile response
|
||||||
ValuationResponse response = new ValuationResponse();
|
var response = new ValuationResponse();
|
||||||
response.lotId = request.lotId;
|
response.lotId = request.lotId;
|
||||||
response.timestamp = LocalDateTime.now().toString();
|
response.timestamp = LocalDateTime.now().toString();
|
||||||
response.fairMarketValue = fmv;
|
response.fairMarketValue = fmv;
|
||||||
@@ -76,8 +76,8 @@ public class ValuationAnalyticsResource {
|
|||||||
response.pricePrediction = prediction;
|
response.pricePrediction = prediction;
|
||||||
response.biddingStrategy = strategy;
|
response.biddingStrategy = strategy;
|
||||||
response.parameters = request;
|
response.parameters = request;
|
||||||
|
|
||||||
long duration = System.currentTimeMillis() - startTime;
|
var duration = System.currentTimeMillis() - startTime;
|
||||||
LOG.infof("Valuation completed in %d ms", duration);
|
LOG.infof("Valuation completed in %d ms", duration);
|
||||||
|
|
||||||
return Response.ok(response).build();
|
return Response.ok(response).build();
|
||||||
@@ -115,24 +115,24 @@ public class ValuationAnalyticsResource {
|
|||||||
* Where weights are exponential/logistic functions of similarity
|
* Where weights are exponential/logistic functions of similarity
|
||||||
*/
|
*/
|
||||||
private FairMarketValue calculateFairMarketValue(ValuationRequest req, List<ComparableLot> comparables) {
|
private FairMarketValue calculateFairMarketValue(ValuationRequest req, List<ComparableLot> comparables) {
|
||||||
double weightedSum = 0.0;
|
var weightedSum = 0.0;
|
||||||
double weightSum = 0.0;
|
var weightSum = 0.0;
|
||||||
List<WeightedComparable> weightedComps = new ArrayList<>();
|
List<WeightedComparable> weightedComps = new ArrayList<>();
|
||||||
|
|
||||||
for (ComparableLot comp : comparables) {
|
for (var comp : comparables) {
|
||||||
// Condition weight: ω_c = exp(-λ_c · |C_target - C_i|)
|
// Condition weight: ω_c = exp(-λ_c · |C_target - C_i|)
|
||||||
double omegaC = Math.exp(-0.693 * Math.abs(req.conditionScore - comp.conditionScore));
|
var omegaC = Math.exp(-0.693 * Math.abs(req.conditionScore - comp.conditionScore));
|
||||||
|
|
||||||
// Time weight: ω_t = exp(-λ_t · |T_target - T_i|)
|
// Time weight: ω_t = exp(-λ_t · |T_target - T_i|)
|
||||||
double omegaT = Math.exp(-0.048 * Math.abs(req.manufacturingYear - comp.manufacturingYear));
|
var omegaT = Math.exp(-0.048 * Math.abs(req.manufacturingYear - comp.manufacturingYear));
|
||||||
|
|
||||||
// Provenance weight: ω_p = 1 + δ_p · (P_target - P_i)
|
// Provenance weight: ω_p = 1 + δ_p · (P_target - P_i)
|
||||||
double omegaP = 1 + 0.15 * ((req.provenanceDocs > 0 ? 1 : 0) - comp.hasProvenance);
|
var omegaP = 1 + 0.15 * ((req.provenanceDocs > 0 ? 1 : 0) - comp.hasProvenance);
|
||||||
|
|
||||||
// Historical weight: ω_h = 1 / (1 + e^(-kh · (D_i - D_median)))
|
// Historical weight: ω_h = 1 / (1 + e^(-kh · (D_i - D_median)))
|
||||||
double omegaH = 1.0 / (1 + Math.exp(-0.01 * (comp.daysAgo - 40)));
|
var omegaH = 1.0 / (1 + Math.exp(-0.01 * (comp.daysAgo - 40)));
|
||||||
|
|
||||||
double totalWeight = omegaC * omegaT * omegaP * omegaH;
|
var totalWeight = omegaC * omegaT * omegaP * omegaH;
|
||||||
|
|
||||||
weightedSum += comp.finalPrice * totalWeight;
|
weightedSum += comp.finalPrice * totalWeight;
|
||||||
weightSum += totalWeight;
|
weightSum += totalWeight;
|
||||||
@@ -140,20 +140,20 @@ public class ValuationAnalyticsResource {
|
|||||||
// Store for transparency
|
// Store for transparency
|
||||||
weightedComps.add(new WeightedComparable(comp, totalWeight, omegaC, omegaT, omegaP, omegaH));
|
weightedComps.add(new WeightedComparable(comp, totalWeight, omegaC, omegaT, omegaP, omegaH));
|
||||||
}
|
}
|
||||||
|
|
||||||
double baseFMV = weightSum > 0 ? weightedSum / weightSum : (req.estimatedMin + req.estimatedMax) / 2;
|
var baseFMV = weightSum > 0 ? weightedSum / weightSum : (req.estimatedMin + req.estimatedMax) / 2;
|
||||||
|
|
||||||
// Apply condition multiplier: M_cond = exp(α_c · √C_target - β_c)
|
// Apply condition multiplier: M_cond = exp(α_c · √C_target - β_c)
|
||||||
double conditionMultiplier = Math.exp(0.15 * Math.sqrt(req.conditionScore) - 0.40);
|
var conditionMultiplier = Math.exp(0.15 * Math.sqrt(req.conditionScore) - 0.40);
|
||||||
baseFMV *= conditionMultiplier;
|
baseFMV *= conditionMultiplier;
|
||||||
|
|
||||||
// Apply provenance premium: Δ_prov = V_base · (η_0 + η_1 · ln(1 + N_docs))
|
// Apply provenance premium: Δ_prov = V_base · (η_0 + η_1 · ln(1 + N_docs))
|
||||||
if (req.provenanceDocs > 0) {
|
if (req.provenanceDocs > 0) {
|
||||||
double provenancePremium = 0.08 + 0.035 * Math.log(1 + req.provenanceDocs);
|
var provenancePremium = 0.08 + 0.035 * Math.log(1 + req.provenanceDocs);
|
||||||
baseFMV *= (1 + provenancePremium);
|
baseFMV *= (1 + provenancePremium);
|
||||||
}
|
}
|
||||||
|
|
||||||
FairMarketValue fmv = new FairMarketValue();
|
var fmv = new FairMarketValue();
|
||||||
fmv.value = Math.round(baseFMV * 100.0) / 100.0;
|
fmv.value = Math.round(baseFMV * 100.0) / 100.0;
|
||||||
fmv.conditionMultiplier = Math.round(conditionMultiplier * 1000.0) / 1000.0;
|
fmv.conditionMultiplier = Math.round(conditionMultiplier * 1000.0) / 1000.0;
|
||||||
fmv.provenancePremium = req.provenanceDocs > 0 ? 0.08 + 0.035 * Math.log(1 + req.provenanceDocs) : 0.0;
|
fmv.provenancePremium = req.provenanceDocs > 0 ? 0.08 + 0.035 * Math.log(1 + req.provenanceDocs) : 0.0;
|
||||||
@@ -170,12 +170,12 @@ public class ValuationAnalyticsResource {
|
|||||||
*/
|
*/
|
||||||
private double calculateUndervaluationScore(ValuationRequest req, double fmv) {
|
private double calculateUndervaluationScore(ValuationRequest req, double fmv) {
|
||||||
if (fmv <= 0) return 0.0;
|
if (fmv <= 0) return 0.0;
|
||||||
|
|
||||||
double priceGap = (fmv - req.currentBid) / fmv;
|
var priceGap = (fmv - req.currentBid) / fmv;
|
||||||
double velocityFactor = 1 + req.bidVelocity / 10.0;
|
var velocityFactor = 1 + req.bidVelocity / 10.0;
|
||||||
double watchRatio = Math.log(1 + req.watchCount / Math.max(req.bidCount, 1));
|
var watchRatio = Math.log(1 + req.watchCount / Math.max(req.bidCount, 1));
|
||||||
|
|
||||||
double uScore = priceGap * req.marketVolatility * velocityFactor * watchRatio;
|
var uScore = priceGap * req.marketVolatility * velocityFactor * watchRatio;
|
||||||
|
|
||||||
return Math.max(0.0, Math.round(uScore * 1000.0) / 1000.0);
|
return Math.max(0.0, Math.round(uScore * 1000.0) / 1000.0);
|
||||||
}
|
}
|
||||||
@@ -186,22 +186,22 @@ public class ValuationAnalyticsResource {
|
|||||||
*/
|
*/
|
||||||
private PricePrediction calculateFinalPrice(ValuationRequest req, double fmv) {
|
private PricePrediction calculateFinalPrice(ValuationRequest req, double fmv) {
|
||||||
// Bid momentum error: ε_bid = tanh(φ_1 · Λ_b - φ_2 · P_current/FMV)
|
// Bid momentum error: ε_bid = tanh(φ_1 · Λ_b - φ_2 · P_current/FMV)
|
||||||
double epsilonBid = Math.tanh(0.15 * req.bidVelocity - 0.10 * (req.currentBid / fmv));
|
var epsilonBid = Math.tanh(0.15 * req.bidVelocity - 0.10 * (req.currentBid / fmv));
|
||||||
|
|
||||||
// Time pressure error: ε_time = ψ · exp(-t_close/30)
|
// Time pressure error: ε_time = ψ · exp(-t_close/30)
|
||||||
double epsilonTime = 0.20 * Math.exp(-req.minutesUntilClose / 30.0);
|
var epsilonTime = 0.20 * Math.exp(-req.minutesUntilClose / 30.0);
|
||||||
|
|
||||||
// Competition error: ε_comp = ρ · ln(1 + W_watch/50)
|
// Competition error: ε_comp = ρ · ln(1 + W_watch/50)
|
||||||
double epsilonComp = 0.08 * Math.log(1 + req.watchCount / 50.0);
|
var epsilonComp = 0.08 * Math.log(1 + req.watchCount / 50.0);
|
||||||
|
|
||||||
double predictedPrice = fmv * (1 + epsilonBid + epsilonTime + epsilonComp);
|
var predictedPrice = fmv * (1 + epsilonBid + epsilonTime + epsilonComp);
|
||||||
|
|
||||||
// 95% confidence interval: ± 1.96 · σ_residual
|
// 95% confidence interval: ± 1.96 · σ_residual
|
||||||
double residualStdDev = fmv * 0.08; // Mock residual standard deviation
|
var residualStdDev = fmv * 0.08; // Mock residual standard deviation
|
||||||
double ciLower = predictedPrice - 1.96 * residualStdDev;
|
var ciLower = predictedPrice - 1.96 * residualStdDev;
|
||||||
double ciUpper = predictedPrice + 1.96 * residualStdDev;
|
var ciUpper = predictedPrice + 1.96 * residualStdDev;
|
||||||
|
|
||||||
PricePrediction pred = new PricePrediction();
|
var pred = new PricePrediction();
|
||||||
pred.predictedPrice = Math.round(predictedPrice * 100.0) / 100.0;
|
pred.predictedPrice = Math.round(predictedPrice * 100.0) / 100.0;
|
||||||
pred.confidenceIntervalLower = Math.round(ciLower * 100.0) / 100.0;
|
pred.confidenceIntervalLower = Math.round(ciLower * 100.0) / 100.0;
|
||||||
pred.confidenceIntervalUpper = Math.round(ciUpper * 100.0) / 100.0;
|
pred.confidenceIntervalUpper = Math.round(ciUpper * 100.0) / 100.0;
|
||||||
@@ -218,7 +218,7 @@ public class ValuationAnalyticsResource {
|
|||||||
* Generates optimal bidding strategy based on market conditions
|
* Generates optimal bidding strategy based on market conditions
|
||||||
*/
|
*/
|
||||||
private BiddingStrategy generateBiddingStrategy(ValuationRequest req, FairMarketValue fmv, PricePrediction pred) {
|
private BiddingStrategy generateBiddingStrategy(ValuationRequest req, FairMarketValue fmv, PricePrediction pred) {
|
||||||
BiddingStrategy strategy = new BiddingStrategy();
|
var strategy = new BiddingStrategy();
|
||||||
|
|
||||||
// Determine competition level
|
// Determine competition level
|
||||||
if (req.bidVelocity > 5.0) {
|
if (req.bidVelocity > 5.0) {
|
||||||
@@ -236,7 +236,7 @@ public class ValuationAnalyticsResource {
|
|||||||
strategy.recommendedTiming = "FINAL_10_MINUTES";
|
strategy.recommendedTiming = "FINAL_10_MINUTES";
|
||||||
|
|
||||||
// Adjust max bid based on undervaluation
|
// Adjust max bid based on undervaluation
|
||||||
double undervaluationScore = calculateUndervaluationScore(req, fmv.value);
|
var undervaluationScore = calculateUndervaluationScore(req, fmv.value);
|
||||||
if (undervaluationScore > 0.25) {
|
if (undervaluationScore > 0.25) {
|
||||||
// Aggressive strategy for undervalued lots
|
// Aggressive strategy for undervalued lots
|
||||||
strategy.maxBid = fmv.value * (1 + 0.05); // Conservative overbid
|
strategy.maxBid = fmv.value * (1 + 0.05); // Conservative overbid
|
||||||
@@ -270,7 +270,7 @@ public class ValuationAnalyticsResource {
|
|||||||
* Calculates confidence score based on number and quality of comparables
|
* Calculates confidence score based on number and quality of comparables
|
||||||
*/
|
*/
|
||||||
private double calculateFMVConfidence(int comparableCount, double totalWeight) {
|
private double calculateFMVConfidence(int comparableCount, double totalWeight) {
|
||||||
double confidence = 0.5; // Base confidence
|
var confidence = 0.5; // Base confidence
|
||||||
|
|
||||||
// Boost for more comparables
|
// Boost for more comparables
|
||||||
confidence += Math.min(comparableCount * 0.05, 0.3);
|
confidence += Math.min(comparableCount * 0.05, 0.3);
|
||||||
|
|||||||
164
src/main/java/auctiora/db/AuctionRepository.java
Normal file
164
src/main/java/auctiora/db/AuctionRepository.java
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
package auctiora.db;
|
||||||
|
|
||||||
|
import auctiora.AuctionInfo;
|
||||||
|
import lombok.RequiredArgsConstructor;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.jdbi.v3.core.Jdbi;
|
||||||
|
|
||||||
|
import java.time.LocalDateTime;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Repository for auction-related database operations using JDBI3.
|
||||||
|
* Handles CRUD operations and queries for auctions.
|
||||||
|
*/
|
||||||
|
@Slf4j
|
||||||
|
@RequiredArgsConstructor
|
||||||
|
public class AuctionRepository {
|
||||||
|
|
||||||
|
private final Jdbi jdbi;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Inserts or updates an auction record.
|
||||||
|
* Handles both auction_id conflicts and url uniqueness constraints.
|
||||||
|
*/
|
||||||
|
public void upsert(AuctionInfo auction) {
|
||||||
|
jdbi.useTransaction(handle -> {
|
||||||
|
try {
|
||||||
|
// Try INSERT with ON CONFLICT on auction_id
|
||||||
|
handle.createUpdate("""
|
||||||
|
INSERT INTO auctions (
|
||||||
|
auction_id, title, location, city, country, url, type, lot_count, closing_time, discovered_at
|
||||||
|
) VALUES (
|
||||||
|
:auctionId, :title, :location, :city, :country, :url, :type, :lotCount, :closingTime, :discoveredAt
|
||||||
|
)
|
||||||
|
ON CONFLICT(auction_id) DO UPDATE SET
|
||||||
|
title = excluded.title,
|
||||||
|
location = excluded.location,
|
||||||
|
city = excluded.city,
|
||||||
|
country = excluded.country,
|
||||||
|
url = excluded.url,
|
||||||
|
type = excluded.type,
|
||||||
|
lot_count = excluded.lot_count,
|
||||||
|
closing_time = excluded.closing_time
|
||||||
|
""")
|
||||||
|
.bind("auctionId", auction.auctionId())
|
||||||
|
.bind("title", auction.title())
|
||||||
|
.bind("location", auction.location())
|
||||||
|
.bind("city", auction.city())
|
||||||
|
.bind("country", auction.country())
|
||||||
|
.bind("url", auction.url())
|
||||||
|
.bind("type", auction.typePrefix())
|
||||||
|
.bind("lotCount", auction.lotCount())
|
||||||
|
.bind("closingTime", auction.firstLotClosingTime() != null ? auction.firstLotClosingTime().toString() : null)
|
||||||
|
.bind("discoveredAt", java.time.Instant.now().getEpochSecond())
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
} catch (Exception e) {
|
||||||
|
// If UNIQUE constraint on url fails, try updating by url
|
||||||
|
String errMsg = e.getMessage();
|
||||||
|
if (errMsg != null && (errMsg.contains("UNIQUE constraint failed") ||
|
||||||
|
errMsg.contains("PRIMARY KEY constraint failed"))) {
|
||||||
|
log.debug("Auction conflict detected, attempting update by URL: {}", auction.url());
|
||||||
|
|
||||||
|
int updated = handle.createUpdate("""
|
||||||
|
UPDATE auctions SET
|
||||||
|
auction_id = :auctionId,
|
||||||
|
title = :title,
|
||||||
|
location = :location,
|
||||||
|
city = :city,
|
||||||
|
country = :country,
|
||||||
|
type = :type,
|
||||||
|
lot_count = :lotCount,
|
||||||
|
closing_time = :closingTime
|
||||||
|
WHERE url = :url
|
||||||
|
""")
|
||||||
|
.bind("auctionId", auction.auctionId())
|
||||||
|
.bind("title", auction.title())
|
||||||
|
.bind("location", auction.location())
|
||||||
|
.bind("city", auction.city())
|
||||||
|
.bind("country", auction.country())
|
||||||
|
.bind("type", auction.typePrefix())
|
||||||
|
.bind("lotCount", auction.lotCount())
|
||||||
|
.bind("closingTime", auction.firstLotClosingTime() != null ? auction.firstLotClosingTime().toString() : null)
|
||||||
|
.bind("url", auction.url())
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
if (updated == 0) {
|
||||||
|
log.warn("Failed to update auction by URL: {}", auction.url());
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
log.error("Unexpected error upserting auction: {}", e.getMessage(), e);
|
||||||
|
throw e;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves all auctions from the database.
|
||||||
|
*/
|
||||||
|
public List<AuctionInfo> getAll() {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT * FROM auctions")
|
||||||
|
.map((rs, ctx) -> {
|
||||||
|
String closingStr = rs.getString("closing_time");
|
||||||
|
LocalDateTime closingTime = null;
|
||||||
|
if (closingStr != null && !closingStr.isBlank()) {
|
||||||
|
try {
|
||||||
|
closingTime = LocalDateTime.parse(closingStr);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("Invalid closing_time format: {}", closingStr);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return new AuctionInfo(
|
||||||
|
rs.getLong("auction_id"),
|
||||||
|
rs.getString("title"),
|
||||||
|
rs.getString("location"),
|
||||||
|
rs.getString("city"),
|
||||||
|
rs.getString("country"),
|
||||||
|
rs.getString("url"),
|
||||||
|
rs.getString("type"),
|
||||||
|
rs.getInt("lot_count"),
|
||||||
|
closingTime
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves auctions filtered by country code.
|
||||||
|
*/
|
||||||
|
public List<AuctionInfo> getByCountry(String countryCode) {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT * FROM auctions WHERE country = :country")
|
||||||
|
.bind("country", countryCode)
|
||||||
|
.map((rs, ctx) -> {
|
||||||
|
String closingStr = rs.getString("closing_time");
|
||||||
|
LocalDateTime closingTime = null;
|
||||||
|
if (closingStr != null && !closingStr.isBlank()) {
|
||||||
|
try {
|
||||||
|
closingTime = LocalDateTime.parse(closingStr);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.warn("Invalid closing_time format: {}", closingStr);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return new AuctionInfo(
|
||||||
|
rs.getLong("auction_id"),
|
||||||
|
rs.getString("title"),
|
||||||
|
rs.getString("location"),
|
||||||
|
rs.getString("city"),
|
||||||
|
rs.getString("country"),
|
||||||
|
rs.getString("url"),
|
||||||
|
rs.getString("type"),
|
||||||
|
rs.getInt("lot_count"),
|
||||||
|
closingTime
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
154
src/main/java/auctiora/db/DatabaseSchema.java
Normal file
154
src/main/java/auctiora/db/DatabaseSchema.java
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
package auctiora.db;
|
||||||
|
|
||||||
|
import lombok.experimental.UtilityClass;
|
||||||
|
import org.jdbi.v3.core.Jdbi;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Database schema DDL definitions for all tables and indexes.
|
||||||
|
* Uses text blocks (Java 15+) for clean SQL formatting.
|
||||||
|
*/
|
||||||
|
@UtilityClass
|
||||||
|
public class DatabaseSchema {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initializes all database tables and indexes if they don't exist.
|
||||||
|
*/
|
||||||
|
public void ensureSchema(Jdbi jdbi) {
|
||||||
|
jdbi.useHandle(handle -> {
|
||||||
|
// Enable WAL mode for better concurrent access
|
||||||
|
handle.execute("PRAGMA journal_mode=WAL");
|
||||||
|
handle.execute("PRAGMA busy_timeout=10000");
|
||||||
|
handle.execute("PRAGMA synchronous=NORMAL");
|
||||||
|
|
||||||
|
createTables(handle);
|
||||||
|
createIndexes(handle);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createTables(org.jdbi.v3.core.Handle handle) {
|
||||||
|
// Cache table (for HTTP caching)
|
||||||
|
handle.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS cache (
|
||||||
|
url TEXT PRIMARY KEY,
|
||||||
|
content BLOB,
|
||||||
|
timestamp REAL,
|
||||||
|
status_code INTEGER
|
||||||
|
)""");
|
||||||
|
|
||||||
|
// Auctions table (populated by external scraper)
|
||||||
|
handle.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS auctions (
|
||||||
|
auction_id TEXT PRIMARY KEY,
|
||||||
|
url TEXT UNIQUE,
|
||||||
|
title TEXT,
|
||||||
|
location TEXT,
|
||||||
|
lots_count INTEGER,
|
||||||
|
first_lot_closing_time TEXT,
|
||||||
|
scraped_at TEXT,
|
||||||
|
city TEXT,
|
||||||
|
country TEXT,
|
||||||
|
type TEXT,
|
||||||
|
lot_count INTEGER DEFAULT 0,
|
||||||
|
closing_time TEXT,
|
||||||
|
discovered_at INTEGER
|
||||||
|
)""");
|
||||||
|
|
||||||
|
// Lots table (populated by external scraper)
|
||||||
|
handle.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS lots (
|
||||||
|
lot_id TEXT PRIMARY KEY,
|
||||||
|
auction_id TEXT,
|
||||||
|
url TEXT UNIQUE,
|
||||||
|
title TEXT,
|
||||||
|
current_bid TEXT,
|
||||||
|
bid_count INTEGER,
|
||||||
|
closing_time TEXT,
|
||||||
|
viewing_time TEXT,
|
||||||
|
pickup_date TEXT,
|
||||||
|
location TEXT,
|
||||||
|
description TEXT,
|
||||||
|
category TEXT,
|
||||||
|
scraped_at TEXT,
|
||||||
|
sale_id INTEGER,
|
||||||
|
manufacturer TEXT,
|
||||||
|
type TEXT,
|
||||||
|
year INTEGER,
|
||||||
|
currency TEXT DEFAULT 'EUR',
|
||||||
|
closing_notified INTEGER DEFAULT 0,
|
||||||
|
starting_bid TEXT,
|
||||||
|
minimum_bid TEXT,
|
||||||
|
status TEXT,
|
||||||
|
brand TEXT,
|
||||||
|
model TEXT,
|
||||||
|
attributes_json TEXT,
|
||||||
|
first_bid_time TEXT,
|
||||||
|
last_bid_time TEXT,
|
||||||
|
bid_velocity REAL,
|
||||||
|
bid_increment REAL,
|
||||||
|
year_manufactured INTEGER,
|
||||||
|
condition_score REAL,
|
||||||
|
condition_description TEXT,
|
||||||
|
serial_number TEXT,
|
||||||
|
damage_description TEXT,
|
||||||
|
followers_count INTEGER DEFAULT 0,
|
||||||
|
estimated_min_price REAL,
|
||||||
|
estimated_max_price REAL,
|
||||||
|
lot_condition TEXT,
|
||||||
|
appearance TEXT,
|
||||||
|
estimated_min REAL,
|
||||||
|
estimated_max REAL,
|
||||||
|
next_bid_step_cents INTEGER,
|
||||||
|
condition TEXT,
|
||||||
|
category_path TEXT,
|
||||||
|
city_location TEXT,
|
||||||
|
country_code TEXT,
|
||||||
|
bidding_status TEXT,
|
||||||
|
packaging TEXT,
|
||||||
|
quantity INTEGER,
|
||||||
|
vat REAL,
|
||||||
|
buyer_premium_percentage REAL,
|
||||||
|
remarks TEXT,
|
||||||
|
reserve_price REAL,
|
||||||
|
reserve_met INTEGER,
|
||||||
|
view_count INTEGER,
|
||||||
|
FOREIGN KEY (auction_id) REFERENCES auctions(auction_id)
|
||||||
|
)""");
|
||||||
|
|
||||||
|
// Images table (populated by external scraper with URLs and local_path)
|
||||||
|
handle.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS images (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
lot_id TEXT,
|
||||||
|
url TEXT,
|
||||||
|
local_path TEXT,
|
||||||
|
downloaded INTEGER DEFAULT 0,
|
||||||
|
labels TEXT,
|
||||||
|
processed_at INTEGER,
|
||||||
|
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
||||||
|
)""");
|
||||||
|
|
||||||
|
// Bid history table
|
||||||
|
handle.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS bid_history (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
lot_id TEXT NOT NULL,
|
||||||
|
bid_amount REAL NOT NULL,
|
||||||
|
bid_time TEXT NOT NULL,
|
||||||
|
is_autobid INTEGER DEFAULT 0,
|
||||||
|
bidder_id TEXT,
|
||||||
|
bidder_number INTEGER,
|
||||||
|
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (lot_id) REFERENCES lots(lot_id)
|
||||||
|
)""");
|
||||||
|
}
|
||||||
|
|
||||||
|
private void createIndexes(org.jdbi.v3.core.Handle handle) {
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_timestamp ON cache(timestamp)");
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_auctions_country ON auctions(country)");
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_lots_sale_id ON lots(sale_id)");
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_images_lot_id ON images(lot_id)");
|
||||||
|
handle.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_lot_url ON images(lot_id, url)");
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_bid_history_lot_time ON bid_history(lot_id, bid_time)");
|
||||||
|
handle.execute("CREATE INDEX IF NOT EXISTS idx_bid_history_bidder ON bid_history(bidder_id)");
|
||||||
|
}
|
||||||
|
}
|
||||||
137
src/main/java/auctiora/db/ImageRepository.java
Normal file
137
src/main/java/auctiora/db/ImageRepository.java
Normal file
@@ -0,0 +1,137 @@
|
|||||||
|
package auctiora.db;
|
||||||
|
|
||||||
|
import lombok.RequiredArgsConstructor;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.jdbi.v3.core.Jdbi;
|
||||||
|
|
||||||
|
import java.time.Instant;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Repository for image-related database operations using JDBI3.
|
||||||
|
* Handles image storage, object detection labels, and processing status.
|
||||||
|
*/
|
||||||
|
@Slf4j
|
||||||
|
@RequiredArgsConstructor
|
||||||
|
public class ImageRepository {
|
||||||
|
|
||||||
|
private final Jdbi jdbi;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Image record containing all image metadata.
|
||||||
|
*/
|
||||||
|
public record ImageRecord(int id, long lotId, String url, String filePath, String labels) {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Minimal record for images needing object detection processing.
|
||||||
|
*/
|
||||||
|
public record ImageDetectionRecord(int id, long lotId, String filePath) {}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Inserts a complete image record (for testing/legacy compatibility).
|
||||||
|
* In production, scraper inserts with local_path, monitor updates labels via updateLabels.
|
||||||
|
*/
|
||||||
|
public void insert(long lotId, String url, String filePath, List<String> labels) {
|
||||||
|
jdbi.useHandle(handle ->
|
||||||
|
handle.createUpdate("""
|
||||||
|
INSERT INTO images (lot_id, url, local_path, labels, processed_at, downloaded)
|
||||||
|
VALUES (:lotId, :url, :localPath, :labels, :processedAt, 1)
|
||||||
|
""")
|
||||||
|
.bind("lotId", lotId)
|
||||||
|
.bind("url", url)
|
||||||
|
.bind("localPath", filePath)
|
||||||
|
.bind("labels", String.join(",", labels))
|
||||||
|
.bind("processedAt", Instant.now().getEpochSecond())
|
||||||
|
.execute()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Updates the labels field for an image after object detection.
|
||||||
|
*/
|
||||||
|
public void updateLabels(int imageId, List<String> labels) {
|
||||||
|
jdbi.useHandle(handle ->
|
||||||
|
handle.createUpdate("UPDATE images SET labels = :labels, processed_at = :processedAt WHERE id = :id")
|
||||||
|
.bind("labels", String.join(",", labels))
|
||||||
|
.bind("processedAt", Instant.now().getEpochSecond())
|
||||||
|
.bind("id", imageId)
|
||||||
|
.execute()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gets the labels for a specific image.
|
||||||
|
*/
|
||||||
|
public List<String> getLabels(int imageId) {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT labels FROM images WHERE id = :id")
|
||||||
|
.bind("id", imageId)
|
||||||
|
.mapTo(String.class)
|
||||||
|
.findOne()
|
||||||
|
.map(labelsStr -> {
|
||||||
|
if (labelsStr != null && !labelsStr.isEmpty()) {
|
||||||
|
return List.of(labelsStr.split(","));
|
||||||
|
}
|
||||||
|
return List.<String>of();
|
||||||
|
})
|
||||||
|
.orElse(List.of())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves images for a specific lot.
|
||||||
|
*/
|
||||||
|
public List<ImageRecord> getImagesForLot(long lotId) {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT id, lot_id, url, local_path, labels FROM images WHERE lot_id = :lotId")
|
||||||
|
.bind("lotId", lotId)
|
||||||
|
.map((rs, ctx) -> new ImageRecord(
|
||||||
|
rs.getInt("id"),
|
||||||
|
rs.getLong("lot_id"),
|
||||||
|
rs.getString("url"),
|
||||||
|
rs.getString("local_path"),
|
||||||
|
rs.getString("labels")
|
||||||
|
))
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gets images that have been downloaded by the scraper but need object detection.
|
||||||
|
* Only returns images that have local_path set but no labels yet.
|
||||||
|
*/
|
||||||
|
public List<ImageDetectionRecord> getImagesNeedingDetection() {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("""
|
||||||
|
SELECT i.id, i.lot_id, i.local_path
|
||||||
|
FROM images i
|
||||||
|
WHERE i.local_path IS NOT NULL
|
||||||
|
AND i.local_path != ''
|
||||||
|
AND (i.labels IS NULL OR i.labels = '')
|
||||||
|
""")
|
||||||
|
.map((rs, ctx) -> {
|
||||||
|
// Extract numeric lot ID from TEXT field (e.g., "A1-34732-49" -> 3473249)
|
||||||
|
String lotIdStr = rs.getString("lot_id");
|
||||||
|
long lotId = auctiora.ScraperDataAdapter.extractNumericId(lotIdStr);
|
||||||
|
|
||||||
|
return new ImageDetectionRecord(
|
||||||
|
rs.getInt("id"),
|
||||||
|
lotId,
|
||||||
|
rs.getString("local_path")
|
||||||
|
);
|
||||||
|
})
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gets the total number of images in the database.
|
||||||
|
*/
|
||||||
|
public int getImageCount() {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT COUNT(*) FROM images")
|
||||||
|
.mapTo(Integer.class)
|
||||||
|
.one()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
275
src/main/java/auctiora/db/LotRepository.java
Normal file
275
src/main/java/auctiora/db/LotRepository.java
Normal file
@@ -0,0 +1,275 @@
|
|||||||
|
package auctiora.db;
|
||||||
|
|
||||||
|
import auctiora.Lot;
|
||||||
|
import auctiora.BidHistory;
|
||||||
|
import lombok.RequiredArgsConstructor;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.jdbi.v3.core.Jdbi;
|
||||||
|
|
||||||
|
import java.time.LocalDateTime;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import static java.sql.Types.*;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Repository for lot-related database operations using JDBI3.
|
||||||
|
* Handles CRUD operations and queries for auction lots.
|
||||||
|
*/
|
||||||
|
@Slf4j
|
||||||
|
@RequiredArgsConstructor
|
||||||
|
public class LotRepository {
|
||||||
|
|
||||||
|
private final Jdbi jdbi;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Inserts or updates a lot (upsert operation).
|
||||||
|
* First tries UPDATE, then falls back to INSERT if lot doesn't exist.
|
||||||
|
*/
|
||||||
|
public void upsert(Lot lot) {
|
||||||
|
jdbi.useTransaction(handle -> {
|
||||||
|
// Try UPDATE first
|
||||||
|
int updated = handle.createUpdate("""
|
||||||
|
UPDATE lots SET
|
||||||
|
sale_id = :saleId,
|
||||||
|
auction_id = :auctionId,
|
||||||
|
title = :title,
|
||||||
|
description = :description,
|
||||||
|
manufacturer = :manufacturer,
|
||||||
|
type = :type,
|
||||||
|
year = :year,
|
||||||
|
category = :category,
|
||||||
|
current_bid = :currentBid,
|
||||||
|
currency = :currency,
|
||||||
|
url = :url,
|
||||||
|
closing_time = :closingTime
|
||||||
|
WHERE lot_id = :lotId
|
||||||
|
""")
|
||||||
|
.bind("saleId", String.valueOf(lot.saleId()))
|
||||||
|
.bind("auctionId", String.valueOf(lot.saleId())) // auction_id = sale_id
|
||||||
|
.bind("title", lot.title())
|
||||||
|
.bind("description", lot.description())
|
||||||
|
.bind("manufacturer", lot.manufacturer())
|
||||||
|
.bind("type", lot.type())
|
||||||
|
.bind("year", lot.year())
|
||||||
|
.bind("category", lot.category())
|
||||||
|
.bind("currentBid", lot.currentBid())
|
||||||
|
.bind("currency", lot.currency())
|
||||||
|
.bind("url", lot.url())
|
||||||
|
.bind("closingTime", lot.closingTime() != null ? lot.closingTime().toString() : null)
|
||||||
|
.bind("lotId", String.valueOf(lot.lotId()))
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
if (updated == 0) {
|
||||||
|
// No rows updated, perform INSERT
|
||||||
|
handle.createUpdate("""
|
||||||
|
INSERT OR IGNORE INTO lots (
|
||||||
|
lot_id, sale_id, auction_id, title, description, manufacturer, type, year,
|
||||||
|
category, current_bid, currency, url, closing_time, closing_notified
|
||||||
|
) VALUES (
|
||||||
|
:lotId, :saleId, :auctionId, :title, :description, :manufacturer, :type, :year,
|
||||||
|
:category, :currentBid, :currency, :url, :closingTime, :closingNotified
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
.bind("lotId", String.valueOf(lot.lotId()))
|
||||||
|
.bind("saleId", String.valueOf(lot.saleId()))
|
||||||
|
.bind("auctionId", String.valueOf(lot.saleId())) // auction_id = sale_id
|
||||||
|
.bind("title", lot.title())
|
||||||
|
.bind("description", lot.description())
|
||||||
|
.bind("manufacturer", lot.manufacturer())
|
||||||
|
.bind("type", lot.type())
|
||||||
|
.bind("year", lot.year())
|
||||||
|
.bind("category", lot.category())
|
||||||
|
.bind("currentBid", lot.currentBid())
|
||||||
|
.bind("currency", lot.currency())
|
||||||
|
.bind("url", lot.url())
|
||||||
|
.bind("closingTime", lot.closingTime() != null ? lot.closingTime().toString() : null)
|
||||||
|
.bind("closingNotified", lot.closingNotified() ? 1 : 0)
|
||||||
|
.execute();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Updates a lot with full intelligence data from GraphQL enrichment.
|
||||||
|
* Includes all 24+ intelligence fields from bidding platform.
|
||||||
|
*/
|
||||||
|
public void upsertWithIntelligence(Lot lot) {
|
||||||
|
jdbi.useHandle(handle -> {
|
||||||
|
var update = handle.createUpdate("""
|
||||||
|
UPDATE lots SET
|
||||||
|
sale_id = :saleId,
|
||||||
|
title = :title,
|
||||||
|
description = :description,
|
||||||
|
manufacturer = :manufacturer,
|
||||||
|
type = :type,
|
||||||
|
year = :year,
|
||||||
|
category = :category,
|
||||||
|
current_bid = :currentBid,
|
||||||
|
currency = :currency,
|
||||||
|
url = :url,
|
||||||
|
closing_time = :closingTime,
|
||||||
|
followers_count = :followersCount,
|
||||||
|
estimated_min = :estimatedMin,
|
||||||
|
estimated_max = :estimatedMax,
|
||||||
|
next_bid_step_cents = :nextBidStepInCents,
|
||||||
|
condition = :condition,
|
||||||
|
category_path = :categoryPath,
|
||||||
|
city_location = :cityLocation,
|
||||||
|
country_code = :countryCode,
|
||||||
|
bidding_status = :biddingStatus,
|
||||||
|
appearance = :appearance,
|
||||||
|
packaging = :packaging,
|
||||||
|
quantity = :quantity,
|
||||||
|
vat = :vat,
|
||||||
|
buyer_premium_percentage = :buyerPremiumPercentage,
|
||||||
|
remarks = :remarks,
|
||||||
|
starting_bid = :startingBid,
|
||||||
|
reserve_price = :reservePrice,
|
||||||
|
reserve_met = :reserveMet,
|
||||||
|
bid_increment = :bidIncrement,
|
||||||
|
view_count = :viewCount,
|
||||||
|
first_bid_time = :firstBidTime,
|
||||||
|
last_bid_time = :lastBidTime,
|
||||||
|
bid_velocity = :bidVelocity
|
||||||
|
WHERE lot_id = :lotId
|
||||||
|
""")
|
||||||
|
.bind("saleId", lot.saleId())
|
||||||
|
.bind("title", lot.title())
|
||||||
|
.bind("description", lot.description())
|
||||||
|
.bind("manufacturer", lot.manufacturer())
|
||||||
|
.bind("type", lot.type())
|
||||||
|
.bind("year", lot.year())
|
||||||
|
.bind("category", lot.category())
|
||||||
|
.bind("currentBid", lot.currentBid())
|
||||||
|
.bind("currency", lot.currency())
|
||||||
|
.bind("url", lot.url())
|
||||||
|
.bind("closingTime", lot.closingTime() != null ? lot.closingTime().toString() : null)
|
||||||
|
.bind("followersCount", lot.followersCount())
|
||||||
|
.bind("estimatedMin", lot.estimatedMin())
|
||||||
|
.bind("estimatedMax", lot.estimatedMax())
|
||||||
|
.bind("nextBidStepInCents", lot.nextBidStepInCents())
|
||||||
|
.bind("condition", lot.condition())
|
||||||
|
.bind("categoryPath", lot.categoryPath())
|
||||||
|
.bind("cityLocation", lot.cityLocation())
|
||||||
|
.bind("countryCode", lot.countryCode())
|
||||||
|
.bind("biddingStatus", lot.biddingStatus())
|
||||||
|
.bind("appearance", lot.appearance())
|
||||||
|
.bind("packaging", lot.packaging())
|
||||||
|
.bind("quantity", lot.quantity())
|
||||||
|
.bind("vat", lot.vat())
|
||||||
|
.bind("buyerPremiumPercentage", lot.buyerPremiumPercentage())
|
||||||
|
.bind("remarks", lot.remarks())
|
||||||
|
.bind("startingBid", lot.startingBid())
|
||||||
|
.bind("reservePrice", lot.reservePrice())
|
||||||
|
.bind("reserveMet", lot.reserveMet() != null && lot.reserveMet() ? 1 : null)
|
||||||
|
.bind("bidIncrement", lot.bidIncrement())
|
||||||
|
.bind("viewCount", lot.viewCount())
|
||||||
|
.bind("firstBidTime", lot.firstBidTime() != null ? lot.firstBidTime().toString() : null)
|
||||||
|
.bind("lastBidTime", lot.lastBidTime() != null ? lot.lastBidTime().toString() : null)
|
||||||
|
.bind("bidVelocity", lot.bidVelocity())
|
||||||
|
.bind("lotId", lot.lotId());
|
||||||
|
|
||||||
|
int updated = update.execute();
|
||||||
|
if (updated == 0) {
|
||||||
|
log.warn("Failed to update lot {} - lot not found in database", lot.lotId());
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Updates only the current bid for a lot (lightweight update).
|
||||||
|
*/
|
||||||
|
public void updateCurrentBid(Lot lot) {
|
||||||
|
jdbi.useHandle(handle ->
|
||||||
|
handle.createUpdate("UPDATE lots SET current_bid = :bid WHERE lot_id = :lotId")
|
||||||
|
.bind("bid", lot.currentBid())
|
||||||
|
.bind("lotId", String.valueOf(lot.lotId()))
|
||||||
|
.execute()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Updates notification flags for a lot.
|
||||||
|
*/
|
||||||
|
public void updateNotificationFlags(Lot lot) {
|
||||||
|
jdbi.useHandle(handle ->
|
||||||
|
handle.createUpdate("UPDATE lots SET closing_notified = :notified WHERE lot_id = :lotId")
|
||||||
|
.bind("notified", lot.closingNotified() ? 1 : 0)
|
||||||
|
.bind("lotId", String.valueOf(lot.lotId()))
|
||||||
|
.execute()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves all active lots.
|
||||||
|
* Note: Despite the name, this returns ALL lots (legacy behavior for backward compatibility).
|
||||||
|
*/
|
||||||
|
public List<Lot> getActiveLots() {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT * FROM lots")
|
||||||
|
.map((rs, ctx) -> auctiora.ScraperDataAdapter.fromScraperLot(rs))
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves all lots from the database.
|
||||||
|
*/
|
||||||
|
public List<Lot> getAllLots() {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("SELECT * FROM lots")
|
||||||
|
.map((rs, ctx) -> auctiora.ScraperDataAdapter.fromScraperLot(rs))
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Retrieves bid history for a specific lot.
|
||||||
|
*/
|
||||||
|
public List<BidHistory> getBidHistory(String lotId) {
|
||||||
|
return jdbi.withHandle(handle ->
|
||||||
|
handle.createQuery("""
|
||||||
|
SELECT id, lot_id, bid_amount, bid_time, is_autobid, bidder_id, bidder_number
|
||||||
|
FROM bid_history
|
||||||
|
WHERE lot_id = :lotId
|
||||||
|
ORDER BY bid_time DESC
|
||||||
|
""")
|
||||||
|
.bind("lotId", lotId)
|
||||||
|
.map((rs, ctx) -> new BidHistory(
|
||||||
|
rs.getInt("id"),
|
||||||
|
rs.getString("lot_id"),
|
||||||
|
rs.getDouble("bid_amount"),
|
||||||
|
LocalDateTime.parse(rs.getString("bid_time")),
|
||||||
|
rs.getInt("is_autobid") != 0,
|
||||||
|
rs.getString("bidder_id"),
|
||||||
|
(Integer) rs.getObject("bidder_number")
|
||||||
|
))
|
||||||
|
.list()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Inserts bid history records in batch.
|
||||||
|
*/
|
||||||
|
public void insertBidHistory(List<BidHistory> bidHistory) {
|
||||||
|
jdbi.useHandle(handle -> {
|
||||||
|
var batch = handle.prepareBatch("""
|
||||||
|
INSERT OR IGNORE INTO bid_history (
|
||||||
|
lot_id, bid_amount, bid_time, is_autobid, bidder_id, bidder_number
|
||||||
|
) VALUES (:lotId, :bidAmount, :bidTime, :isAutobid, :bidderId, :bidderNumber)
|
||||||
|
""");
|
||||||
|
|
||||||
|
bidHistory.forEach(bid ->
|
||||||
|
batch.bind("lotId", bid.lotId())
|
||||||
|
.bind("bidAmount", bid.bidAmount())
|
||||||
|
.bind("bidTime", bid.bidTime().toString())
|
||||||
|
.bind("isAutobid", bid.isAutobid() ? 1 : 0)
|
||||||
|
.bind("bidderId", bid.bidderId())
|
||||||
|
.bind("bidderNumber", bid.bidderNumber())
|
||||||
|
.add()
|
||||||
|
);
|
||||||
|
|
||||||
|
batch.execute();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -44,12 +44,15 @@ quarkus.rest.path=/
|
|||||||
quarkus.http.root-path=/
|
quarkus.http.root-path=/
|
||||||
|
|
||||||
# Auction Monitor Configuration
|
# Auction Monitor Configuration
|
||||||
auction.database.path=C:\\mnt\\okcomputer\\output\\cache.db
|
auction.database.path=/mnt/okcomputer/output/cache.db
|
||||||
auction.images.path=C:\\mnt\\okcomputer\\output\\images
|
auction.images.path=/mnt/okcomputer/output/images
|
||||||
auction.notification.config=desktop
|
# auction.notification.config=desktop
|
||||||
auction.yolo.config=models/yolov4.cfg
|
# Format: smtp:username:password:recipient_email
|
||||||
auction.yolo.weights=models/yolov4.weights
|
auction.notification.config=smtp:michael.bakker1986@gmail.com:agrepolhlnvhipkv:michael.bakker1986@gmail.com
|
||||||
auction.yolo.classes=models/coco.names
|
|
||||||
|
auction.yolo.config=/mnt/okcomputer/output/models/yolov4.cfg
|
||||||
|
auction.yolo.weights=/mnt/okcomputer/output/models/yolov4.weights
|
||||||
|
auction.yolo.classes=/mnt/okcomputer/output/models/coco.names
|
||||||
|
|
||||||
# Scheduler Configuration
|
# Scheduler Configuration
|
||||||
quarkus.scheduler.enabled=true
|
quarkus.scheduler.enabled=true
|
||||||
@@ -69,3 +72,4 @@ auction.http.timeout-seconds=30
|
|||||||
|
|
||||||
# Health Check Configuration
|
# Health Check Configuration
|
||||||
quarkus.smallrye-health.root-path=/health
|
quarkus.smallrye-health.root-path=/health
|
||||||
|
|
||||||
|
|||||||
@@ -1,115 +0,0 @@
|
|||||||
package auctiora;
|
|
||||||
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
|
||||||
import org.jsoup.Jsoup;
|
|
||||||
import org.jsoup.nodes.Document;
|
|
||||||
import org.jsoup.nodes.Element;
|
|
||||||
import org.jsoup.select.Elements;
|
|
||||||
import org.junit.jupiter.api.BeforeAll;
|
|
||||||
import org.junit.jupiter.api.Test;
|
|
||||||
|
|
||||||
import java.io.IOException;
|
|
||||||
import java.nio.file.Files;
|
|
||||||
import java.nio.file.Paths;
|
|
||||||
import java.util.ArrayList;
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
import static org.junit.jupiter.api.Assertions.*;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Test auction parsing logic using saved HTML from test.html
|
|
||||||
* Tests the markup data extraction for each auction found
|
|
||||||
*/
|
|
||||||
@Slf4j
|
|
||||||
public class AuctionParsingTest {
|
|
||||||
|
|
||||||
private static String testHtml;
|
|
||||||
|
|
||||||
@BeforeAll
|
|
||||||
public static void loadTestHtml() throws IOException {
|
|
||||||
// Load the test HTML file
|
|
||||||
testHtml = Files.readString(Paths.get("src/test/resources/test_auctions.html"));
|
|
||||||
log.info("Loaded test HTML ({} characters)", testHtml.length());
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testLocationPatternMatching() {
|
|
||||||
log.info("\n=== Location Pattern Tests ===");
|
|
||||||
|
|
||||||
// Test different location formats
|
|
||||||
var testCases = new String[]{
|
|
||||||
"<p>Amsterdam, NL</p>",
|
|
||||||
"<p class=\"flex truncate\"><span class=\"w-full truncate\">Sofia,<!-- --> </span>BG</p>",
|
|
||||||
"<p>Berlin, DE</p>",
|
|
||||||
"<span>Brussels,</span>BE"
|
|
||||||
};
|
|
||||||
|
|
||||||
for (var testHtml : testCases) {
|
|
||||||
var doc = Jsoup.parse(testHtml);
|
|
||||||
var elem = doc.select("p, span").first();
|
|
||||||
|
|
||||||
if (elem != null) {
|
|
||||||
var text = elem.text();
|
|
||||||
log.info("\nTest: {}", testHtml);
|
|
||||||
log.info("Text: {}", text);
|
|
||||||
|
|
||||||
// Test regex pattern
|
|
||||||
if (text.matches(".*[A-Z]{2}$")) {
|
|
||||||
var countryCode = text.substring(text.length() - 2);
|
|
||||||
var cityPart = text.substring(0, text.length() - 2).trim().replaceAll("[,\\s]+$", "");
|
|
||||||
log.info("→ Extracted: {}, {}", cityPart, countryCode);
|
|
||||||
} else {
|
|
||||||
log.info("→ No match");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
public void testFullTextPatternMatching() {
|
|
||||||
log.info("\n=== Full Text Pattern Tests ===");
|
|
||||||
|
|
||||||
// Test the complete auction text format
|
|
||||||
var testCases = new String[]{
|
|
||||||
"woensdag om 18:00 1 Vrachtwagens voor bedrijfsvoertuigen Loßburg, DE",
|
|
||||||
"maandag om 14:30 5 Industriële machines Amsterdam, NL",
|
|
||||||
"vrijdag om 10:00 12 Landbouwmachines Antwerpen, BE"
|
|
||||||
};
|
|
||||||
|
|
||||||
for (var testText : testCases) {
|
|
||||||
log.info("\nParsing: \"{}\"", testText);
|
|
||||||
|
|
||||||
// Simulated extraction
|
|
||||||
var remaining = testText;
|
|
||||||
|
|
||||||
// Extract time
|
|
||||||
var timePattern = java.util.regex.Pattern.compile("(\\w+)\\s+om\\s+(\\d{1,2}:\\d{2})");
|
|
||||||
var timeMatcher = timePattern.matcher(remaining);
|
|
||||||
if (timeMatcher.find()) {
|
|
||||||
log.info(" Time: {} om {}", timeMatcher.group(1), timeMatcher.group(2));
|
|
||||||
remaining = remaining.substring(timeMatcher.end()).trim();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract location
|
|
||||||
var locPattern = java.util.regex.Pattern.compile(
|
|
||||||
"([A-ZÀ-ÿa-z][A-ZÀ-ÿa-z\\s\\-'öäüßàèéêëïôùûç]+?),\\s*([A-Z]{2})\\s*$"
|
|
||||||
);
|
|
||||||
var locMatcher = locPattern.matcher(remaining);
|
|
||||||
if (locMatcher.find()) {
|
|
||||||
log.info(" Location: {}, {}", locMatcher.group(1), locMatcher.group(2));
|
|
||||||
remaining = remaining.substring(0, locMatcher.start()).trim();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract lot count
|
|
||||||
var lotPattern = java.util.regex.Pattern.compile("^(\\d+)\\s+");
|
|
||||||
var lotMatcher = lotPattern.matcher(remaining);
|
|
||||||
if (lotMatcher.find()) {
|
|
||||||
log.info(" Lot count: {}", lotMatcher.group(1));
|
|
||||||
remaining = remaining.substring(lotMatcher.end()).trim();
|
|
||||||
}
|
|
||||||
|
|
||||||
// What remains is title
|
|
||||||
log.info(" Title: {}", remaining);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
138
src/test/java/auctiora/ClosingTimeCalculationTest.java
Normal file
138
src/test/java/auctiora/ClosingTimeCalculationTest.java
Normal file
@@ -0,0 +1,138 @@
|
|||||||
|
package auctiora;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.*;
|
||||||
|
|
||||||
|
import java.time.LocalDateTime;
|
||||||
|
|
||||||
|
import static org.junit.jupiter.api.Assertions.*;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests for closing time calculations that power the UI
|
||||||
|
* Tests the minutesUntilClose() logic used in dashboard and alerts
|
||||||
|
*/
|
||||||
|
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
|
||||||
|
@DisplayName("Closing Time Calculation Tests")
|
||||||
|
class ClosingTimeCalculationTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(1)
|
||||||
|
@DisplayName("Should calculate minutes until close for lot closing in 15 minutes")
|
||||||
|
void testMinutesUntilClose15Minutes() {
|
||||||
|
var lot = createLot(LocalDateTime.now().plusMinutes(15));
|
||||||
|
long minutes = lot.minutesUntilClose();
|
||||||
|
|
||||||
|
assertTrue(minutes >= 14 && minutes <= 16,
|
||||||
|
"Should be approximately 15 minutes, was: " + minutes);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(2)
|
||||||
|
@DisplayName("Should calculate minutes until close for lot closing in 2 hours")
|
||||||
|
void testMinutesUntilClose2Hours() {
|
||||||
|
var lot = createLot(LocalDateTime.now().plusHours(2));
|
||||||
|
long minutes = lot.minutesUntilClose();
|
||||||
|
|
||||||
|
assertTrue(minutes >= 119 && minutes <= 121,
|
||||||
|
"Should be approximately 120 minutes, was: " + minutes);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(3)
|
||||||
|
@DisplayName("Should return negative value for already closed lot")
|
||||||
|
void testMinutesUntilCloseNegative() {
|
||||||
|
var lot = createLot(LocalDateTime.now().minusHours(1));
|
||||||
|
long minutes = lot.minutesUntilClose();
|
||||||
|
|
||||||
|
assertTrue(minutes < 0,
|
||||||
|
"Should be negative for closed lots, was: " + minutes);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(4)
|
||||||
|
@DisplayName("Should return MAX_VALUE when lot has no closing time")
|
||||||
|
void testMinutesUntilCloseNoTime() {
|
||||||
|
var lot = Lot.basic(100, 1001, "No closing time", "", "", "", 0, "General",
|
||||||
|
100.0, "EUR", "http://test.com/1001", null, false);
|
||||||
|
long minutes = lot.minutesUntilClose();
|
||||||
|
|
||||||
|
assertEquals(Long.MAX_VALUE, minutes,
|
||||||
|
"Should return MAX_VALUE when no closing time set");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(5)
|
||||||
|
@DisplayName("Should identify lots closing within 5 minutes (critical threshold)")
|
||||||
|
void testCriticalClosingThreshold() {
|
||||||
|
var closing4Min = createLot(LocalDateTime.now().plusMinutes(4));
|
||||||
|
var closing5Min = createLot(LocalDateTime.now().plusMinutes(5));
|
||||||
|
var closing6Min = createLot(LocalDateTime.now().plusMinutes(6));
|
||||||
|
|
||||||
|
assertTrue(closing4Min.minutesUntilClose() < 5,
|
||||||
|
"Lot closing in 4 min should be < 5 minutes");
|
||||||
|
assertTrue(closing5Min.minutesUntilClose() >= 5,
|
||||||
|
"Lot closing in 5 min should be >= 5 minutes");
|
||||||
|
assertTrue(closing6Min.minutesUntilClose() > 5,
|
||||||
|
"Lot closing in 6 min should be > 5 minutes");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(6)
|
||||||
|
@DisplayName("Should identify lots closing within 30 minutes (dashboard threshold)")
|
||||||
|
void testDashboardClosingThreshold() {
|
||||||
|
var closing20Min = createLot(LocalDateTime.now().plusMinutes(20));
|
||||||
|
var closing31Min = createLot(LocalDateTime.now().plusMinutes(31)); // Use 31 to avoid boundary timing issue
|
||||||
|
var closing40Min = createLot(LocalDateTime.now().plusMinutes(40));
|
||||||
|
|
||||||
|
assertTrue(closing20Min.minutesUntilClose() < 30,
|
||||||
|
"Lot closing in 20 min should be < 30 minutes");
|
||||||
|
assertTrue(closing31Min.minutesUntilClose() >= 30,
|
||||||
|
"Lot closing in 31 min should be >= 30 minutes");
|
||||||
|
assertTrue(closing40Min.minutesUntilClose() > 30,
|
||||||
|
"Lot closing in 40 min should be > 30 minutes");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(7)
|
||||||
|
@DisplayName("Should calculate correctly for lots closing soon (boundary cases)")
|
||||||
|
void testBoundaryCases() {
|
||||||
|
// Just closed (< 1 minute ago)
|
||||||
|
var justClosed = createLot(LocalDateTime.now().minusSeconds(30));
|
||||||
|
assertTrue(justClosed.minutesUntilClose() <= 0, "Just closed should be <= 0");
|
||||||
|
|
||||||
|
// Closing very soon (< 1 minute)
|
||||||
|
var closingVerySoon = createLot(LocalDateTime.now().plusSeconds(30));
|
||||||
|
assertTrue(closingVerySoon.minutesUntilClose() < 1, "Closing in 30 sec should be < 1 minute");
|
||||||
|
|
||||||
|
// Closing in exactly 1 hour
|
||||||
|
var closing1Hour = createLot(LocalDateTime.now().plusHours(1));
|
||||||
|
long minutes1Hour = closing1Hour.minutesUntilClose();
|
||||||
|
assertTrue(minutes1Hour >= 59 && minutes1Hour <= 61,
|
||||||
|
"Closing in 1 hour should be ~60 minutes, was: " + minutes1Hour);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
@Order(8)
|
||||||
|
@DisplayName("Multiple lots should sort correctly by urgency")
|
||||||
|
void testSortingByUrgency() {
|
||||||
|
var lot5Min = createLot(LocalDateTime.now().plusMinutes(5));
|
||||||
|
var lot30Min = createLot(LocalDateTime.now().plusMinutes(30));
|
||||||
|
var lot1Hour = createLot(LocalDateTime.now().plusHours(1));
|
||||||
|
var lot3Hours = createLot(LocalDateTime.now().plusHours(3));
|
||||||
|
|
||||||
|
var lots = java.util.List.of(lot3Hours, lot30Min, lot5Min, lot1Hour);
|
||||||
|
var sorted = lots.stream()
|
||||||
|
.sorted((a, b) -> Long.compare(a.minutesUntilClose(), b.minutesUntilClose()))
|
||||||
|
.toList();
|
||||||
|
|
||||||
|
assertEquals(lot5Min, sorted.get(0), "Most urgent should be first");
|
||||||
|
assertEquals(lot30Min, sorted.get(1), "Second most urgent");
|
||||||
|
assertEquals(lot1Hour, sorted.get(2), "Third most urgent");
|
||||||
|
assertEquals(lot3Hours, sorted.get(3), "Least urgent should be last");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper method
|
||||||
|
private Lot createLot(LocalDateTime closingTime) {
|
||||||
|
return Lot.basic(100, 1001, "Test Item", "", "", "", 0, "General",
|
||||||
|
100.0, "EUR", "http://test.com/1001", closingTime, false);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -22,6 +22,13 @@ class DatabaseServiceTest {
|
|||||||
|
|
||||||
@BeforeAll
|
@BeforeAll
|
||||||
void setUp() throws SQLException {
|
void setUp() throws SQLException {
|
||||||
|
// Load SQLite JDBC driver
|
||||||
|
try {
|
||||||
|
Class.forName("org.sqlite.JDBC");
|
||||||
|
} catch (ClassNotFoundException e) {
|
||||||
|
throw new SQLException("SQLite JDBC driver not found", e);
|
||||||
|
}
|
||||||
|
|
||||||
testDbPath = "test_database_" + System.currentTimeMillis() + ".db";
|
testDbPath = "test_database_" + System.currentTimeMillis() + ".db";
|
||||||
db = new DatabaseService(testDbPath);
|
db = new DatabaseService(testDbPath);
|
||||||
db.ensureSchema();
|
db.ensureSchema();
|
||||||
@@ -350,7 +357,7 @@ class DatabaseServiceTest {
|
|||||||
100.0, "EUR", "https://example.com/" + i, null, false
|
100.0, "EUR", "https://example.com/" + i, null, false
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
fail("Thread 1 failed: " + e.getMessage());
|
fail("Thread 1 failed: " + e.getMessage());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -363,7 +370,7 @@ class DatabaseServiceTest {
|
|||||||
200.0, "EUR", "https://example.com/" + i, null, false
|
200.0, "EUR", "https://example.com/" + i, null, false
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
fail("Thread 2 failed: " + e.getMessage());
|
fail("Thread 2 failed: " + e.getMessage());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -20,37 +20,51 @@ class ImageProcessingServiceTest {
|
|||||||
private DatabaseService mockDb;
|
private DatabaseService mockDb;
|
||||||
private ObjectDetectionService mockDetector;
|
private ObjectDetectionService mockDetector;
|
||||||
private ImageProcessingService service;
|
private ImageProcessingService service;
|
||||||
|
private java.io.File testImage;
|
||||||
|
|
||||||
@BeforeEach
|
@BeforeEach
|
||||||
void setUp() {
|
void setUp() throws Exception {
|
||||||
mockDb = mock(DatabaseService.class);
|
mockDb = mock(DatabaseService.class);
|
||||||
mockDetector = mock(ObjectDetectionService.class);
|
mockDetector = mock(ObjectDetectionService.class);
|
||||||
service = new ImageProcessingService(mockDb, mockDetector);
|
service = new ImageProcessingService(mockDb, mockDetector);
|
||||||
|
|
||||||
|
// Create a temporary test image file
|
||||||
|
testImage = java.io.File.createTempFile("test_image_", ".jpg");
|
||||||
|
testImage.deleteOnExit();
|
||||||
|
// Write minimal JPEG header to make it a valid file
|
||||||
|
try (var out = new java.io.FileOutputStream(testImage)) {
|
||||||
|
out.write(new byte[]{(byte)0xFF, (byte)0xD8, (byte)0xFF, (byte)0xE0});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
@DisplayName("Should process single image and update labels")
|
@DisplayName("Should process single image and update labels")
|
||||||
void testProcessImage() throws SQLException {
|
void testProcessImage() throws SQLException {
|
||||||
// Mock object detection
|
// Normalize path (convert backslashes to forward slashes)
|
||||||
when(mockDetector.detectObjects("/path/to/image.jpg"))
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
|
// Mock object detection with normalized path
|
||||||
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenReturn(List.of("car", "vehicle"));
|
.thenReturn(List.of("car", "vehicle"));
|
||||||
|
|
||||||
// Process image
|
// Process image
|
||||||
boolean result = service.processImage(1, "/path/to/image.jpg", 12345);
|
boolean result = service.processImage(1, testImage.getAbsolutePath(), 12345);
|
||||||
|
|
||||||
// Verify success
|
// Verify success
|
||||||
assertTrue(result);
|
assertTrue(result);
|
||||||
verify(mockDetector).detectObjects("/path/to/image.jpg");
|
verify(mockDetector).detectObjects(normalizedPath);
|
||||||
verify(mockDb).updateImageLabels(1, List.of("car", "vehicle"));
|
verify(mockDb).updateImageLabels(1, List.of("car", "vehicle"));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
@DisplayName("Should handle empty detection results")
|
@DisplayName("Should handle empty detection results")
|
||||||
void testProcessImageWithNoDetections() throws SQLException {
|
void testProcessImageWithNoDetections() throws SQLException {
|
||||||
when(mockDetector.detectObjects(anyString()))
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenReturn(List.of());
|
.thenReturn(List.of());
|
||||||
|
|
||||||
boolean result = service.processImage(2, "/path/to/image2.jpg", 12346);
|
boolean result = service.processImage(2, testImage.getAbsolutePath(), 12346);
|
||||||
|
|
||||||
assertTrue(result);
|
assertTrue(result);
|
||||||
verify(mockDb).updateImageLabels(2, List.of());
|
verify(mockDb).updateImageLabels(2, List.of());
|
||||||
@@ -58,15 +72,17 @@ class ImageProcessingServiceTest {
|
|||||||
|
|
||||||
@Test
|
@Test
|
||||||
@DisplayName("Should handle database error gracefully")
|
@DisplayName("Should handle database error gracefully")
|
||||||
void testProcessImageDatabaseError() throws SQLException {
|
void testProcessImageDatabaseError() {
|
||||||
when(mockDetector.detectObjects(anyString()))
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenReturn(List.of("object"));
|
.thenReturn(List.of("object"));
|
||||||
|
|
||||||
doThrow(new SQLException("Database error"))
|
doThrow(new RuntimeException("Database error"))
|
||||||
.when(mockDb).updateImageLabels(anyInt(), anyList());
|
.when(mockDb).updateImageLabels(anyInt(), anyList());
|
||||||
|
|
||||||
// Should return false on error
|
// Should return false on error
|
||||||
boolean result = service.processImage(3, "/path/to/image3.jpg", 12347);
|
boolean result = service.processImage(3, testImage.getAbsolutePath(), 12347);
|
||||||
assertFalse(result);
|
assertFalse(result);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -84,13 +100,15 @@ class ImageProcessingServiceTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("Should process pending images batch")
|
@DisplayName("Should process pending images batch")
|
||||||
void testProcessPendingImages() throws SQLException {
|
void testProcessPendingImages() throws SQLException {
|
||||||
// Mock pending images from database
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
|
// Mock pending images from database - use real test image path
|
||||||
when(mockDb.getImagesNeedingDetection()).thenReturn(List.of(
|
when(mockDb.getImagesNeedingDetection()).thenReturn(List.of(
|
||||||
new DatabaseService.ImageDetectionRecord(1, 100L, "/images/100/001.jpg"),
|
new DatabaseService.ImageDetectionRecord(1, 100L, testImage.getAbsolutePath()),
|
||||||
new DatabaseService.ImageDetectionRecord(2, 101L, "/images/101/001.jpg")
|
new DatabaseService.ImageDetectionRecord(2, 101L, testImage.getAbsolutePath())
|
||||||
));
|
));
|
||||||
|
|
||||||
when(mockDetector.detectObjects(anyString()))
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenReturn(List.of("item1"))
|
.thenReturn(List.of("item1"))
|
||||||
.thenReturn(List.of("item2"));
|
.thenReturn(List.of("item2"));
|
||||||
|
|
||||||
@@ -103,7 +121,7 @@ class ImageProcessingServiceTest {
|
|||||||
|
|
||||||
// Verify all images were processed
|
// Verify all images were processed
|
||||||
verify(mockDb).getImagesNeedingDetection();
|
verify(mockDb).getImagesNeedingDetection();
|
||||||
verify(mockDetector, times(2)).detectObjects(anyString());
|
verify(mockDetector, times(2)).detectObjects(normalizedPath);
|
||||||
verify(mockDb, times(2)).updateImageLabels(anyInt(), anyList());
|
verify(mockDb, times(2)).updateImageLabels(anyInt(), anyList());
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -121,15 +139,16 @@ class ImageProcessingServiceTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("Should continue processing after single image failure")
|
@DisplayName("Should continue processing after single image failure")
|
||||||
void testProcessPendingImagesWithFailure() throws SQLException {
|
void testProcessPendingImagesWithFailure() throws SQLException {
|
||||||
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
when(mockDb.getImagesNeedingDetection()).thenReturn(List.of(
|
when(mockDb.getImagesNeedingDetection()).thenReturn(List.of(
|
||||||
new DatabaseService.ImageDetectionRecord(1, 100L, "/images/100/001.jpg"),
|
new DatabaseService.ImageDetectionRecord(1, 100L, testImage.getAbsolutePath()),
|
||||||
new DatabaseService.ImageDetectionRecord(2, 101L, "/images/101/001.jpg")
|
new DatabaseService.ImageDetectionRecord(2, 101L, testImage.getAbsolutePath())
|
||||||
));
|
));
|
||||||
|
|
||||||
// First image fails, second succeeds
|
// First image fails, second succeeds
|
||||||
when(mockDetector.detectObjects("/images/100/001.jpg"))
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenThrow(new RuntimeException("Detection error"));
|
.thenThrow(new RuntimeException("Detection error"))
|
||||||
when(mockDetector.detectObjects("/images/101/001.jpg"))
|
|
||||||
.thenReturn(List.of("item"));
|
.thenReturn(List.of("item"));
|
||||||
|
|
||||||
when(mockDb.getImageLabels(2))
|
when(mockDb.getImageLabels(2))
|
||||||
@@ -138,14 +157,14 @@ class ImageProcessingServiceTest {
|
|||||||
service.processPendingImages();
|
service.processPendingImages();
|
||||||
|
|
||||||
// Verify second image was still processed
|
// Verify second image was still processed
|
||||||
verify(mockDetector, times(2)).detectObjects(anyString());
|
verify(mockDetector, times(2)).detectObjects(normalizedPath);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
@DisplayName("Should handle database query error in batch processing")
|
@DisplayName("Should handle database query error in batch processing")
|
||||||
void testProcessPendingImagesDatabaseError() throws SQLException {
|
void testProcessPendingImagesDatabaseError() {
|
||||||
when(mockDb.getImagesNeedingDetection())
|
when(mockDb.getImagesNeedingDetection())
|
||||||
.thenThrow(new SQLException("Database connection failed"));
|
.thenThrow(new RuntimeException("Database connection failed"));
|
||||||
|
|
||||||
// Should not throw exception
|
// Should not throw exception
|
||||||
assertDoesNotThrow(() -> service.processPendingImages());
|
assertDoesNotThrow(() -> service.processPendingImages());
|
||||||
@@ -154,10 +173,12 @@ class ImageProcessingServiceTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("Should process images with multiple detected objects")
|
@DisplayName("Should process images with multiple detected objects")
|
||||||
void testProcessImageMultipleDetections() throws SQLException {
|
void testProcessImageMultipleDetections() throws SQLException {
|
||||||
when(mockDetector.detectObjects(anyString()))
|
String normalizedPath = testImage.getAbsolutePath().replace('\\', '/');
|
||||||
|
|
||||||
|
when(mockDetector.detectObjects(normalizedPath))
|
||||||
.thenReturn(List.of("car", "truck", "vehicle", "road"));
|
.thenReturn(List.of("car", "truck", "vehicle", "road"));
|
||||||
|
|
||||||
boolean result = service.processImage(5, "/path/to/image5.jpg", 12349);
|
boolean result = service.processImage(5, testImage.getAbsolutePath(), 12349);
|
||||||
|
|
||||||
assertTrue(result);
|
assertTrue(result);
|
||||||
verify(mockDb).updateImageLabels(5, List.of("car", "truck", "vehicle", "road"));
|
verify(mockDb).updateImageLabels(5, List.of("car", "truck", "vehicle", "road"));
|
||||||
|
|||||||
@@ -370,11 +370,11 @@ class IntegrationTest {
|
|||||||
"https://example.com/60" + i, "A1", 5, null
|
"https://example.com/60" + i, "A1", 5, null
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
fail("Auction thread failed: " + e.getMessage());
|
fail("Auction thread failed: " + e.getMessage());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
var lotThread = new Thread(() -> {
|
var lotThread = new Thread(() -> {
|
||||||
try {
|
try {
|
||||||
for (var i = 0; i < 10; i++) {
|
for (var i = 0; i < 10; i++) {
|
||||||
@@ -383,7 +383,7 @@ class IntegrationTest {
|
|||||||
100.0 * i, "EUR", "https://example.com/70" + i, null, false
|
100.0 * i, "EUR", "https://example.com/70" + i, null, false
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
} catch (SQLException e) {
|
} catch (Exception e) {
|
||||||
fail("Lot thread failed: " + e.getMessage());
|
fail("Lot thread failed: " + e.getMessage());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -21,7 +21,6 @@ class ObjectDetectionServiceTest {
|
|||||||
@Test
|
@Test
|
||||||
@DisplayName("Should initialize with missing YOLO models (disabled mode)")
|
@DisplayName("Should initialize with missing YOLO models (disabled mode)")
|
||||||
void testInitializeWithoutModels() throws IOException {
|
void testInitializeWithoutModels() throws IOException {
|
||||||
// When models don't exist, service should initialize in disabled mode
|
|
||||||
ObjectDetectionService service = new ObjectDetectionService(
|
ObjectDetectionService service = new ObjectDetectionService(
|
||||||
"non_existent.cfg",
|
"non_existent.cfg",
|
||||||
"non_existent.weights",
|
"non_existent.weights",
|
||||||
@@ -82,9 +81,8 @@ class ObjectDetectionServiceTest {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
@DisplayName("Should throw IOException when model files exist but OpenCV fails to load")
|
@DisplayName("Should gracefully handle when model files exist but OpenCV fails to load")
|
||||||
void testInitializeWithValidModels() throws IOException {
|
void testInitializeWithValidModels() throws IOException {
|
||||||
// Create dummy model files for testing initialization
|
|
||||||
var cfgPath = Paths.get(TEST_CFG);
|
var cfgPath = Paths.get(TEST_CFG);
|
||||||
var weightsPath = Paths.get(TEST_WEIGHTS);
|
var weightsPath = Paths.get(TEST_WEIGHTS);
|
||||||
var classesPath = Paths.get(TEST_CLASSES);
|
var classesPath = Paths.get(TEST_CLASSES);
|
||||||
@@ -95,10 +93,11 @@ class ObjectDetectionServiceTest {
|
|||||||
Files.writeString(classesPath, "person\ncar\ntruck\n");
|
Files.writeString(classesPath, "person\ncar\ntruck\n");
|
||||||
|
|
||||||
// When files exist but OpenCV native library isn't loaded,
|
// When files exist but OpenCV native library isn't loaded,
|
||||||
// constructor should throw IOException wrapping the UnsatisfiedLinkError
|
// service should construct successfully but be disabled (handled in @PostConstruct)
|
||||||
assertThrows(IOException.class, () -> {
|
var service = new ObjectDetectionService(TEST_CFG, TEST_WEIGHTS, TEST_CLASSES);
|
||||||
new ObjectDetectionService(TEST_CFG, TEST_WEIGHTS, TEST_CLASSES);
|
// Service is created, but init() handles failures gracefully
|
||||||
});
|
// detectObjects should return empty list when disabled
|
||||||
|
assertNotNull(service);
|
||||||
} finally {
|
} finally {
|
||||||
Files.deleteIfExists(cfgPath);
|
Files.deleteIfExists(cfgPath);
|
||||||
Files.deleteIfExists(weightsPath);
|
Files.deleteIfExists(weightsPath);
|
||||||
|
|||||||
@@ -16,366 +16,365 @@ import static org.junit.jupiter.api.Assertions.*;
|
|||||||
*/
|
*/
|
||||||
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
|
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
|
||||||
class TroostwijkMonitorTest {
|
class TroostwijkMonitorTest {
|
||||||
|
|
||||||
private String testDbPath;
|
private String testDbPath;
|
||||||
private TroostwijkMonitor monitor;
|
private TroostwijkMonitor monitor;
|
||||||
|
|
||||||
@BeforeAll
|
@BeforeAll
|
||||||
void setUp() throws SQLException, IOException {
|
void setUp() throws SQLException, IOException {
|
||||||
testDbPath = "test_monitor_" + System.currentTimeMillis() + ".db";
|
testDbPath = "test_monitor_" + System.currentTimeMillis() + ".db";
|
||||||
|
|
||||||
// Initialize with non-existent YOLO models (disabled mode)
|
monitor = new TroostwijkMonitor(
|
||||||
monitor = new TroostwijkMonitor(
|
testDbPath,
|
||||||
testDbPath,
|
"desktop",
|
||||||
"desktop",
|
"non_existent.cfg",
|
||||||
"non_existent.cfg",
|
"non_existent.weights",
|
||||||
"non_existent.weights",
|
"non_existent.txt"
|
||||||
"non_existent.txt"
|
);
|
||||||
);
|
}
|
||||||
}
|
|
||||||
|
@AfterAll
|
||||||
@AfterAll
|
void tearDown() throws Exception {
|
||||||
void tearDown() throws Exception {
|
Files.deleteIfExists(Paths.get(testDbPath));
|
||||||
Files.deleteIfExists(Paths.get(testDbPath));
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should initialize monitor successfully")
|
||||||
@DisplayName("Should initialize monitor successfully")
|
void testMonitorInitialization() {
|
||||||
void testMonitorInitialization() {
|
assertNotNull(monitor);
|
||||||
assertNotNull(monitor);
|
assertNotNull(monitor.getDb());
|
||||||
assertNotNull(monitor.getDb());
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should print database stats without error")
|
||||||
@DisplayName("Should print database stats without error")
|
void testPrintDatabaseStats() {
|
||||||
void testPrintDatabaseStats() {
|
assertDoesNotThrow(() -> monitor.printDatabaseStats());
|
||||||
assertDoesNotThrow(() -> monitor.printDatabaseStats());
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should process pending images without error")
|
||||||
@DisplayName("Should process pending images without error")
|
void testProcessPendingImages() {
|
||||||
void testProcessPendingImages() {
|
assertDoesNotThrow(() -> monitor.processPendingImages());
|
||||||
assertDoesNotThrow(() -> monitor.processPendingImages());
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should handle empty database gracefully")
|
||||||
@DisplayName("Should handle empty database gracefully")
|
void testEmptyDatabaseHandling() throws SQLException {
|
||||||
void testEmptyDatabaseHandling() throws SQLException {
|
var auctions = monitor.getDb().getAllAuctions();
|
||||||
var auctions = monitor.getDb().getAllAuctions();
|
var lots = monitor.getDb().getAllLots();
|
||||||
var lots = monitor.getDb().getAllLots();
|
|
||||||
|
assertNotNull(auctions);
|
||||||
assertNotNull(auctions);
|
assertNotNull(lots);
|
||||||
assertNotNull(lots);
|
assertTrue(auctions.isEmpty() || auctions.size() >= 0);
|
||||||
assertTrue(auctions.isEmpty() || auctions.size() >= 0);
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should track lots in database")
|
||||||
@DisplayName("Should track lots in database")
|
void testLotTracking() throws SQLException {
|
||||||
void testLotTracking() throws SQLException {
|
// Insert test lot
|
||||||
// Insert test lot
|
var lot = Lot.basic(
|
||||||
var lot = Lot.basic(
|
11111, 22222,
|
||||||
11111, 22222,
|
"Test Forklift",
|
||||||
"Test Forklift",
|
"Electric forklift in good condition",
|
||||||
"Electric forklift in good condition",
|
"Toyota",
|
||||||
"Toyota",
|
"Electric",
|
||||||
"Electric",
|
2020,
|
||||||
2020,
|
"Machinery",
|
||||||
"Machinery",
|
1500.00,
|
||||||
1500.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/22222",
|
||||||
"https://example.com/lot/22222",
|
LocalDateTime.now().plusDays(1),
|
||||||
LocalDateTime.now().plusDays(1),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(lot);
|
||||||
monitor.getDb().upsertLot(lot);
|
|
||||||
|
var lots = monitor.getDb().getAllLots();
|
||||||
var lots = monitor.getDb().getAllLots();
|
assertTrue(lots.stream().anyMatch(l -> l.lotId() == 22222));
|
||||||
assertTrue(lots.stream().anyMatch(l -> l.lotId() == 22222));
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should monitor lots closing soon")
|
||||||
@DisplayName("Should monitor lots closing soon")
|
void testClosingSoonMonitoring() throws SQLException {
|
||||||
void testClosingSoonMonitoring() throws SQLException {
|
// Insert lot closing in 4 minutes
|
||||||
// Insert lot closing in 4 minutes
|
var closingSoon = Lot.basic(
|
||||||
var closingSoon = Lot.basic(
|
33333, 44444,
|
||||||
33333, 44444,
|
"Closing Soon Item",
|
||||||
"Closing Soon Item",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
100.00,
|
||||||
100.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/44444",
|
||||||
"https://example.com/lot/44444",
|
LocalDateTime.now().plusMinutes(4),
|
||||||
LocalDateTime.now().plusMinutes(4),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(closingSoon);
|
||||||
monitor.getDb().upsertLot(closingSoon);
|
|
||||||
|
var lots = monitor.getDb().getActiveLots();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
var found = lots.stream()
|
||||||
var found = lots.stream()
|
.filter(l -> l.lotId() == 44444)
|
||||||
.filter(l -> l.lotId() == 44444)
|
.findFirst()
|
||||||
.findFirst()
|
.orElse(null);
|
||||||
.orElse(null);
|
|
||||||
|
assertNotNull(found);
|
||||||
assertNotNull(found);
|
assertTrue(found.minutesUntilClose() < 30);
|
||||||
assertTrue(found.minutesUntilClose() < 30);
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should identify lots with time remaining")
|
||||||
@DisplayName("Should identify lots with time remaining")
|
void testTimeRemainingCalculation() throws SQLException {
|
||||||
void testTimeRemainingCalculation() throws SQLException {
|
var futureLot = Lot.basic(
|
||||||
var futureLot = Lot.basic(
|
55555, 66666,
|
||||||
55555, 66666,
|
"Future Lot",
|
||||||
"Future Lot",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
200.00,
|
||||||
200.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/66666",
|
||||||
"https://example.com/lot/66666",
|
LocalDateTime.now().plusHours(2),
|
||||||
LocalDateTime.now().plusHours(2),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(futureLot);
|
||||||
monitor.getDb().upsertLot(futureLot);
|
|
||||||
|
var lots = monitor.getDb().getActiveLots();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
var found = lots.stream()
|
||||||
var found = lots.stream()
|
.filter(l -> l.lotId() == 66666)
|
||||||
.filter(l -> l.lotId() == 66666)
|
.findFirst()
|
||||||
.findFirst()
|
.orElse(null);
|
||||||
.orElse(null);
|
|
||||||
|
assertNotNull(found);
|
||||||
assertNotNull(found);
|
assertTrue(found.minutesUntilClose() > 60);
|
||||||
assertTrue(found.minutesUntilClose() > 60);
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should handle lots without closing time")
|
||||||
@DisplayName("Should handle lots without closing time")
|
void testLotsWithoutClosingTime() throws SQLException {
|
||||||
void testLotsWithoutClosingTime() throws SQLException {
|
var noClosing = Lot.basic(
|
||||||
var noClosing = Lot.basic(
|
77777, 88888,
|
||||||
77777, 88888,
|
"No Closing Time",
|
||||||
"No Closing Time",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
150.00,
|
||||||
150.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/88888",
|
||||||
"https://example.com/lot/88888",
|
null,
|
||||||
null,
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(noClosing);
|
||||||
monitor.getDb().upsertLot(noClosing);
|
|
||||||
|
var lots = monitor.getDb().getActiveLots();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
var found = lots.stream()
|
||||||
var found = lots.stream()
|
.filter(l -> l.lotId() == 88888)
|
||||||
.filter(l -> l.lotId() == 88888)
|
.findFirst()
|
||||||
.findFirst()
|
.orElse(null);
|
||||||
.orElse(null);
|
|
||||||
|
assertNotNull(found);
|
||||||
assertNotNull(found);
|
assertNull(found.closingTime());
|
||||||
assertNull(found.closingTime());
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should track notification status")
|
||||||
@DisplayName("Should track notification status")
|
void testNotificationStatusTracking() throws SQLException {
|
||||||
void testNotificationStatusTracking() throws SQLException {
|
var lot = Lot.basic(
|
||||||
var lot = Lot.basic(
|
99999, 11110,
|
||||||
99999, 11110,
|
"Test Notification",
|
||||||
"Test Notification",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
100.00,
|
||||||
100.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/11110",
|
||||||
"https://example.com/lot/11110",
|
LocalDateTime.now().plusMinutes(3),
|
||||||
LocalDateTime.now().plusMinutes(3),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(lot);
|
||||||
monitor.getDb().upsertLot(lot);
|
|
||||||
|
// Update notification flag
|
||||||
// Update notification flag
|
var notified = Lot.basic(
|
||||||
var notified = Lot.basic(
|
99999, 11110,
|
||||||
99999, 11110,
|
"Test Notification",
|
||||||
"Test Notification",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
100.00,
|
||||||
100.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/11110",
|
||||||
"https://example.com/lot/11110",
|
LocalDateTime.now().plusMinutes(3),
|
||||||
LocalDateTime.now().plusMinutes(3),
|
true
|
||||||
true
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().updateLotNotificationFlags(notified);
|
||||||
monitor.getDb().updateLotNotificationFlags(notified);
|
|
||||||
|
var lots = monitor.getDb().getActiveLots();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
var found = lots.stream()
|
||||||
var found = lots.stream()
|
.filter(l -> l.lotId() == 11110)
|
||||||
.filter(l -> l.lotId() == 11110)
|
.findFirst()
|
||||||
.findFirst()
|
.orElse(null);
|
||||||
.orElse(null);
|
|
||||||
|
assertNotNull(found);
|
||||||
assertNotNull(found);
|
assertTrue(found.closingNotified());
|
||||||
assertTrue(found.closingNotified());
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should update bid amounts")
|
||||||
@DisplayName("Should update bid amounts")
|
void testBidAmountUpdates() throws SQLException {
|
||||||
void testBidAmountUpdates() throws SQLException {
|
var lot = Lot.basic(
|
||||||
var lot = Lot.basic(
|
12121, 13131,
|
||||||
12121, 13131,
|
"Bid Update Test",
|
||||||
"Bid Update Test",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
100.00,
|
||||||
100.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/13131",
|
||||||
"https://example.com/lot/13131",
|
LocalDateTime.now().plusDays(1),
|
||||||
LocalDateTime.now().plusDays(1),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().upsertLot(lot);
|
||||||
monitor.getDb().upsertLot(lot);
|
|
||||||
|
// Simulate bid increase
|
||||||
// Simulate bid increase
|
var higherBid = Lot.basic(
|
||||||
var higherBid = Lot.basic(
|
12121, 13131,
|
||||||
12121, 13131,
|
"Bid Update Test",
|
||||||
"Bid Update Test",
|
"Description",
|
||||||
"Description",
|
"",
|
||||||
"",
|
"",
|
||||||
"",
|
0,
|
||||||
0,
|
"Category",
|
||||||
"Category",
|
250.00,
|
||||||
250.00,
|
"EUR",
|
||||||
"EUR",
|
"https://example.com/lot/13131",
|
||||||
"https://example.com/lot/13131",
|
LocalDateTime.now().plusDays(1),
|
||||||
LocalDateTime.now().plusDays(1),
|
false
|
||||||
false
|
);
|
||||||
);
|
|
||||||
|
monitor.getDb().updateLotCurrentBid(higherBid);
|
||||||
monitor.getDb().updateLotCurrentBid(higherBid);
|
|
||||||
|
var lots = monitor.getDb().getActiveLots();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
var found = lots.stream()
|
||||||
var found = lots.stream()
|
.filter(l -> l.lotId() == 13131)
|
||||||
.filter(l -> l.lotId() == 13131)
|
.findFirst()
|
||||||
.findFirst()
|
.orElse(null);
|
||||||
.orElse(null);
|
|
||||||
|
assertNotNull(found);
|
||||||
assertNotNull(found);
|
assertEquals(250.00, found.currentBid(), 0.01);
|
||||||
assertEquals(250.00, found.currentBid(), 0.01);
|
}
|
||||||
}
|
|
||||||
|
@Test
|
||||||
@Test
|
@DisplayName("Should handle multiple concurrent lot updates")
|
||||||
@DisplayName("Should handle multiple concurrent lot updates")
|
void testConcurrentLotUpdates() throws InterruptedException, SQLException {
|
||||||
void testConcurrentLotUpdates() throws InterruptedException, SQLException {
|
Thread t1 = new Thread(() -> {
|
||||||
Thread t1 = new Thread(() -> {
|
try {
|
||||||
try {
|
for (int i = 0; i < 5; i++) {
|
||||||
for (int i = 0; i < 5; i++) {
|
monitor.getDb().upsertLot(Lot.basic(
|
||||||
monitor.getDb().upsertLot(Lot.basic(
|
20000 + i, 30000 + i, "Concurrent " + i, "Desc", "", "", 0, "Cat",
|
||||||
20000 + i, 30000 + i, "Concurrent " + i, "Desc", "", "", 0, "Cat",
|
100.0, "EUR", "https://example.com/" + i, null, false
|
||||||
100.0, "EUR", "https://example.com/" + i, null, false
|
));
|
||||||
));
|
|
||||||
}
|
|
||||||
} catch (SQLException e) {
|
|
||||||
fail("Thread 1 failed: " + e.getMessage());
|
|
||||||
}
|
}
|
||||||
});
|
} catch (Exception e) {
|
||||||
|
fail("Thread 1 failed: " + e.getMessage());
|
||||||
Thread t2 = new Thread(() -> {
|
}
|
||||||
try {
|
});
|
||||||
for (int i = 5; i < 10; i++) {
|
|
||||||
monitor.getDb().upsertLot(Lot.basic(
|
Thread t2 = new Thread(() -> {
|
||||||
20000 + i, 30000 + i, "Concurrent " + i, "Desc", "", "", 0, "Cat",
|
try {
|
||||||
200.0, "EUR", "https://example.com/" + i, null, false
|
for (int i = 5; i < 10; i++) {
|
||||||
));
|
monitor.getDb().upsertLot(Lot.basic(
|
||||||
}
|
20000 + i, 30000 + i, "Concurrent " + i, "Desc", "", "", 0, "Cat",
|
||||||
} catch (SQLException e) {
|
200.0, "EUR", "https://example.com/" + i, null, false
|
||||||
fail("Thread 2 failed: " + e.getMessage());
|
));
|
||||||
}
|
}
|
||||||
});
|
} catch (Exception e) {
|
||||||
|
fail("Thread 2 failed: " + e.getMessage());
|
||||||
t1.start();
|
}
|
||||||
t2.start();
|
});
|
||||||
t1.join();
|
|
||||||
t2.join();
|
t1.start();
|
||||||
|
t2.start();
|
||||||
var lots = monitor.getDb().getActiveLots();
|
t1.join();
|
||||||
long count = lots.stream()
|
t2.join();
|
||||||
.filter(l -> l.lotId() >= 30000 && l.lotId() < 30010)
|
|
||||||
.count();
|
var lots = monitor.getDb().getActiveLots();
|
||||||
|
long count = lots.stream()
|
||||||
assertTrue(count >= 10);
|
.filter(l -> l.lotId() >= 30000 && l.lotId() < 30010)
|
||||||
}
|
.count();
|
||||||
|
|
||||||
@Test
|
assertTrue(count >= 10);
|
||||||
@DisplayName("Should schedule monitoring without error")
|
}
|
||||||
void testScheduleMonitoring() {
|
|
||||||
// This just tests that scheduling doesn't throw
|
@Test
|
||||||
// Actual monitoring would run in background
|
@DisplayName("Should schedule monitoring without error")
|
||||||
assertDoesNotThrow(() -> {
|
void testScheduleMonitoring() {
|
||||||
// Don't actually start monitoring in test
|
// This just tests that scheduling doesn't throw
|
||||||
// Just verify monitor is ready
|
// Actual monitoring would run in background
|
||||||
assertNotNull(monitor);
|
assertDoesNotThrow(() -> {
|
||||||
});
|
// Don't actually start monitoring in test
|
||||||
}
|
// Just verify monitor is ready
|
||||||
|
assertNotNull(monitor);
|
||||||
@Test
|
});
|
||||||
@DisplayName("Should handle database with auctions and lots")
|
}
|
||||||
void testDatabaseWithData() throws SQLException {
|
|
||||||
// Insert auction
|
@Test
|
||||||
var auction = new AuctionInfo(
|
@DisplayName("Should handle database with auctions and lots")
|
||||||
40000,
|
void testDatabaseWithData() throws SQLException {
|
||||||
"Test Auction",
|
// Insert auction
|
||||||
"Amsterdam, NL",
|
var auction = new AuctionInfo(
|
||||||
"Amsterdam",
|
40000,
|
||||||
"NL",
|
"Test Auction",
|
||||||
"https://example.com/auction/40000",
|
"Amsterdam, NL",
|
||||||
"A7",
|
"Amsterdam",
|
||||||
10,
|
"NL",
|
||||||
LocalDateTime.now().plusDays(2)
|
"https://example.com/auction/40000",
|
||||||
);
|
"A7",
|
||||||
|
10,
|
||||||
monitor.getDb().upsertAuction(auction);
|
LocalDateTime.now().plusDays(2)
|
||||||
|
);
|
||||||
// Insert related lot
|
|
||||||
var lot = Lot.basic(
|
monitor.getDb().upsertAuction(auction);
|
||||||
40000, 50000,
|
|
||||||
"Test Lot",
|
// Insert related lot
|
||||||
"Description",
|
var lot = Lot.basic(
|
||||||
"",
|
40000, 50000,
|
||||||
"",
|
"Test Lot",
|
||||||
0,
|
"Description",
|
||||||
"Category",
|
"",
|
||||||
500.00,
|
"",
|
||||||
"EUR",
|
0,
|
||||||
"https://example.com/lot/50000",
|
"Category",
|
||||||
LocalDateTime.now().plusDays(2),
|
500.00,
|
||||||
false
|
"EUR",
|
||||||
);
|
"https://example.com/lot/50000",
|
||||||
|
LocalDateTime.now().plusDays(2),
|
||||||
monitor.getDb().upsertLot(lot);
|
false
|
||||||
|
);
|
||||||
// Verify
|
|
||||||
var auctions = monitor.getDb().getAllAuctions();
|
monitor.getDb().upsertLot(lot);
|
||||||
var lots = monitor.getDb().getAllLots();
|
|
||||||
|
// Verify
|
||||||
assertTrue(auctions.stream().anyMatch(a -> a.auctionId() == 40000));
|
var auctions = monitor.getDb().getAllAuctions();
|
||||||
assertTrue(lots.stream().anyMatch(l -> l.lotId() == 50000));
|
var lots = monitor.getDb().getAllLots();
|
||||||
}
|
|
||||||
|
assertTrue(auctions.stream().anyMatch(a -> a.auctionId() == 40000));
|
||||||
|
assertTrue(lots.stream().anyMatch(l -> l.lotId() == 50000));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -1,26 +0,0 @@
|
|||||||
Configure your devices to use the Pi-hole as their DNS server │
|
|
||||||
│ using: │
|
|
||||||
│ │
|
|
||||||
│ IPv4: 192.168.1.159 │
|
|
||||||
│ IPv6: fdc5:59a6:9ac1:f11f:2c86:25d3:6282:37ef │
|
|
||||||
│ If you have not done so already, the above IP should be set to │
|
|
||||||
│ static. │
|
|
||||||
│ View the web interface at http://pi.hole:80/admin or │
|
|
||||||
│ http://192.168.1.159:80/admin │
|
|
||||||
│ │
|
|
||||||
│ Your Admin Webpage login password is gYj7Enh- │
|
|
||||||
│ │
|
|
||||||
│ │
|
|
||||||
│ To allow your user to use all CLI functions without │
|
|
||||||
│ authentication, │
|
|
||||||
│ refer to https://docs.pi-hole.net/main/post-install/ │
|
|
||||||
├─────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
127.0.0.1
|
|
||||||
192.168.1.159
|
|
||||||
::1
|
|
||||||
fdc5:59a6:9ac1:f11f:2c86:25d3:6282:37ef
|
|
||||||
fdc5:59a6:9ac1:f11f:bd8c:6e87:65f0:243c
|
|
||||||
fe80::a05b:bbc6:d47f:3002%enp9s0
|
|
||||||
2IXD-XJN9-C337-1K4Y-BBEO-HV1F-3BVI
|
|
||||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user