Fix mock tests
This commit is contained in:
Tour
@@ -1,589 +0,0 @@
|
||||
# ✅ Quarkus Integration Complete
|
||||
|
||||
## 🎯 Summary
|
||||
|
||||
The Troostwijk Auction Monitor is now **fully integrated with Quarkus Framework** with all components production-ready.
|
||||
|
||||
---
|
||||
|
||||
## 📦 What You Added
|
||||
|
||||
✅ **Quarkus BOM** in pom.xml (version 3.17.7)
|
||||
✅ **application.properties** with configuration
|
||||
✅ **Dockerfile** for Quarkus fast-jar
|
||||
|
||||
## 🚀 What I Added
|
||||
|
||||
✅ **Quarkus Dependencies** - scheduler, health, rest
|
||||
✅ **QuarkusWorkflowScheduler** - @Scheduled workflows
|
||||
✅ **AuctionMonitorProducer** - CDI service producers
|
||||
✅ **AuctionMonitorResource** - Complete REST API
|
||||
✅ **AuctionMonitorHealthCheck** - Health probes
|
||||
✅ **docker-compose.yml** - Easy local deployment
|
||||
✅ **k8s/deployment.yaml** - Kubernetes manifests
|
||||
✅ **Complete Documentation** - 3 comprehensive guides
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ QUARKUS APPLICATION │
|
||||
├────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ HTTP Server (Port 8081) │
|
||||
│ ├─ /api/monitor/* → REST API endpoints │
|
||||
│ ├─ /health/* → Health check probes │
|
||||
│ └─ /q/dev/* → Dev UI (dev mode only) │
|
||||
│ │
|
||||
│ Scheduler (Quarkus @Scheduled) │
|
||||
│ ├─ Scraper Import → Every 30 minutes │
|
||||
│ ├─ Image Processing → Every 1 hour │
|
||||
│ ├─ Bid Monitoring → Every 15 minutes │
|
||||
│ └─ Closing Alerts → Every 5 minutes │
|
||||
│ │
|
||||
│ CDI Container (Dependency Injection) │
|
||||
│ ├─ DatabaseService (@Singleton) │
|
||||
│ ├─ NotificationService (@Singleton) │
|
||||
│ ├─ ObjectDetectionService (@Singleton) │
|
||||
│ └─ ImageProcessingService (@Singleton) │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
[SQLite DB] [Desktop/Email] [YOLO Models]
|
||||
cache.db Notifications Object Detection
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 How to Run
|
||||
|
||||
### 1. Development Mode (with Live Reload)
|
||||
|
||||
```bash
|
||||
mvn quarkus:dev
|
||||
|
||||
# Features:
|
||||
# ✓ Live reload (no restart needed)
|
||||
# ✓ Dev UI: http://localhost:8081/q/dev/
|
||||
# ✓ Continuous testing
|
||||
# ✓ Debug on port 5005
|
||||
```
|
||||
|
||||
### 2. Production JAR
|
||||
|
||||
```bash
|
||||
# Build
|
||||
mvn clean package
|
||||
|
||||
# Run
|
||||
java -jar target/quarkus-app/quarkus-run.jar
|
||||
```
|
||||
|
||||
### 3. Docker
|
||||
|
||||
```bash
|
||||
# Build
|
||||
docker build -t auction-monitor .
|
||||
|
||||
# Run
|
||||
docker run -p 8081:8081 \
|
||||
-v $(pwd)/data:/mnt/okcomputer/output \
|
||||
auction-monitor
|
||||
```
|
||||
|
||||
### 4. Docker Compose (Recommended)
|
||||
|
||||
```bash
|
||||
# Start
|
||||
docker-compose up -d
|
||||
|
||||
# Logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Stop
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
### 5. Kubernetes
|
||||
|
||||
```bash
|
||||
# Deploy
|
||||
kubectl apply -f k8s/deployment.yaml
|
||||
|
||||
# Port forward
|
||||
kubectl port-forward svc/auction-monitor 8081:8081 -n auction-monitor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📡 API Endpoints
|
||||
|
||||
Base URL: `http://localhost:8081`
|
||||
|
||||
### Status & Statistics
|
||||
|
||||
```bash
|
||||
# Get status
|
||||
curl http://localhost:8081/api/monitor/status
|
||||
|
||||
# Get statistics
|
||||
curl http://localhost:8081/api/monitor/statistics
|
||||
```
|
||||
|
||||
### Manual Triggers
|
||||
|
||||
```bash
|
||||
# Trigger scraper import
|
||||
curl -X POST http://localhost:8081/api/monitor/trigger/scraper-import
|
||||
|
||||
# Trigger image processing
|
||||
curl -X POST http://localhost:8081/api/monitor/trigger/image-processing
|
||||
|
||||
# Trigger bid monitoring
|
||||
curl -X POST http://localhost:8081/api/monitor/trigger/bid-monitoring
|
||||
|
||||
# Trigger closing alerts
|
||||
curl -X POST http://localhost:8081/api/monitor/trigger/closing-alerts
|
||||
```
|
||||
|
||||
### Data Access
|
||||
|
||||
```bash
|
||||
# List auctions
|
||||
curl http://localhost:8081/api/monitor/auctions
|
||||
|
||||
# Filter by country
|
||||
curl http://localhost:8081/api/monitor/auctions?country=NL
|
||||
|
||||
# List active lots
|
||||
curl http://localhost:8081/api/monitor/lots
|
||||
|
||||
# Lots closing soon
|
||||
curl http://localhost:8081/api/monitor/lots/closing-soon?minutes=30
|
||||
|
||||
# Get lot images
|
||||
curl http://localhost:8081/api/monitor/lots/12345/images
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Test notification
|
||||
curl -X POST http://localhost:8081/api/monitor/test-notification \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"message":"Test","title":"Test","priority":"0"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏥 Health Checks
|
||||
|
||||
```bash
|
||||
# Liveness (is app alive?)
|
||||
curl http://localhost:8081/health/live
|
||||
|
||||
# Readiness (is app ready?)
|
||||
curl http://localhost:8081/health/ready
|
||||
|
||||
# Startup (has app started?)
|
||||
curl http://localhost:8081/health/started
|
||||
|
||||
# All health checks
|
||||
curl http://localhost:8081/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Database
|
||||
export AUCTION_DATABASE_PATH=/path/to/cache.db
|
||||
export AUCTION_IMAGES_PATH=/path/to/images
|
||||
|
||||
# Notifications
|
||||
export AUCTION_NOTIFICATION_CONFIG=desktop
|
||||
# Or for email:
|
||||
export AUCTION_NOTIFICATION_CONFIG=smtp:user@gmail.com:password:recipient@example.com
|
||||
|
||||
# Workflow schedules (cron)
|
||||
export AUCTION_WORKFLOW_SCRAPER_IMPORT_CRON="0 */30 * * * ?"
|
||||
export AUCTION_WORKFLOW_IMAGE_PROCESSING_CRON="0 0 * * * ?"
|
||||
export AUCTION_WORKFLOW_BID_MONITORING_CRON="0 */15 * * * ?"
|
||||
export AUCTION_WORKFLOW_CLOSING_ALERTS_CRON="0 */5 * * * ?"
|
||||
|
||||
# HTTP
|
||||
export QUARKUS_HTTP_PORT=8081
|
||||
export QUARKUS_LOG_CONSOLE_LEVEL=INFO
|
||||
```
|
||||
|
||||
### Cron Expressions
|
||||
|
||||
| Expression | Meaning |
|
||||
|------------|---------|
|
||||
| `0 */30 * * * ?` | Every 30 minutes |
|
||||
| `0 0 * * * ?` | Every hour (at minute 0) |
|
||||
| `0 */15 * * * ?` | Every 15 minutes |
|
||||
| `0 */5 * * * ?` | Every 5 minutes |
|
||||
| `0 0 0 * * ?` | Daily at midnight |
|
||||
| `0 0 */2 * * ?` | Every 2 hours |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Performance
|
||||
|
||||
### Startup Time
|
||||
- **JVM Mode**: ~0.5 seconds
|
||||
- **Native**: ~0.014 seconds (with GraalVM)
|
||||
|
||||
### Memory
|
||||
- **JVM Mode**: ~50MB RSS
|
||||
- **Native**: ~15MB RSS
|
||||
|
||||
### Container Size
|
||||
- **Image**: ~150MB (with JRE)
|
||||
- **Native**: ~50MB (native executable)
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── src/main/java/com/auction/
|
||||
│ ├── QuarkusWorkflowScheduler.java # Scheduled workflows
|
||||
│ ├── AuctionMonitorProducer.java # CDI producers
|
||||
│ ├── AuctionMonitorResource.java # REST API
|
||||
│ ├── AuctionMonitorHealthCheck.java # Health checks
|
||||
│ ├── DatabaseService.java # Database operations
|
||||
│ ├── NotificationService.java # Notifications
|
||||
│ ├── ObjectDetectionService.java # YOLO detection
|
||||
│ ├── ImageProcessingService.java # Image processing
|
||||
│ ├── TroostwijkMonitor.java # Original monitor
|
||||
│ └── Main.java # Entry point (legacy)
|
||||
│
|
||||
├── src/main/resources/
|
||||
│ └── application.properties # Configuration
|
||||
│
|
||||
├── k8s/
|
||||
│ ├── deployment.yaml # Kubernetes manifests
|
||||
│ └── README.md # K8s guide
|
||||
│
|
||||
├── pom.xml # Maven config
|
||||
├── Dockerfile # Container build
|
||||
├── docker-compose.yml # Docker Compose
|
||||
│
|
||||
├── QUARKUS_GUIDE.md # Complete user guide
|
||||
├── QUARKUS_IMPLEMENTATION.md # Implementation details
|
||||
└── QUARKUS_COMPLETE.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### Primary Documentation
|
||||
|
||||
1. **QUARKUS_GUIDE.md** ⭐
|
||||
- Complete usage guide
|
||||
- All endpoints documented
|
||||
- Configuration examples
|
||||
- Deployment instructions
|
||||
|
||||
2. **QUARKUS_IMPLEMENTATION.md**
|
||||
- Technical implementation details
|
||||
- Architecture diagrams
|
||||
- Code structure
|
||||
- Design decisions
|
||||
|
||||
3. **QUARKUS_COMPLETE.md** (This file)
|
||||
- Quick reference
|
||||
- Summary of features
|
||||
- Quick start commands
|
||||
|
||||
### Supporting Documentation
|
||||
|
||||
4. **k8s/README.md** - Kubernetes deployment
|
||||
5. **docker-compose.yml** - Docker Compose reference
|
||||
6. **README.md** - Main project README
|
||||
7. **WORKFLOW_GUIDE.md** - Original workflow guide
|
||||
8. **TEST_SUITE_SUMMARY.md** - Test documentation
|
||||
|
||||
---
|
||||
|
||||
## ✨ Key Features
|
||||
|
||||
### 1. **Quarkus Scheduler**
|
||||
- ✅ Cron-based scheduling
|
||||
- ✅ Configuration via properties
|
||||
- ✅ No manual thread management
|
||||
- ✅ Timezone support
|
||||
|
||||
### 2. **REST API**
|
||||
- ✅ Status and statistics endpoints
|
||||
- ✅ Manual workflow triggers
|
||||
- ✅ Data access endpoints
|
||||
- ✅ Test notification endpoint
|
||||
|
||||
### 3. **Health Checks**
|
||||
- ✅ Liveness probe (Kubernetes)
|
||||
- ✅ Readiness probe (Kubernetes)
|
||||
- ✅ Startup probe (Kubernetes)
|
||||
- ✅ Database connectivity check
|
||||
|
||||
### 4. **CDI/Dependency Injection**
|
||||
- ✅ Type-safe injection
|
||||
- ✅ Singleton services
|
||||
- ✅ Configuration injection
|
||||
- ✅ Centralized producers
|
||||
|
||||
### 5. **Docker Support**
|
||||
- ✅ Optimized Dockerfile
|
||||
- ✅ Fast-jar packaging
|
||||
- ✅ Docker Compose config
|
||||
- ✅ Health checks included
|
||||
|
||||
### 6. **Kubernetes Support**
|
||||
- ✅ Complete manifests
|
||||
- ✅ ConfigMap for configuration
|
||||
- ✅ Secrets for credentials
|
||||
- ✅ Persistent storage
|
||||
- ✅ Auto-scaling (HPA)
|
||||
- ✅ Ingress configuration
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### Quick Test
|
||||
|
||||
```bash
|
||||
# Start application
|
||||
mvn quarkus:dev
|
||||
|
||||
# Test status endpoint
|
||||
curl http://localhost:8081/api/monitor/status
|
||||
|
||||
# Test health check
|
||||
curl http://localhost:8081/health/live
|
||||
|
||||
# Trigger workflow
|
||||
curl -X POST http://localhost:8081/api/monitor/trigger/scraper-import
|
||||
|
||||
# Test notification
|
||||
curl -X POST http://localhost:8081/api/monitor/test-notification \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"message":"Hello from Quarkus!"}'
|
||||
```
|
||||
|
||||
### Docker Test
|
||||
|
||||
```bash
|
||||
# Start with Docker Compose
|
||||
docker-compose up -d
|
||||
|
||||
# Wait for startup
|
||||
sleep 10
|
||||
|
||||
# Test health
|
||||
curl http://localhost:8081/health/ready
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Stop
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Issue: Port 8081 already in use
|
||||
|
||||
```bash
|
||||
# Change port
|
||||
export QUARKUS_HTTP_PORT=8082
|
||||
mvn quarkus:dev
|
||||
|
||||
# Or in application.properties
|
||||
quarkus.http.port=8082
|
||||
```
|
||||
|
||||
### Issue: Database not found
|
||||
|
||||
```bash
|
||||
# Check path
|
||||
ls -la C:/mnt/okcomputer/output/cache.db
|
||||
|
||||
# Create directory
|
||||
mkdir -p C:/mnt/okcomputer/output
|
||||
|
||||
# Check permissions
|
||||
chmod 755 C:/mnt/okcomputer/output
|
||||
```
|
||||
|
||||
### Issue: Scheduler not running
|
||||
|
||||
```bash
|
||||
# Enable debug logging
|
||||
export QUARKUS_LOG_CATEGORY__IO_QUARKUS_SCHEDULER__LEVEL=DEBUG
|
||||
|
||||
# Check scheduler config
|
||||
curl http://localhost:8081/q/dev/io.quarkus.quarkus-scheduler/scheduled-methods
|
||||
```
|
||||
|
||||
### Issue: Health check failing
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs auction-monitor
|
||||
|
||||
# Test directly
|
||||
curl -v http://localhost:8081/health/ready
|
||||
|
||||
# Verify database
|
||||
sqlite3 C:/mnt/okcomputer/output/cache.db "SELECT COUNT(*) FROM auctions;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Next Steps
|
||||
|
||||
### Immediate Actions
|
||||
|
||||
1. **Build and Run**
|
||||
```bash
|
||||
mvn clean package
|
||||
java -jar target/quarkus-app/quarkus-run.jar
|
||||
```
|
||||
|
||||
2. **Test API**
|
||||
```bash
|
||||
curl http://localhost:8081/api/monitor/status
|
||||
```
|
||||
|
||||
3. **Deploy**
|
||||
```bash
|
||||
docker-compose up -d
|
||||
# Or
|
||||
kubectl apply -f k8s/deployment.yaml
|
||||
```
|
||||
|
||||
### Optional Enhancements
|
||||
|
||||
1. **Native Image** (for ultra-fast startup)
|
||||
```bash
|
||||
mvn package -Pnative
|
||||
```
|
||||
|
||||
2. **Metrics** (add Micrometer)
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>io.quarkus</groupId>
|
||||
<artifactId>quarkus-micrometer-registry-prometheus</artifactId>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
3. **OpenAPI** (API documentation)
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>io.quarkus</groupId>
|
||||
<artifactId>quarkus-smallrye-openapi</artifactId>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
4. **Tracing** (distributed tracing)
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>io.quarkus</groupId>
|
||||
<artifactId>quarkus-opentelemetry</artifactId>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Comparison: Before vs After
|
||||
|
||||
| Aspect | Before | After (Quarkus) |
|
||||
|--------|--------|-----------------|
|
||||
| **Framework** | Plain Java | Quarkus |
|
||||
| **Startup** | ~3-5s | ~0.5s |
|
||||
| **Memory** | ~200MB | ~50MB |
|
||||
| **Scheduling** | Manual ExecutorService | @Scheduled |
|
||||
| **DI** | Manual | CDI @Inject |
|
||||
| **REST API** | ❌ None | ✅ Full API |
|
||||
| **Health** | ❌ None | ✅ Probes |
|
||||
| **Config** | Hard-coded | Properties |
|
||||
| **Dev Mode** | Manual restart | Live reload |
|
||||
| **Docker** | Basic | Optimized |
|
||||
| **K8s** | Not ready | Ready |
|
||||
| **Monitoring** | Logs only | REST + Health |
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Summary
|
||||
|
||||
### ✅ What Works
|
||||
|
||||
- **Quarkus Framework** - Fully integrated and working
|
||||
- **Scheduled Workflows** - Running on cron expressions
|
||||
- **REST API** - All endpoints functional
|
||||
- **Health Checks** - Kubernetes ready
|
||||
- **Docker** - Optimized image and compose file
|
||||
- **Kubernetes** - Complete deployment manifests
|
||||
- **Configuration** - Externalized and flexible
|
||||
- **Documentation** - Comprehensive guides
|
||||
|
||||
### 🚀 Ready for Production
|
||||
|
||||
The application is now:
|
||||
- ✅ Cloud-native (Kubernetes)
|
||||
- ✅ Container-ready (Docker)
|
||||
- ✅ API-enabled (REST)
|
||||
- ✅ Observable (Health checks)
|
||||
- ✅ Configurable (Properties)
|
||||
- ✅ Fast (0.5s startup)
|
||||
- ✅ Efficient (50MB memory)
|
||||
- ✅ Documented (3 guides)
|
||||
|
||||
### 🎯 Quick Commands
|
||||
|
||||
```bash
|
||||
# Development
|
||||
mvn quarkus:dev
|
||||
|
||||
# Production
|
||||
mvn clean package && java -jar target/quarkus-app/quarkus-run.jar
|
||||
|
||||
# Docker
|
||||
docker-compose up -d
|
||||
|
||||
# Kubernetes
|
||||
kubectl apply -f k8s/deployment.yaml
|
||||
|
||||
# Test
|
||||
curl http://localhost:8081/api/monitor/status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**🎊 Quarkus integration is complete and production-ready! 🎊**
|
||||
|
||||
---
|
||||
|
||||
## 📞 Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check **QUARKUS_GUIDE.md** for detailed usage
|
||||
2. Check **QUARKUS_IMPLEMENTATION.md** for technical details
|
||||
3. Check **k8s/README.md** for Kubernetes deployment
|
||||
4. Review logs: `docker-compose logs -f`
|
||||
5. Test health: `curl http://localhost:8081/health`
|
||||
|
||||
**Enjoy your Quarkus-powered Auction Monitor! 🚀**
|
||||
@@ -1,118 +0,0 @@
|
||||
# Refactoring Summary: Troostwijk Auction Monitor
|
||||
|
||||
## Overview
|
||||
This project has been refactored to focus on **image processing and monitoring**, removing all auction/lot scraping functionality which is now handled by the external `ARCHITECTURE-TROOSTWIJK-SCRAPER` process.
|
||||
|
||||
## Architecture Changes
|
||||
|
||||
### Removed Components
|
||||
- ❌ **TroostwijkScraper.java** - Removed (replaced by TroostwijkMonitor)
|
||||
- ❌ Auction discovery and scraping logic
|
||||
- ❌ Lot scraping via Playwright/JSoup
|
||||
- ❌ CacheDatabase (can be removed if not used elsewhere)
|
||||
|
||||
### New/Updated Components
|
||||
|
||||
#### New Classes
|
||||
- ✅ **TroostwijkMonitor.java** - Monitors bids and coordinates services (no scraping)
|
||||
- ✅ **ImageProcessingService.java** - Downloads images and runs object detection
|
||||
- ✅ **Console.java** - Simple output utility (renamed from IO to avoid Java 25 conflict)
|
||||
|
||||
#### Modernized Classes
|
||||
- ✅ **AuctionInfo** - Converted to immutable `record`
|
||||
- ✅ **Lot** - Converted to immutable `record` with `minutesUntilClose()` method
|
||||
- ✅ **DatabaseService.java** - Uses modern Java features:
|
||||
- Text blocks (`"""`) for SQL
|
||||
- Record accessor methods
|
||||
- Added `getImagesForLot()` method
|
||||
- Added `processed_at` timestamp to images table
|
||||
- Nested `ImageRecord` record
|
||||
|
||||
#### Preserved Components
|
||||
- ✅ **NotificationService.java** - Desktop/email notifications
|
||||
- ✅ **ObjectDetectionService.java** - YOLO-based object detection
|
||||
- ✅ **Main.java** - Updated to use new architecture
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Populated by External Scraper
|
||||
- `auctions` table - Auction metadata
|
||||
- `lots` table - Lot details with bidding info
|
||||
|
||||
### Populated by This Process
|
||||
- `images` table - Downloaded images with:
|
||||
- `file_path` - Local storage path
|
||||
- `labels` - Detected objects (comma-separated)
|
||||
- `processed_at` - Processing timestamp
|
||||
|
||||
## Modern Java Features Used
|
||||
|
||||
- **Records** - Immutable data carriers (AuctionInfo, Lot, ImageRecord)
|
||||
- **Text Blocks** - Multi-line SQL queries
|
||||
- **var** - Type inference throughout
|
||||
- **Switch expressions** - Where applicable
|
||||
- **Pattern matching** - Ready for future enhancements
|
||||
|
||||
## Responsibilities
|
||||
|
||||
### This Project
|
||||
1. ✅ Image downloading from URLs in database
|
||||
2. ✅ Object detection using YOLO/OpenCV
|
||||
3. ✅ Bid monitoring and change detection
|
||||
4. ✅ Desktop and email notifications
|
||||
5. ✅ Data enrichment with image analysis
|
||||
|
||||
### External ARCHITECTURE-TROOSTWIJK-SCRAPER
|
||||
1. 🔄 Discover auctions from Troostwijk website
|
||||
2. 🔄 Scrape lot details via API
|
||||
3. 🔄 Populate `auctions` and `lots` tables
|
||||
4. 🔄 Share database with this process
|
||||
|
||||
## Usage
|
||||
|
||||
### Running the Monitor
|
||||
```bash
|
||||
# With environment variables
|
||||
export DATABASE_FILE=troostwijk.db
|
||||
export NOTIFICATION_CONFIG=desktop # or smtp:user:pass:email
|
||||
|
||||
java -jar troostwijk-monitor.jar
|
||||
```
|
||||
|
||||
### Expected Output
|
||||
```
|
||||
=== Troostwijk Auction Monitor ===
|
||||
|
||||
✓ OpenCV loaded
|
||||
Initializing monitor...
|
||||
|
||||
📊 Current Database State:
|
||||
Total lots in database: 42
|
||||
Total images processed: 0
|
||||
|
||||
[1/2] Processing images...
|
||||
Processing pending images...
|
||||
|
||||
[2/2] Starting bid monitoring...
|
||||
✓ Monitoring service started
|
||||
|
||||
✓ Monitor is running. Press Ctrl+C to stop.
|
||||
|
||||
NOTE: This process expects auction/lot data from the external scraper.
|
||||
Make sure ARCHITECTURE-TROOSTWIJK-SCRAPER is running and populating the database.
|
||||
```
|
||||
|
||||
## Migration Notes
|
||||
|
||||
1. The project now compiles successfully with Java 25
|
||||
2. All scraping logic removed - rely on external scraper
|
||||
3. Shared database architecture for inter-process communication
|
||||
4. Clean separation of concerns
|
||||
5. Modern, maintainable codebase with records and text blocks
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Remove `CacheDatabase.java` if not needed
|
||||
- Consider adding API endpoint for external scraper to trigger image processing
|
||||
- Add metrics/logging framework
|
||||
- Consider message queue (e.g., Redis, RabbitMQ) for better inter-process communication
|
||||
@@ -1,164 +0,0 @@
|
||||
# Troostwijk Auction Extractor - Run Instructions
|
||||
|
||||
## Fixed Warnings
|
||||
|
||||
All warnings have been resolved:
|
||||
- ✅ SLF4J logging configured (slf4j-simple)
|
||||
- ✅ Native access enabled for SQLite JDBC
|
||||
- ✅ Logging output controlled via simplelogger.properties
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Java 21** installed
|
||||
2. **Maven** installed
|
||||
3. **IntelliJ IDEA** (recommended) or command line
|
||||
|
||||
## Setup (First Time Only)
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
In IntelliJ Terminal or PowerShell:
|
||||
|
||||
```bash
|
||||
# Reload Maven dependencies
|
||||
mvn clean install
|
||||
|
||||
# Install Playwright browser binaries (first time only)
|
||||
mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install"
|
||||
```
|
||||
|
||||
## Running the Application
|
||||
|
||||
### Option A: Using IntelliJ IDEA (Easiest)
|
||||
|
||||
1. **Add VM Options for native access:**
|
||||
- Run → Edit Configurations
|
||||
- Select or create configuration for `TroostwijkAuctionExtractor`
|
||||
- In "VM options" field, add:
|
||||
```
|
||||
--enable-native-access=ALL-UNNAMED
|
||||
```
|
||||
|
||||
2. **Add Program Arguments (optional):**
|
||||
- In "Program arguments" field, add:
|
||||
```
|
||||
--max-visits 3
|
||||
```
|
||||
|
||||
3. **Run the application:**
|
||||
- Click the green Run button
|
||||
|
||||
### Option B: Using Maven (Command Line)
|
||||
|
||||
```bash
|
||||
# Run with 3 page limit
|
||||
mvn exec:java
|
||||
|
||||
# Run with custom arguments (override pom.xml defaults)
|
||||
mvn exec:java -Dexec.args="--max-visits 5"
|
||||
|
||||
# Run without cache
|
||||
mvn exec:java -Dexec.args="--no-cache --max-visits 2"
|
||||
|
||||
# Run with unlimited visits
|
||||
mvn exec:java -Dexec.args=""
|
||||
```
|
||||
|
||||
### Option C: Using Java Directly
|
||||
|
||||
```bash
|
||||
# Compile first
|
||||
mvn clean compile
|
||||
|
||||
# Run with native access enabled
|
||||
java --enable-native-access=ALL-UNNAMED \
|
||||
-cp target/classes:$(mvn dependency:build-classpath -Dmdep.outputFile=/dev/stdout -q) \
|
||||
com.auction.TroostwijkAuctionExtractor --max-visits 3
|
||||
```
|
||||
|
||||
## Command Line Arguments
|
||||
|
||||
```
|
||||
--max-visits <n> Limit actual page fetches to n (0 = unlimited, default)
|
||||
--no-cache Disable page caching
|
||||
--help Show help message
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Test with 3 page visits (cached pages don't count):
|
||||
```bash
|
||||
mvn exec:java -Dexec.args="--max-visits 3"
|
||||
```
|
||||
|
||||
### Fresh extraction without cache:
|
||||
```bash
|
||||
mvn exec:java -Dexec.args="--no-cache --max-visits 5"
|
||||
```
|
||||
|
||||
### Full extraction (all pages, unlimited):
|
||||
```bash
|
||||
mvn exec:java -Dexec.args=""
|
||||
```
|
||||
|
||||
## Expected Output (No Warnings)
|
||||
|
||||
```
|
||||
=== Troostwijk Auction Extractor ===
|
||||
Max page visits set to: 3
|
||||
|
||||
Initializing Playwright browser...
|
||||
✓ Browser ready
|
||||
✓ Cache database initialized
|
||||
|
||||
Starting auction extraction from https://www.troostwijkauctions.com/auctions
|
||||
|
||||
[Page 1] Fetching auctions...
|
||||
✓ Fetched from website (visit 1/3)
|
||||
✓ Found 20 auctions
|
||||
|
||||
[Page 2] Fetching auctions...
|
||||
✓ Loaded from cache
|
||||
✓ Found 20 auctions
|
||||
|
||||
[Page 3] Fetching auctions...
|
||||
✓ Fetched from website (visit 2/3)
|
||||
✓ Found 20 auctions
|
||||
|
||||
✓ Total auctions extracted: 60
|
||||
|
||||
=== Results ===
|
||||
Total auctions found: 60
|
||||
Dutch auctions (NL): 45
|
||||
Actual page visits: 2
|
||||
|
||||
✓ Browser and cache closed
|
||||
```
|
||||
|
||||
## Cache Management
|
||||
|
||||
- Cache is stored in: `cache/page_cache.db`
|
||||
- Cache expires after: 24 hours (configurable in code)
|
||||
- To clear cache: Delete `cache/page_cache.db` file
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### If you still see warnings:
|
||||
|
||||
1. **Reload Maven project in IntelliJ:**
|
||||
- Right-click `pom.xml` → Maven → Reload project
|
||||
|
||||
2. **Verify VM options:**
|
||||
- Ensure `--enable-native-access=ALL-UNNAMED` is in VM options
|
||||
|
||||
3. **Clean and rebuild:**
|
||||
```bash
|
||||
mvn clean install
|
||||
```
|
||||
|
||||
### If Playwright fails:
|
||||
|
||||
```bash
|
||||
# Reinstall browser binaries
|
||||
mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"
|
||||
```
|
||||
Reference in New Issue
Block a user