Fix mock tests
This commit is contained in:
Tour
537
wiki/WORKFLOW_GUIDE.md
Normal file
537
wiki/WORKFLOW_GUIDE.md
Normal file
@@ -0,0 +1,537 @@
|
||||
## Troostwijk Auction Monitor - Workflow Integration Guide
|
||||
|
||||
Complete guide for running the auction monitoring system with scheduled workflows, cron jobs, and event-driven triggers.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Running Modes](#running-modes)
|
||||
3. [Workflow Orchestration](#workflow-orchestration)
|
||||
4. [Windows Scheduling](#windows-scheduling)
|
||||
5. [Event-Driven Triggers](#event-driven-triggers)
|
||||
6. [Configuration](#configuration)
|
||||
7. [Monitoring & Debugging](#monitoring--debugging)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Troostwijk Auction Monitor supports multiple execution modes:
|
||||
|
||||
- **Workflow Mode** (Recommended): Continuous operation with built-in scheduling
|
||||
- **Once Mode**: Single execution for external schedulers (Windows Task Scheduler, cron)
|
||||
- **Legacy Mode**: Original monitoring approach
|
||||
- **Status Mode**: Quick status check
|
||||
|
||||
---
|
||||
|
||||
## Running Modes
|
||||
|
||||
### 1. Workflow Mode (Default - Recommended)
|
||||
|
||||
**Runs all workflows continuously with built-in scheduling.**
|
||||
|
||||
```bash
|
||||
# Windows
|
||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar workflow
|
||||
|
||||
# Or simply (workflow is default)
|
||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar
|
||||
|
||||
# Using batch script
|
||||
run-workflow.bat
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- ✅ Imports scraper data every 30 minutes
|
||||
- ✅ Processes images every 1 hour
|
||||
- ✅ Monitors bids every 15 minutes
|
||||
- ✅ Checks closing times every 5 minutes
|
||||
|
||||
**Best for:**
|
||||
- Production deployment
|
||||
- Long-running services
|
||||
- Development/testing
|
||||
|
||||
---
|
||||
|
||||
### 2. Once Mode (For External Schedulers)
|
||||
|
||||
**Runs complete workflow once and exits.**
|
||||
|
||||
```bash
|
||||
# Windows
|
||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar once
|
||||
|
||||
# Using batch script
|
||||
run-once.bat
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Imports scraper data
|
||||
2. Processes pending images
|
||||
3. Monitors bids
|
||||
4. Checks closing times
|
||||
5. Exits
|
||||
|
||||
**Best for:**
|
||||
- Windows Task Scheduler
|
||||
- Cron jobs (Linux/Mac)
|
||||
- Manual execution
|
||||
- Testing
|
||||
|
||||
---
|
||||
|
||||
### 3. Legacy Mode
|
||||
|
||||
**Original monitoring approach (backward compatibility).**
|
||||
|
||||
```bash
|
||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar legacy
|
||||
```
|
||||
|
||||
**Best for:**
|
||||
- Maintaining existing deployments
|
||||
- Troubleshooting
|
||||
|
||||
---
|
||||
|
||||
### 4. Status Mode
|
||||
|
||||
**Shows current status and exits.**
|
||||
|
||||
```bash
|
||||
java -jar target\troostwijk-scraper-1.0-SNAPSHOT-jar-with-dependencies.jar status
|
||||
|
||||
# Using batch script
|
||||
check-status.bat
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
📊 Workflow Status:
|
||||
Running: No
|
||||
Auctions: 25
|
||||
Lots: 150
|
||||
Images: 300
|
||||
Closing soon (< 30 min): 5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Orchestration
|
||||
|
||||
The `WorkflowOrchestrator` coordinates 4 scheduled workflows:
|
||||
|
||||
### Workflow 1: Scraper Data Import
|
||||
**Frequency:** Every 30 minutes
|
||||
**Purpose:** Import new auctions and lots from external scraper
|
||||
|
||||
**Process:**
|
||||
1. Import auctions from scraper database
|
||||
2. Import lots from scraper database
|
||||
3. Import image URLs
|
||||
4. Send notification if significant data imported
|
||||
|
||||
**Code Location:** `WorkflowOrchestrator.java:110`
|
||||
|
||||
---
|
||||
|
||||
### Workflow 2: Image Processing
|
||||
**Frequency:** Every 1 hour
|
||||
**Purpose:** Download images and run object detection
|
||||
|
||||
**Process:**
|
||||
1. Get unprocessed images from database
|
||||
2. Download each image
|
||||
3. Run YOLO object detection
|
||||
4. Save labels to database
|
||||
5. Send notification for interesting detections (3+ objects)
|
||||
|
||||
**Code Location:** `WorkflowOrchestrator.java:150`
|
||||
|
||||
---
|
||||
|
||||
### Workflow 3: Bid Monitoring
|
||||
**Frequency:** Every 15 minutes
|
||||
**Purpose:** Check for bid changes and send notifications
|
||||
|
||||
**Process:**
|
||||
1. Get all active lots
|
||||
2. Check for bid changes (via external scraper updates)
|
||||
3. Send notifications for bid increases
|
||||
|
||||
**Code Location:** `WorkflowOrchestrator.java:210`
|
||||
|
||||
**Note:** The external scraper updates bids; this workflow monitors and notifies.
|
||||
|
||||
---
|
||||
|
||||
### Workflow 4: Closing Alerts
|
||||
**Frequency:** Every 5 minutes
|
||||
**Purpose:** Send alerts for lots closing soon
|
||||
|
||||
**Process:**
|
||||
1. Get all active lots
|
||||
2. Check closing times
|
||||
3. Send high-priority notification for lots closing in < 5 min
|
||||
4. Mark as notified to prevent duplicates
|
||||
|
||||
**Code Location:** `WorkflowOrchestrator.java:240`
|
||||
|
||||
---
|
||||
|
||||
## Windows Scheduling
|
||||
|
||||
### Option A: Use Built-in Workflow Mode (Recommended)
|
||||
|
||||
**Run as a Windows Service or startup application:**
|
||||
|
||||
1. Create shortcut to `run-workflow.bat`
|
||||
2. Place in: `C:\Users\[YourUser]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup`
|
||||
3. Monitor will start automatically on login
|
||||
|
||||
---
|
||||
|
||||
### Option B: Windows Task Scheduler (Once Mode)
|
||||
|
||||
**Automated setup:**
|
||||
|
||||
```powershell
|
||||
# Run PowerShell as Administrator
|
||||
.\setup-windows-task.ps1
|
||||
```
|
||||
|
||||
This creates two tasks:
|
||||
- `TroostwijkMonitor-Workflow`: Runs every 30 minutes
|
||||
- `TroostwijkMonitor-StatusCheck`: Runs every 6 hours
|
||||
|
||||
**Manual setup:**
|
||||
|
||||
1. Open Task Scheduler
|
||||
2. Create Basic Task
|
||||
3. Configure:
|
||||
- **Name:** `TroostwijkMonitor`
|
||||
- **Trigger:** Every 30 minutes
|
||||
- **Action:** Start a program
|
||||
- **Program:** `java`
|
||||
- **Arguments:** `-jar "C:\path\to\troostwijk-scraper.jar" once`
|
||||
- **Start in:** `C:\path\to\project`
|
||||
|
||||
---
|
||||
|
||||
### Option C: Multiple Scheduled Tasks (Fine-grained Control)
|
||||
|
||||
Create separate tasks for each workflow:
|
||||
|
||||
| Task | Frequency | Command |
|
||||
|------|-----------|---------|
|
||||
| Import Data | Every 30 min | `run-once.bat` |
|
||||
| Process Images | Every 1 hour | `run-once.bat` |
|
||||
| Check Bids | Every 15 min | `run-once.bat` |
|
||||
| Closing Alerts | Every 5 min | `run-once.bat` |
|
||||
|
||||
---
|
||||
|
||||
## Event-Driven Triggers
|
||||
|
||||
The orchestrator supports event-driven execution:
|
||||
|
||||
### 1. New Auction Discovered
|
||||
|
||||
```java
|
||||
orchestrator.onNewAuctionDiscovered(auctionInfo);
|
||||
```
|
||||
|
||||
**Triggered when:**
|
||||
- External scraper finds new auction
|
||||
|
||||
**Actions:**
|
||||
- Insert to database
|
||||
- Send notification
|
||||
|
||||
---
|
||||
|
||||
### 2. Bid Change Detected
|
||||
|
||||
```java
|
||||
orchestrator.onBidChange(lot, previousBid, newBid);
|
||||
```
|
||||
|
||||
**Triggered when:**
|
||||
- Bid increases on monitored lot
|
||||
|
||||
**Actions:**
|
||||
- Update database
|
||||
- Send notification: "Nieuw bod op kavel X: €Y (was €Z)"
|
||||
|
||||
---
|
||||
|
||||
### 3. Objects Detected
|
||||
|
||||
```java
|
||||
orchestrator.onObjectsDetected(lotId, labels);
|
||||
```
|
||||
|
||||
**Triggered when:**
|
||||
- YOLO detects 2+ objects in image
|
||||
|
||||
**Actions:**
|
||||
- Send notification: "Lot X contains: car, truck, machinery"
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Database location
|
||||
set DATABASE_FILE=C:\mnt\okcomputer\output\cache.db
|
||||
|
||||
# Notification configuration
|
||||
set NOTIFICATION_CONFIG=desktop
|
||||
|
||||
# Or for email notifications
|
||||
set NOTIFICATION_CONFIG=smtp:your@gmail.com:app_password:recipient@example.com
|
||||
```
|
||||
|
||||
### Configuration Files
|
||||
|
||||
**YOLO Model Paths** (`Main.java:35-37`):
|
||||
```java
|
||||
String yoloCfg = "models/yolov4.cfg";
|
||||
String yoloWeights = "models/yolov4.weights";
|
||||
String yoloClasses = "models/coco.names";
|
||||
```
|
||||
|
||||
### Customizing Schedules
|
||||
|
||||
Edit `WorkflowOrchestrator.java` to change frequencies:
|
||||
|
||||
```java
|
||||
// Change from 30 minutes to 15 minutes
|
||||
scheduler.scheduleAtFixedRate(() -> {
|
||||
// ... scraper import logic
|
||||
}, 0, 15, TimeUnit.MINUTES); // Changed from 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Debugging
|
||||
|
||||
### Check Status
|
||||
|
||||
```bash
|
||||
# Quick status check
|
||||
java -jar troostwijk-monitor.jar status
|
||||
|
||||
# Or
|
||||
check-status.bat
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
Workflows print timestamped logs:
|
||||
|
||||
```
|
||||
📥 [WORKFLOW 1] Importing scraper data...
|
||||
→ Imported 5 auctions
|
||||
→ Imported 25 lots
|
||||
→ Found 50 unprocessed images
|
||||
✓ Scraper import completed in 1250ms
|
||||
|
||||
🖼️ [WORKFLOW 2] Processing pending images...
|
||||
→ Processing 50 images
|
||||
✓ Processed 50 images, detected objects in 12 (15.3s)
|
||||
```
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. No data being imported
|
||||
|
||||
**Problem:** External scraper not running
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check if scraper is running and populating database
|
||||
sqlite3 C:\mnt\okcomputer\output\cache.db "SELECT COUNT(*) FROM auctions;"
|
||||
```
|
||||
|
||||
#### 2. Images not downloading
|
||||
|
||||
**Problem:** No internet connection or invalid URLs
|
||||
|
||||
**Solution:**
|
||||
- Check network connectivity
|
||||
- Verify image URLs in database
|
||||
- Check firewall settings
|
||||
|
||||
#### 3. Notifications not showing
|
||||
|
||||
**Problem:** System tray not available
|
||||
|
||||
**Solution:**
|
||||
- Use email notifications instead
|
||||
- Check notification permissions in Windows
|
||||
|
||||
#### 4. Workflows not running
|
||||
|
||||
**Problem:** Application crashed or was stopped
|
||||
|
||||
**Solution:**
|
||||
- Check Task Scheduler logs
|
||||
- Review application logs
|
||||
- Restart in workflow mode
|
||||
|
||||
---
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Example 1: Complete Automated Workflow
|
||||
|
||||
**Setup:**
|
||||
1. External scraper runs continuously, populating database
|
||||
2. This monitor runs in workflow mode
|
||||
3. Notifications sent to desktop + email
|
||||
|
||||
**Result:**
|
||||
- New auctions → Notification within 30 min
|
||||
- New images → Processed within 1 hour
|
||||
- Bid changes → Notification within 15 min
|
||||
- Closing alerts → Notification within 5 min
|
||||
|
||||
---
|
||||
|
||||
### Example 2: On-Demand Processing
|
||||
|
||||
**Setup:**
|
||||
1. External scraper runs once per day (cron/Task Scheduler)
|
||||
2. This monitor runs in once mode after scraper completes
|
||||
|
||||
**Script:**
|
||||
```bash
|
||||
# run-daily.bat
|
||||
@echo off
|
||||
REM Run scraper first
|
||||
python scraper.py
|
||||
|
||||
REM Wait for completion
|
||||
timeout /t 30
|
||||
|
||||
REM Run monitor once
|
||||
java -jar troostwijk-monitor.jar once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 3: Event-Driven with External Integration
|
||||
|
||||
**Setup:**
|
||||
1. External system calls orchestrator events
|
||||
2. Workflows run on-demand
|
||||
|
||||
**Java code:**
|
||||
```java
|
||||
WorkflowOrchestrator orchestrator = new WorkflowOrchestrator(...);
|
||||
|
||||
// When external scraper finds new auction
|
||||
AuctionInfo newAuction = parseScraperData();
|
||||
orchestrator.onNewAuctionDiscovered(newAuction);
|
||||
|
||||
// When bid detected
|
||||
orchestrator.onBidChange(lot, 100.0, 150.0);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
### Custom Workflows
|
||||
|
||||
Add custom workflows to `WorkflowOrchestrator`:
|
||||
|
||||
```java
|
||||
// Workflow 5: Value Estimation (every 2 hours)
|
||||
scheduler.scheduleAtFixedRate(() -> {
|
||||
try {
|
||||
Console.println("💰 [WORKFLOW 5] Estimating values...");
|
||||
|
||||
var lotsWithImages = db.getLotsWithImages();
|
||||
for (var lot : lotsWithImages) {
|
||||
var images = db.getImagesForLot(lot.lotId());
|
||||
double estimatedValue = estimateValue(images);
|
||||
|
||||
// Update database
|
||||
db.updateLotEstimatedValue(lot.lotId(), estimatedValue);
|
||||
|
||||
// Notify if high value
|
||||
if (estimatedValue > 5000) {
|
||||
notifier.sendNotification(
|
||||
String.format("High value lot detected: %d (€%.2f)",
|
||||
lot.lotId(), estimatedValue),
|
||||
"Value Alert", 1
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
Console.println(" ❌ Value estimation failed: " + e.getMessage());
|
||||
}
|
||||
}, 10, 120, TimeUnit.MINUTES);
|
||||
```
|
||||
|
||||
### Webhook Integration
|
||||
|
||||
Trigger workflows via HTTP webhooks:
|
||||
|
||||
```java
|
||||
// In a separate web server (e.g., using Javalin)
|
||||
Javalin app = Javalin.create().start(7070);
|
||||
|
||||
app.post("/webhook/new-auction", ctx -> {
|
||||
AuctionInfo auction = ctx.bodyAsClass(AuctionInfo.class);
|
||||
orchestrator.onNewAuctionDiscovered(auction);
|
||||
ctx.result("OK");
|
||||
});
|
||||
|
||||
app.post("/webhook/bid-change", ctx -> {
|
||||
BidChange change = ctx.bodyAsClass(BidChange.class);
|
||||
orchestrator.onBidChange(change.lot, change.oldBid, change.newBid);
|
||||
ctx.result("OK");
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Mode | Use Case | Scheduling | Best For |
|
||||
|------|----------|------------|----------|
|
||||
| **workflow** | Continuous operation | Built-in (Java) | Production, development |
|
||||
| **once** | Single execution | External (Task Scheduler) | Cron jobs, on-demand |
|
||||
| **legacy** | Backward compatibility | Built-in (Java) | Existing deployments |
|
||||
| **status** | Quick check | Manual/External | Health checks, debugging |
|
||||
|
||||
**Recommended Setup for Windows:**
|
||||
1. Install as Windows Service OR
|
||||
2. Add to Startup folder (workflow mode) OR
|
||||
3. Use Task Scheduler (once mode, every 30 min)
|
||||
|
||||
**All workflows automatically:**
|
||||
- Import data from scraper
|
||||
- Process images
|
||||
- Detect objects
|
||||
- Monitor bids
|
||||
- Send notifications
|
||||
- Handle errors gracefully
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check `TEST_SUITE_SUMMARY.md` for test coverage
|
||||
- Review code in `WorkflowOrchestrator.java`
|
||||
- Run `java -jar troostwijk-monitor.jar status` for diagnostics
|
||||
Reference in New Issue
Block a user