Fix mock tests

2025-12-07 06:51:18 +01:00
parent 3efa83bc44
commit a649b629e4
4 changed files with 501 additions and 2 deletions
--- a/README.md
+++ b/README.md
@@ -24,9 +24,22 @@ All dependencies are managed via Maven (see `pom.xml`):
 - **JavaMail 1.6.2** - Email notifications (free)
 - **OpenCV 4.9.0** - Image processing and object detection

+## Quick Start
+
+### Development: Sync Production Data
+
+To work with real production data locally:
+
+```powershell
+# Linux/Mac (Bash)
+./scripts/sync-production-data.sh --db-only
+```
+
+See [scripts/README.md](scripts/README.md) for full documentation.
+
 ## Setup

-### 1.. Notification Options (Choose One)
+### 1. Notification Options (Choose One)

 #### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup)

@@ -541,7 +554,7 @@ Kavel 12345 sluit binnen 5 min.


 ```shell
-git add . | git commit -a -m all | git push
+ssh tour@athena.lan "docker run --rm -v shared-auction-data:/data -v /tmp:/tmp alpine cp /data/cache.db /tmp/cache.db" && scp tour@athena.lan:/tmp/cache.db c:/mnt/okcomputer/cache.db
 ```
 ## License

--- a/docs/DATA_SYNC_SETUP.md
+++ b/docs/DATA_SYNC_SETUP.md
@@ -0,0 +1,109 @@
+# Production Data Sync Setup
+
+Quick reference for syncing production data from `athena.lan` to your local development environment.
+
+## 🚀 One-Command Setup
+
+### Linux/Mac
+```bash
+./scripts/sync-production-data.sh
+```
+
+## 📋 Complete Usage
+
+### Bash (Linux/Mac/Git Bash)
+```bash
+# Database only
+./scripts/sync-production-data.sh --db-only
+
+# Everything
+./scripts/sync-production-data.sh --all
+
+# Images only
+./scripts/sync-production-data.sh --images-only
+```
+
+## 🔧 What It Does
+
+1. **Connects to athena.lan** via SSH
+2. **Copies database** from Docker volume to /tmp
+3. **Downloads to local** machine (c:\mnt\okcomputer\cache.db)
+4. **Backs up** existing local database automatically
+5. **Shows statistics** (auction count, lot count, etc.)
+6. **Cleans up** temporary files on remote server
+
+### With Images
+- Also syncs the `/data/images/` directory
+- Uses rsync for incremental sync (if available)
+- Can be large (several GB)
+
+## 📊 What You Get
+
+### Database (`cache.db`)
+- **~8.9 GB** of production data
+- 16,000+ lots
+- 536,000+ images metadata
+- Full auction history
+- HTTP cache from scraper
+
+### Images (`images/`)
+- Downloaded lot images
+- Organized by lot ID
+- Variable size (can be large)
+
+## ⚡ Quick Workflow
+
+### Daily Development
+```powershell
+# Morning: Get fresh data
+.\scripts\Sync-ProductionData.sh -Force
+
+# Develop & test
+mvn quarkus:dev
+
+# View dashboard
+start http://localhost:8080
+```
+
+## 🔒 Safety Features
+
+- ✅ **Automatic backups** before overwriting
+- ✅ **Confirmation prompts** (unless `-Force`)
+- ✅ **Error handling** with clear messages
+- ✅ **Cleanup** of temporary files
+- ✅ **Non-destructive** - production data is never modified
+
+## 🐛 Troubleshooting
+
+### "Permission denied" or SSH errors
+```bash
+# Test SSH connection
+ssh tour@athena.lan "echo OK"
+
+# If fails, check your SSH key
+ssh-add -l
+```
+
+### Database already exists
+- Script automatically backs up existing database
+- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS`
+
+### Slow image transfer
+- Install rsync for 10x faster incremental sync
+- Or sync database only: `.\scripts\Sync-ProductionData.sh` (default)
+
+## 📚 Full Documentation
+
+See [scripts/README.md](../scripts/README.md) for:
+- Prerequisites
+- Performance tips
+- Automation setup
+- Detailed troubleshooting
+
+## 🎯 Common Use Cases
+
+**Quick Links**:
+- [Main README](../README.md)
+- [Scripts Documentation](../scripts/README.md)
+- [Integration Flowchart](INTEGRATION_FLOWCHART.md)
+- [Intelligence Features](INTELLIGENCE_FEATURES_SUMMARY.md)
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -0,0 +1,206 @@
+# Auctiora Scripts
+
+Utility scripts for managing the Auctiora auction monitoring system.
+
+## 📦 Available Scripts
+
+### 1. Production Data Sync
+
+Sync production database and images from `athena.lan` to your local development environment.
+
+#### Quick Start
+
+**Linux/Mac (Bash)**:
+```bash
+# Make executable (first time only)
+chmod +x scripts/sync-production-data.sh
+
+# Sync database only
+./scripts/sync-production-data.sh --db-only
+
+# Sync everything
+./scripts/sync-production-data.sh --all
+
+# Sync images only
+./scripts/sync-production-data.sh --images-only
+```
+
+## 🔧 Prerequisites
+
+### Required
+- **SSH Client**: OpenSSH or equivalent
+  - Windows: Built-in on Windows 10+, or install [Git Bash](https://git-scm.com/downloads)
+  - Linux/Mac: Pre-installed
+- **SCP**: Secure copy (usually comes with SSH)
+- **SSH Access**: SSH key configured for `tour@athena.lan`
+
+### Optional
+- **rsync**: For efficient incremental image sync
+  - Windows: Install via [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) or [Cygwin](https://www.cygwin.com/)
+  - Linux/Mac: Usually pre-installed
+- **sqlite3**: For showing database statistics
+  - Windows: Download from [sqlite.org](https://www.sqlite.org/download.html)
+  - Linux: `sudo apt install sqlite3`
+  - Mac: Pre-installed
+
+## 📊 What Gets Synced
+
+### Database (`cache.db`)
+- **Size**: ~8.9 GB (as of Dec 2024)
+- **Contains**:
+  - Auctions metadata
+  - Lots (kavels) with bid information
+  - Images metadata and URLs
+  - HTTP cache for scraper
+- **Local Path**: `c:\mnt\okcomputer\cache.db`
+
+### Images Directory
+- **Size**: Varies (can be large)
+- **Contains**:
+  - Downloaded lot images
+  - Organized by lot ID
+- **Local Path**: `c:\mnt\okcomputer\images\`
+
+## 🚀 Usage Examples
+
+## 📁 File Locations
+
+### Remote (Production)
+```
+athena.lan
+├── Docker Volume: shared-auction-data
+│   ├── /data/cache.db          (SQLite database)
+│   └── /data/images/           (Image files)
+└── /tmp/                        (Temporary staging area)
+```
+
+### Local (Development)
+```
+c:\mnt\okcomputer\
+├── cache.db                     (SQLite database)
+├── cache.db.backup-*            (Automatic backups)
+└── images\                      (Image files)
+```
+
+## 🔒 Safety Features
+
+### Automatic Backups
+- Existing local database is automatically backed up before sync
+- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS`
+- Keep recent backups manually or clean up old ones
+
+### Confirmation Prompts
+- PowerShell script prompts for confirmation (unless `-Force` is used)
+- Shows configuration before executing
+- Safe to cancel at any time
+
+### Error Handling
+- Validates SSH connection before starting
+- Cleans up temporary files on remote server
+- Reports clear error messages
+
+## ⚡ Performance Tips
+
+### Faster Image Sync with rsync
+Install rsync for incremental image sync (only new/changed files):
+
+**Windows (WSL)**:
+```powershell
+wsl --install
+wsl -d Ubuntu
+sudo apt install rsync
+```
+
+**Windows (Chocolatey)**:
+```powershell
+choco install rsync
+```
+
+**Benefit**: First sync downloads everything, subsequent syncs only transfer changed files.
+
+Images can be synced separately when needed for image processing tests.
+
+## 🐛 Troubleshooting
+
+### SSH Connection Issues
+```powershell
+# Test SSH connection
+ssh tour@athena.lan "echo 'Connection OK'"
+
+# Check SSH key
+ssh-add -l
+```
+
+### Permission Denied
+```bash
+# Add SSH key (Linux/Mac)
+chmod 600 ~/.ssh/id_rsa
+ssh-add ~/.ssh/id_rsa
+
+# Windows: Use PuTTY or OpenSSH for Windows
+```
+
+### Database Locked Error
+```powershell
+# Make sure no other process is using the database
+Get-Process | Where-Object {$_.Path -like "*java*"} | Stop-Process
+
+# Or restart the monitor
+```
+
+### Slow Image Sync
+- Use rsync instead of scp (see Performance Tips)
+- Consider syncing only database for code development
+- Images only needed for object detection testing
+
+## 📝 Script Details
+
+### sync-production-data.sh (Bash)
+- **Platform**: Linux, Mac, Git Bash on Windows
+- **Best for**: Unix-like environments
+- **Features**: Color output, progress bars, statistics
+
+## 🔄 Automation
+
+### Linux/Mac Cron
+```bash
+# Edit crontab
+crontab -e
+
+# Add daily sync at 7 AM
+0 7 * * * /path/to/auctiora/scripts/sync-production-data.sh --db-only
+```
+
+## 🆘 Support
+
+### Getting Help
+```bash
+# Bash
+./scripts/sync-production-data.sh --help
+```
+
+### Common Commands
+```powershell
+# Check database size
+ls c:\mnt\okcomputer\cache.db -h
+
+# View database contents
+sqlite3 c:\mnt\okcomputer\cache.db
+  .tables
+  .schema lots
+  SELECT COUNT(*) FROM lots;
+  .quit
+
+# Check image count
+(Get-ChildItem c:\mnt\okcomputer\images -Recurse -File).Count
+```
+
+## 📚 Related Documentation
+- [Database Architecture](../wiki/DATABASE_ARCHITECTURE.md)
+- [Integration Flowchart](../docs/INTEGRATION_FLOWCHART.md)
+- [Main README](../README.md)
+
+---
+
+**Last Updated**: December 2025
+**Maintainer**: Auctiora Development Team
--- a/scripts/sync-production-data.sh
+++ b/scripts/sync-production-data.sh
@@ -0,0 +1,171 @@
+#!/bin/bash
+#
+# Sync Production Data to Local
+#
+# This script copies the production SQLite database and images from the remote
+# server (athena.lan) to your local development environment.
+#
+# Usage:
+#   ./scripts/sync-production-data.sh [--db-only|--images-only|--all]
+#
+# Options:
+#   --db-only      Only sync the database (default)
+#   --images-only  Only sync the images
+#   --all          Sync both database and images
+#   --help         Show this help message
+#
+
+set -e  # Exit on error
+
+# Configuration
+REMOTE_HOST="tour@athena.lan"
+REMOTE_VOLUME="shared-auction-data"
+LOCAL_DB_PATH="c:/mnt/okcomputer/cache.db"
+LOCAL_IMAGES_PATH="c:/mnt/okcomputer/images"
+REMOTE_TMP="/tmp"
+
+# Colors for output
+GREEN='\033[0;32m'
+BLUE='\033[0;34m'
+YELLOW='\033[1;33m'
+RED='\033[0;31m'
+NC='\033[0m' # No Color
+
+# Parse arguments
+SYNC_MODE="db"  # Default: database only
+
+case "${1:-}" in
+    --db-only)
+        SYNC_MODE="db"
+        ;;
+    --images-only)
+        SYNC_MODE="images"
+        ;;
+    --all)
+        SYNC_MODE="all"
+        ;;
+    --help|-h)
+        grep '^#' "$0" | sed 's/^# \?//'
+        exit 0
+        ;;
+    "")
+        SYNC_MODE="db"
+        ;;
+    *)
+        echo -e "${RED}Error: Unknown option '$1'${NC}"
+        echo "Use --help for usage information"
+        exit 1
+        ;;
+esac
+
+echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}"
+echo -e "${BLUE}║     Production Data Sync - Auctiora Monitor           ║${NC}"
+echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}"
+echo ""
+
+# Function to sync database
+sync_database() {
+    echo -e "${YELLOW}[1/3] Copying database from Docker volume to /tmp...${NC}"
+    ssh ${REMOTE_HOST} "docker run --rm -v ${REMOTE_VOLUME}:/data -v ${REMOTE_TMP}:${REMOTE_TMP} alpine cp /data/cache.db ${REMOTE_TMP}/cache.db"
+
+    echo -e "${YELLOW}[2/3] Downloading database from remote server...${NC}"
+    # Create backup of existing local database
+    if [ -f "${LOCAL_DB_PATH}" ]; then
+        BACKUP_PATH="${LOCAL_DB_PATH}.backup-$(date +%Y%m%d-%H%M%S)"
+        echo -e "${BLUE}   Backing up existing database to: ${BACKUP_PATH}${NC}"
+        cp "${LOCAL_DB_PATH}" "${BACKUP_PATH}"
+    fi
+
+    # Download new database
+    scp ${REMOTE_HOST}:${REMOTE_TMP}/cache.db "${LOCAL_DB_PATH}"
+
+    echo -e "${YELLOW}[3/3] Cleaning up remote /tmp...${NC}"
+    ssh ${REMOTE_HOST} "rm -f ${REMOTE_TMP}/cache.db"
+
+    # Show database info
+    DB_SIZE=$(du -h "${LOCAL_DB_PATH}" | cut -f1)
+    echo -e "${GREEN}✓ Database synced successfully (${DB_SIZE})${NC}"
+
+    # Show table counts
+    echo -e "${BLUE}   Database statistics:${NC}"
+    sqlite3 "${LOCAL_DB_PATH}" <<EOF
+.mode box
+SELECT
+    'auctions' as table_name, COUNT(*) as count FROM auctions
+UNION ALL
+SELECT 'lots', COUNT(*) FROM lots
+UNION ALL
+SELECT 'images', COUNT(*) FROM images
+UNION ALL
+SELECT 'cache', COUNT(*) FROM cache;
+EOF
+}
+
+# Function to sync images
+sync_images() {
+    echo -e "${YELLOW}[1/4] Getting image directory structure from Docker volume...${NC}"
+
+    # Create local images directory if it doesn't exist
+    mkdir -p "${LOCAL_IMAGES_PATH}"
+
+    echo -e "${YELLOW}[2/4] Copying images from Docker volume to /tmp...${NC}"
+    # Copy entire images directory from volume to /tmp
+    ssh ${REMOTE_HOST} "docker run --rm -v ${REMOTE_VOLUME}:/data -v ${REMOTE_TMP}:${REMOTE_TMP} alpine sh -c 'mkdir -p ${REMOTE_TMP}/auction-images && cp -r /data/images/* ${REMOTE_TMP}/auction-images/ 2>/dev/null || true'"
+
+    echo -e "${YELLOW}[3/4] Syncing images to local directory (this may take a while)...${NC}"
+    # Use rsync for efficient incremental sync
+    if command -v rsync &> /dev/null; then
+        echo -e "${BLUE}   Using rsync for efficient transfer...${NC}"
+        rsync -avz --progress ${REMOTE_HOST}:${REMOTE_TMP}/auction-images/ "${LOCAL_IMAGES_PATH}/"
+    else
+        echo -e "${BLUE}   Using scp for transfer (install rsync for faster incremental sync)...${NC}"
+        scp -r ${REMOTE_HOST}:${REMOTE_TMP}/auction-images/* "${LOCAL_IMAGES_PATH}/"
+    fi
+
+    echo -e "${YELLOW}[4/4] Cleaning up remote /tmp...${NC}"
+    ssh ${REMOTE_HOST} "rm -rf ${REMOTE_TMP}/auction-images"
+
+    # Show image stats
+    IMAGE_COUNT=$(find "${LOCAL_IMAGES_PATH}" -type f 2>/dev/null | wc -l)
+    IMAGE_SIZE=$(du -sh "${LOCAL_IMAGES_PATH}" 2>/dev/null | cut -f1)
+    echo -e "${GREEN}✓ Images synced successfully${NC}"
+    echo -e "${BLUE}   Total images: ${IMAGE_COUNT}${NC}"
+    echo -e "${BLUE}   Total size: ${IMAGE_SIZE}${NC}"
+}
+
+# Execute sync based on mode
+START_TIME=$(date +%s)
+
+case "$SYNC_MODE" in
+    db)
+        echo -e "${BLUE}Mode: Database only${NC}"
+        echo ""
+        sync_database
+        ;;
+    images)
+        echo -e "${BLUE}Mode: Images only${NC}"
+        echo ""
+        sync_images
+        ;;
+    all)
+        echo -e "${BLUE}Mode: Database + Images${NC}"
+        echo ""
+        sync_database
+        echo ""
+        sync_images
+        ;;
+esac
+
+END_TIME=$(date +%s)
+DURATION=$((END_TIME - START_TIME))
+
+echo ""
+echo -e "${GREEN}╔════════════════════════════════════════════════════════╗${NC}"
+echo -e "${GREEN}║  Sync completed successfully in ${DURATION} seconds               ║${NC}"
+echo -e "${GREEN}╚════════════════════════════════════════════════════════╝${NC}"
+echo ""
+echo -e "${BLUE}Next steps:${NC}"
+echo -e "  1. Verify data: sqlite3 ${LOCAL_DB_PATH} 'SELECT COUNT(*) FROM lots;'"
+echo -e "  2. Start monitor: mvn quarkus:dev"
+echo -e "  3. Open dashboard: http://localhost:8080"
+echo ""