diff --git a/README.md b/README.md index 7e675c8..bb790df 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,22 @@ All dependencies are managed via Maven (see `pom.xml`): - **JavaMail 1.6.2** - Email notifications (free) - **OpenCV 4.9.0** - Image processing and object detection +## Quick Start + +### Development: Sync Production Data + +To work with real production data locally: + +```powershell +# Linux/Mac (Bash) +./scripts/sync-production-data.sh --db-only +``` + +See [scripts/README.md](scripts/README.md) for full documentation. + ## Setup -### 1.. Notification Options (Choose One) +### 1. Notification Options (Choose One) #### Option A: Desktop Notifications Only ⭐ (Recommended - Zero Setup) @@ -541,7 +554,7 @@ Kavel 12345 sluit binnen 5 min. ```shell -git add . | git commit -a -m all | git push +ssh tour@athena.lan "docker run --rm -v shared-auction-data:/data -v /tmp:/tmp alpine cp /data/cache.db /tmp/cache.db" && scp tour@athena.lan:/tmp/cache.db c:/mnt/okcomputer/cache.db ``` ## License diff --git a/docs/DATA_SYNC_SETUP.md b/docs/DATA_SYNC_SETUP.md new file mode 100644 index 0000000..34e329e --- /dev/null +++ b/docs/DATA_SYNC_SETUP.md @@ -0,0 +1,109 @@ +# Production Data Sync Setup + +Quick reference for syncing production data from `athena.lan` to your local development environment. + +## 🚀 One-Command Setup + +### Linux/Mac +```bash +./scripts/sync-production-data.sh +``` + +## 📋 Complete Usage + +### Bash (Linux/Mac/Git Bash) +```bash +# Database only +./scripts/sync-production-data.sh --db-only + +# Everything +./scripts/sync-production-data.sh --all + +# Images only +./scripts/sync-production-data.sh --images-only +``` + +## 🔧 What It Does + +1. **Connects to athena.lan** via SSH +2. **Copies database** from Docker volume to /tmp +3. **Downloads to local** machine (c:\mnt\okcomputer\cache.db) +4. **Backs up** existing local database automatically +5. **Shows statistics** (auction count, lot count, etc.) +6. **Cleans up** temporary files on remote server + +### With Images +- Also syncs the `/data/images/` directory +- Uses rsync for incremental sync (if available) +- Can be large (several GB) + +## 📊 What You Get + +### Database (`cache.db`) +- **~8.9 GB** of production data +- 16,000+ lots +- 536,000+ images metadata +- Full auction history +- HTTP cache from scraper + +### Images (`images/`) +- Downloaded lot images +- Organized by lot ID +- Variable size (can be large) + +## ⚡ Quick Workflow + +### Daily Development +```powershell +# Morning: Get fresh data +.\scripts\Sync-ProductionData.sh -Force + +# Develop & test +mvn quarkus:dev + +# View dashboard +start http://localhost:8080 +``` + +## 🔒 Safety Features + +- ✅ **Automatic backups** before overwriting +- ✅ **Confirmation prompts** (unless `-Force`) +- ✅ **Error handling** with clear messages +- ✅ **Cleanup** of temporary files +- ✅ **Non-destructive** - production data is never modified + +## 🐛 Troubleshooting + +### "Permission denied" or SSH errors +```bash +# Test SSH connection +ssh tour@athena.lan "echo OK" + +# If fails, check your SSH key +ssh-add -l +``` + +### Database already exists +- Script automatically backs up existing database +- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS` + +### Slow image transfer +- Install rsync for 10x faster incremental sync +- Or sync database only: `.\scripts\Sync-ProductionData.sh` (default) + +## 📚 Full Documentation + +See [scripts/README.md](../scripts/README.md) for: +- Prerequisites +- Performance tips +- Automation setup +- Detailed troubleshooting + +## 🎯 Common Use Cases + +**Quick Links**: +- [Main README](../README.md) +- [Scripts Documentation](../scripts/README.md) +- [Integration Flowchart](INTEGRATION_FLOWCHART.md) +- [Intelligence Features](INTELLIGENCE_FEATURES_SUMMARY.md) diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..3dd20c5 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,206 @@ +# Auctiora Scripts + +Utility scripts for managing the Auctiora auction monitoring system. + +## 📦 Available Scripts + +### 1. Production Data Sync + +Sync production database and images from `athena.lan` to your local development environment. + +#### Quick Start + +**Linux/Mac (Bash)**: +```bash +# Make executable (first time only) +chmod +x scripts/sync-production-data.sh + +# Sync database only +./scripts/sync-production-data.sh --db-only + +# Sync everything +./scripts/sync-production-data.sh --all + +# Sync images only +./scripts/sync-production-data.sh --images-only +``` + +## 🔧 Prerequisites + +### Required +- **SSH Client**: OpenSSH or equivalent + - Windows: Built-in on Windows 10+, or install [Git Bash](https://git-scm.com/downloads) + - Linux/Mac: Pre-installed +- **SCP**: Secure copy (usually comes with SSH) +- **SSH Access**: SSH key configured for `tour@athena.lan` + +### Optional +- **rsync**: For efficient incremental image sync + - Windows: Install via [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) or [Cygwin](https://www.cygwin.com/) + - Linux/Mac: Usually pre-installed +- **sqlite3**: For showing database statistics + - Windows: Download from [sqlite.org](https://www.sqlite.org/download.html) + - Linux: `sudo apt install sqlite3` + - Mac: Pre-installed + +## 📊 What Gets Synced + +### Database (`cache.db`) +- **Size**: ~8.9 GB (as of Dec 2024) +- **Contains**: + - Auctions metadata + - Lots (kavels) with bid information + - Images metadata and URLs + - HTTP cache for scraper +- **Local Path**: `c:\mnt\okcomputer\cache.db` + +### Images Directory +- **Size**: Varies (can be large) +- **Contains**: + - Downloaded lot images + - Organized by lot ID +- **Local Path**: `c:\mnt\okcomputer\images\` + +## 🚀 Usage Examples + +## 📁 File Locations + +### Remote (Production) +``` +athena.lan +├── Docker Volume: shared-auction-data +│ ├── /data/cache.db (SQLite database) +│ └── /data/images/ (Image files) +└── /tmp/ (Temporary staging area) +``` + +### Local (Development) +``` +c:\mnt\okcomputer\ +├── cache.db (SQLite database) +├── cache.db.backup-* (Automatic backups) +└── images\ (Image files) +``` + +## 🔒 Safety Features + +### Automatic Backups +- Existing local database is automatically backed up before sync +- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS` +- Keep recent backups manually or clean up old ones + +### Confirmation Prompts +- PowerShell script prompts for confirmation (unless `-Force` is used) +- Shows configuration before executing +- Safe to cancel at any time + +### Error Handling +- Validates SSH connection before starting +- Cleans up temporary files on remote server +- Reports clear error messages + +## ⚡ Performance Tips + +### Faster Image Sync with rsync +Install rsync for incremental image sync (only new/changed files): + +**Windows (WSL)**: +```powershell +wsl --install +wsl -d Ubuntu +sudo apt install rsync +``` + +**Windows (Chocolatey)**: +```powershell +choco install rsync +``` + +**Benefit**: First sync downloads everything, subsequent syncs only transfer changed files. + +Images can be synced separately when needed for image processing tests. + +## 🐛 Troubleshooting + +### SSH Connection Issues +```powershell +# Test SSH connection +ssh tour@athena.lan "echo 'Connection OK'" + +# Check SSH key +ssh-add -l +``` + +### Permission Denied +```bash +# Add SSH key (Linux/Mac) +chmod 600 ~/.ssh/id_rsa +ssh-add ~/.ssh/id_rsa + +# Windows: Use PuTTY or OpenSSH for Windows +``` + +### Database Locked Error +```powershell +# Make sure no other process is using the database +Get-Process | Where-Object {$_.Path -like "*java*"} | Stop-Process + +# Or restart the monitor +``` + +### Slow Image Sync +- Use rsync instead of scp (see Performance Tips) +- Consider syncing only database for code development +- Images only needed for object detection testing + +## 📝 Script Details + +### sync-production-data.sh (Bash) +- **Platform**: Linux, Mac, Git Bash on Windows +- **Best for**: Unix-like environments +- **Features**: Color output, progress bars, statistics + +## 🔄 Automation + +### Linux/Mac Cron +```bash +# Edit crontab +crontab -e + +# Add daily sync at 7 AM +0 7 * * * /path/to/auctiora/scripts/sync-production-data.sh --db-only +``` + +## 🆘 Support + +### Getting Help +```bash +# Bash +./scripts/sync-production-data.sh --help +``` + +### Common Commands +```powershell +# Check database size +ls c:\mnt\okcomputer\cache.db -h + +# View database contents +sqlite3 c:\mnt\okcomputer\cache.db + .tables + .schema lots + SELECT COUNT(*) FROM lots; + .quit + +# Check image count +(Get-ChildItem c:\mnt\okcomputer\images -Recurse -File).Count +``` + +## 📚 Related Documentation +- [Database Architecture](../wiki/DATABASE_ARCHITECTURE.md) +- [Integration Flowchart](../docs/INTEGRATION_FLOWCHART.md) +- [Main README](../README.md) + +--- + +**Last Updated**: December 2025 +**Maintainer**: Auctiora Development Team diff --git a/scripts/sync-production-data.sh b/scripts/sync-production-data.sh new file mode 100644 index 0000000..56070f5 --- /dev/null +++ b/scripts/sync-production-data.sh @@ -0,0 +1,171 @@ +#!/bin/bash +# +# Sync Production Data to Local +# +# This script copies the production SQLite database and images from the remote +# server (athena.lan) to your local development environment. +# +# Usage: +# ./scripts/sync-production-data.sh [--db-only|--images-only|--all] +# +# Options: +# --db-only Only sync the database (default) +# --images-only Only sync the images +# --all Sync both database and images +# --help Show this help message +# + +set -e # Exit on error + +# Configuration +REMOTE_HOST="tour@athena.lan" +REMOTE_VOLUME="shared-auction-data" +LOCAL_DB_PATH="c:/mnt/okcomputer/cache.db" +LOCAL_IMAGES_PATH="c:/mnt/okcomputer/images" +REMOTE_TMP="/tmp" + +# Colors for output +GREEN='\033[0;32m' +BLUE='\033[0;34m' +YELLOW='\033[1;33m' +RED='\033[0;31m' +NC='\033[0m' # No Color + +# Parse arguments +SYNC_MODE="db" # Default: database only + +case "${1:-}" in + --db-only) + SYNC_MODE="db" + ;; + --images-only) + SYNC_MODE="images" + ;; + --all) + SYNC_MODE="all" + ;; + --help|-h) + grep '^#' "$0" | sed 's/^# \?//' + exit 0 + ;; + "") + SYNC_MODE="db" + ;; + *) + echo -e "${RED}Error: Unknown option '$1'${NC}" + echo "Use --help for usage information" + exit 1 + ;; +esac + +echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}" +echo -e "${BLUE}║ Production Data Sync - Auctiora Monitor ║${NC}" +echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}" +echo "" + +# Function to sync database +sync_database() { + echo -e "${YELLOW}[1/3] Copying database from Docker volume to /tmp...${NC}" + ssh ${REMOTE_HOST} "docker run --rm -v ${REMOTE_VOLUME}:/data -v ${REMOTE_TMP}:${REMOTE_TMP} alpine cp /data/cache.db ${REMOTE_TMP}/cache.db" + + echo -e "${YELLOW}[2/3] Downloading database from remote server...${NC}" + # Create backup of existing local database + if [ -f "${LOCAL_DB_PATH}" ]; then + BACKUP_PATH="${LOCAL_DB_PATH}.backup-$(date +%Y%m%d-%H%M%S)" + echo -e "${BLUE} Backing up existing database to: ${BACKUP_PATH}${NC}" + cp "${LOCAL_DB_PATH}" "${BACKUP_PATH}" + fi + + # Download new database + scp ${REMOTE_HOST}:${REMOTE_TMP}/cache.db "${LOCAL_DB_PATH}" + + echo -e "${YELLOW}[3/3] Cleaning up remote /tmp...${NC}" + ssh ${REMOTE_HOST} "rm -f ${REMOTE_TMP}/cache.db" + + # Show database info + DB_SIZE=$(du -h "${LOCAL_DB_PATH}" | cut -f1) + echo -e "${GREEN}✓ Database synced successfully (${DB_SIZE})${NC}" + + # Show table counts + echo -e "${BLUE} Database statistics:${NC}" + sqlite3 "${LOCAL_DB_PATH}" </dev/null || true'" + + echo -e "${YELLOW}[3/4] Syncing images to local directory (this may take a while)...${NC}" + # Use rsync for efficient incremental sync + if command -v rsync &> /dev/null; then + echo -e "${BLUE} Using rsync for efficient transfer...${NC}" + rsync -avz --progress ${REMOTE_HOST}:${REMOTE_TMP}/auction-images/ "${LOCAL_IMAGES_PATH}/" + else + echo -e "${BLUE} Using scp for transfer (install rsync for faster incremental sync)...${NC}" + scp -r ${REMOTE_HOST}:${REMOTE_TMP}/auction-images/* "${LOCAL_IMAGES_PATH}/" + fi + + echo -e "${YELLOW}[4/4] Cleaning up remote /tmp...${NC}" + ssh ${REMOTE_HOST} "rm -rf ${REMOTE_TMP}/auction-images" + + # Show image stats + IMAGE_COUNT=$(find "${LOCAL_IMAGES_PATH}" -type f 2>/dev/null | wc -l) + IMAGE_SIZE=$(du -sh "${LOCAL_IMAGES_PATH}" 2>/dev/null | cut -f1) + echo -e "${GREEN}✓ Images synced successfully${NC}" + echo -e "${BLUE} Total images: ${IMAGE_COUNT}${NC}" + echo -e "${BLUE} Total size: ${IMAGE_SIZE}${NC}" +} + +# Execute sync based on mode +START_TIME=$(date +%s) + +case "$SYNC_MODE" in + db) + echo -e "${BLUE}Mode: Database only${NC}" + echo "" + sync_database + ;; + images) + echo -e "${BLUE}Mode: Images only${NC}" + echo "" + sync_images + ;; + all) + echo -e "${BLUE}Mode: Database + Images${NC}" + echo "" + sync_database + echo "" + sync_images + ;; +esac + +END_TIME=$(date +%s) +DURATION=$((END_TIME - START_TIME)) + +echo "" +echo -e "${GREEN}╔════════════════════════════════════════════════════════╗${NC}" +echo -e "${GREEN}║ Sync completed successfully in ${DURATION} seconds ║${NC}" +echo -e "${GREEN}╚════════════════════════════════════════════════════════╝${NC}" +echo "" +echo -e "${BLUE}Next steps:${NC}" +echo -e " 1. Verify data: sqlite3 ${LOCAL_DB_PATH} 'SELECT COUNT(*) FROM lots;'" +echo -e " 2. Start monitor: mvn quarkus:dev" +echo -e " 3. Open dashboard: http://localhost:8080" +echo ""