Initial clean commit
This commit is contained in:
109
docs/DATA_SYNC_SETUP.md
Normal file
109
docs/DATA_SYNC_SETUP.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Production Data Sync Setup
|
||||
|
||||
Quick reference for syncing production data from `athena.lan` to your local development environment.
|
||||
|
||||
## 🚀 One-Command Setup
|
||||
|
||||
### Linux/Mac
|
||||
```bash
|
||||
./scripts/sync-production-data.sh
|
||||
```
|
||||
|
||||
## 📋 Complete Usage
|
||||
|
||||
### Bash (Linux/Mac/Git Bash)
|
||||
```bash
|
||||
# Database only
|
||||
./scripts/sync-production-data.sh --db-only
|
||||
|
||||
# Everything
|
||||
./scripts/sync-production-data.sh --all
|
||||
|
||||
# Images only
|
||||
./scripts/sync-production-data.sh --images-only
|
||||
```
|
||||
|
||||
## 🔧 What It Does
|
||||
|
||||
1. **Connects to athena.lan** via SSH
|
||||
2. **Copies database** from Docker volume to /tmp
|
||||
3. **Downloads to local** machine (c:\mnt\okcomputer\cache.db)
|
||||
4. **Backs up** existing local database automatically
|
||||
5. **Shows statistics** (auction count, lot count, etc.)
|
||||
6. **Cleans up** temporary files on remote server
|
||||
|
||||
### With Images
|
||||
- Also syncs the `/data/images/` directory
|
||||
- Uses rsync for incremental sync (if available)
|
||||
- Can be large (several GB)
|
||||
|
||||
## 📊 What You Get
|
||||
|
||||
### Database (`cache.db`)
|
||||
- **~8.9 GB** of production data
|
||||
- 16,000+ lots
|
||||
- 536,000+ images metadata
|
||||
- Full auction history
|
||||
- HTTP cache from scraper
|
||||
|
||||
### Images (`images/`)
|
||||
- Downloaded lot images
|
||||
- Organized by lot ID
|
||||
- Variable size (can be large)
|
||||
|
||||
## ⚡ Quick Workflow
|
||||
|
||||
### Daily Development
|
||||
```powershell
|
||||
# Morning: Get fresh data
|
||||
.\scripts\Sync-ProductionData.sh -Force
|
||||
|
||||
# Develop & test
|
||||
mvn quarkus:dev
|
||||
|
||||
# View dashboard
|
||||
start http://localhost:8080
|
||||
```
|
||||
|
||||
## 🔒 Safety Features
|
||||
|
||||
- ✅ **Automatic backups** before overwriting
|
||||
- ✅ **Confirmation prompts** (unless `-Force`)
|
||||
- ✅ **Error handling** with clear messages
|
||||
- ✅ **Cleanup** of temporary files
|
||||
- ✅ **Non-destructive** - production data is never modified
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### "Permission denied" or SSH errors
|
||||
```bash
|
||||
# Test SSH connection
|
||||
ssh tour@athena.lan "echo OK"
|
||||
|
||||
# If fails, check your SSH key
|
||||
ssh-add -l
|
||||
```
|
||||
|
||||
### Database already exists
|
||||
- Script automatically backs up existing database
|
||||
- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS`
|
||||
|
||||
### Slow image transfer
|
||||
- Install rsync for 10x faster incremental sync
|
||||
- Or sync database only: `.\scripts\Sync-ProductionData.sh` (default)
|
||||
|
||||
## 📚 Full Documentation
|
||||
|
||||
See [scripts/README.md](../scripts/README.md) for:
|
||||
- Prerequisites
|
||||
- Performance tips
|
||||
- Automation setup
|
||||
- Detailed troubleshooting
|
||||
|
||||
## 🎯 Common Use Cases
|
||||
|
||||
**Quick Links**:
|
||||
- [Main README](../README.md)
|
||||
- [Scripts Documentation](../scripts/README.md)
|
||||
- [Integration Flowchart](INTEGRATION_FLOWCHART.md)
|
||||
- [Intelligence Features](INTELLIGENCE_FEATURES_SUMMARY.md)
|
||||
Reference in New Issue
Block a user