Initial clean commit
This commit is contained in:
Tour
206
scripts/README.md
Normal file
206
scripts/README.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# Auctiora Scripts
|
||||
|
||||
Utility scripts for managing the Auctiora auction monitoring system.
|
||||
|
||||
## 📦 Available Scripts
|
||||
|
||||
### 1. Production Data Sync
|
||||
|
||||
Sync production database and images from `athena.lan` to your local development environment.
|
||||
|
||||
#### Quick Start
|
||||
|
||||
**Linux/Mac (Bash)**:
|
||||
```bash
|
||||
# Make executable (first time only)
|
||||
chmod +x scripts/sync-production-data.sh
|
||||
|
||||
# Sync database only
|
||||
./scripts/sync-production-data.sh --db-only
|
||||
|
||||
# Sync everything
|
||||
./scripts/sync-production-data.sh --all
|
||||
|
||||
# Sync images only
|
||||
./scripts/sync-production-data.sh --images-only
|
||||
```
|
||||
|
||||
## 🔧 Prerequisites
|
||||
|
||||
### Required
|
||||
- **SSH Client**: OpenSSH or equivalent
|
||||
- Windows: Built-in on Windows 10+, or install [Git Bash](https://git-scm.com/downloads)
|
||||
- Linux/Mac: Pre-installed
|
||||
- **SCP**: Secure copy (usually comes with SSH)
|
||||
- **SSH Access**: SSH key configured for `tour@athena.lan`
|
||||
|
||||
### Optional
|
||||
- **rsync**: For efficient incremental image sync
|
||||
- Windows: Install via [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) or [Cygwin](https://www.cygwin.com/)
|
||||
- Linux/Mac: Usually pre-installed
|
||||
- **sqlite3**: For showing database statistics
|
||||
- Windows: Download from [sqlite.org](https://www.sqlite.org/download.html)
|
||||
- Linux: `sudo apt install sqlite3`
|
||||
- Mac: Pre-installed
|
||||
|
||||
## 📊 What Gets Synced
|
||||
|
||||
### Database (`cache.db`)
|
||||
- **Size**: ~8.9 GB (as of Dec 2024)
|
||||
- **Contains**:
|
||||
- Auctions metadata
|
||||
- Lots (kavels) with bid information
|
||||
- Images metadata and URLs
|
||||
- HTTP cache for scraper
|
||||
- **Local Path**: `c:\mnt\okcomputer\cache.db`
|
||||
|
||||
### Images Directory
|
||||
- **Size**: Varies (can be large)
|
||||
- **Contains**:
|
||||
- Downloaded lot images
|
||||
- Organized by lot ID
|
||||
- **Local Path**: `c:\mnt\okcomputer\images\`
|
||||
|
||||
## 🚀 Usage Examples
|
||||
|
||||
## 📁 File Locations
|
||||
|
||||
### Remote (Production)
|
||||
```
|
||||
athena.lan
|
||||
├── Docker Volume: shared-auction-data
|
||||
│ ├── /data/cache.db (SQLite database)
|
||||
│ └── /data/images/ (Image files)
|
||||
└── /tmp/ (Temporary staging area)
|
||||
```
|
||||
|
||||
### Local (Development)
|
||||
```
|
||||
c:\mnt\okcomputer\
|
||||
├── cache.db (SQLite database)
|
||||
├── cache.db.backup-* (Automatic backups)
|
||||
└── images\ (Image files)
|
||||
```
|
||||
|
||||
## 🔒 Safety Features
|
||||
|
||||
### Automatic Backups
|
||||
- Existing local database is automatically backed up before sync
|
||||
- Backup format: `cache.db.backup-YYYYMMDD-HHMMSS`
|
||||
- Keep recent backups manually or clean up old ones
|
||||
|
||||
### Confirmation Prompts
|
||||
- PowerShell script prompts for confirmation (unless `-Force` is used)
|
||||
- Shows configuration before executing
|
||||
- Safe to cancel at any time
|
||||
|
||||
### Error Handling
|
||||
- Validates SSH connection before starting
|
||||
- Cleans up temporary files on remote server
|
||||
- Reports clear error messages
|
||||
|
||||
## ⚡ Performance Tips
|
||||
|
||||
### Faster Image Sync with rsync
|
||||
Install rsync for incremental image sync (only new/changed files):
|
||||
|
||||
**Windows (WSL)**:
|
||||
```powershell
|
||||
wsl --install
|
||||
wsl -d Ubuntu
|
||||
sudo apt install rsync
|
||||
```
|
||||
|
||||
**Windows (Chocolatey)**:
|
||||
```powershell
|
||||
choco install rsync
|
||||
```
|
||||
|
||||
**Benefit**: First sync downloads everything, subsequent syncs only transfer changed files.
|
||||
|
||||
Images can be synced separately when needed for image processing tests.
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### SSH Connection Issues
|
||||
```powershell
|
||||
# Test SSH connection
|
||||
ssh tour@athena.lan "echo 'Connection OK'"
|
||||
|
||||
# Check SSH key
|
||||
ssh-add -l
|
||||
```
|
||||
|
||||
### Permission Denied
|
||||
```bash
|
||||
# Add SSH key (Linux/Mac)
|
||||
chmod 600 ~/.ssh/id_rsa
|
||||
ssh-add ~/.ssh/id_rsa
|
||||
|
||||
# Windows: Use PuTTY or OpenSSH for Windows
|
||||
```
|
||||
|
||||
### Database Locked Error
|
||||
```powershell
|
||||
# Make sure no other process is using the database
|
||||
Get-Process | Where-Object {$_.Path -like "*java*"} | Stop-Process
|
||||
|
||||
# Or restart the monitor
|
||||
```
|
||||
|
||||
### Slow Image Sync
|
||||
- Use rsync instead of scp (see Performance Tips)
|
||||
- Consider syncing only database for code development
|
||||
- Images only needed for object detection testing
|
||||
|
||||
## 📝 Script Details
|
||||
|
||||
### sync-production-data.sh (Bash)
|
||||
- **Platform**: Linux, Mac, Git Bash on Windows
|
||||
- **Best for**: Unix-like environments
|
||||
- **Features**: Color output, progress bars, statistics
|
||||
|
||||
## 🔄 Automation
|
||||
|
||||
### Linux/Mac Cron
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add daily sync at 7 AM
|
||||
0 7 * * * /path/to/auctiora/scripts/sync-production-data.sh --db-only
|
||||
```
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
### Getting Help
|
||||
```bash
|
||||
# Bash
|
||||
./scripts/sync-production-data.sh --help
|
||||
```
|
||||
|
||||
### Common Commands
|
||||
```powershell
|
||||
# Check database size
|
||||
ls c:\mnt\okcomputer\cache.db -h
|
||||
|
||||
# View database contents
|
||||
sqlite3 c:\mnt\okcomputer\cache.db
|
||||
.tables
|
||||
.schema lots
|
||||
SELECT COUNT(*) FROM lots;
|
||||
.quit
|
||||
|
||||
# Check image count
|
||||
(Get-ChildItem c:\mnt\okcomputer\images -Recurse -File).Count
|
||||
```
|
||||
|
||||
## 📚 Related Documentation
|
||||
- [Database Architecture](../wiki/DATABASE_ARCHITECTURE.md)
|
||||
- [Integration Flowchart](../docs/INTEGRATION_FLOWCHART.md)
|
||||
- [Main README](../README.md)
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: December 2025
|
||||
**Maintainer**: Auctiora Development Team
|
||||
Reference in New Issue
Block a user