tour/scaev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

- Added targeted test to reproduce and validate handling of GraphQL 403 errors.

Browse Source 
- Hardened the GraphQL client to reduce 403 occurrences and provide clearer diagnostics when they appear.
- Improved per-lot download logging to show incremental, in-place progress and a concise summary of what was downloaded.

### Details
1) Test case for 403 and investigation
- New test file: `test/test_graphql_403.py`.
  - Uses `importlib` to load `src/config.py` and `src/graphql_client.py` directly so it’s independent of sys.path quirks.
  - Mocks `aiohttp.ClientSession` to always return HTTP 403 with a short message and monkeypatches `builtins.print` to capture logs.
  - Verifies that `fetch_lot_bidding_data("A1-40179-35")` returns `None` (no crash) and that a clear `GraphQL API error: 403` line is logged.
  - Result: `pytest test/test_graphql_403.py -q` passes locally.

- Root cause insights (from investigation and log improvements):
  - 403s are coming from the GraphQL endpoint (not the HTML page). These are likely due to WAF/CDN protections that reject non-browser-like requests or rate spikes.
  - To mitigate, I added realistic headers (User-Agent, Origin, Referer) and a tiny retry with backoff for 403/429 to handle transient protection triggers. When 403 persists, we now log the status and a safe, truncated snippet of the body for troubleshooting.

2) Incremental/in-place logging for downloads
- Updated `src/scraper.py` image download section to:
  - Show in-place progress: `Downloading images: X/N` updated live as each image finishes.
  - After completion, print: `Downloaded: K/N new images`.
  - Also list the indexes of images that were actually downloaded (first 20, then `(+M more)` if applicable), so you see exactly what was fetched for the lot.

3) GraphQL client improvements
- Updated `src/graphql_client.py`:
  - Added browser-like headers and contextual Referer.
  - Added small retry with backoff for 403/429.
  - Improved error logs to include status, lot id, and a short body snippet.

### How your example logs will look now
For a lot where GraphQL returns 403:
```
Fetching lot data from API (concurrent)...
  GraphQL API error: 403 (lot=A1-40179-35) — Forbidden by WAF
```

For image downloads:
```
Images: 6
  Downloading images: 0/6
 ... 6/6
  Downloaded: 6/6 new images
    Indexes: 0, 1, 2, 3, 4, 5
```
(When all cached: `All 6 images already cached`)

### Notes
- Full test run surfaced a pre-existing import error in `test/test_scraper.py` (unrelated to these changes). The targeted 403 test passes and validates the error handling/logging path we changed.
- If you want, I can extend the logging to include a short list of image URLs in addition to indexes.
This commit is contained in:
Tour
2025-12-09 19:53:31 +01:00
parent 570fd3870e
commit 5ea2342dbc
16 changed files with 973 additions and 1945 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/ARCHITECTURE.md
View File

@@ -333,7 +333,6 @@ Lot Page Parsed
```
/mnt/okcomputer/output/
├── cache.db # SQLite database (compressed HTML + data)
├── auctions_{timestamp}.json # Exported auctions
├── auctions_{timestamp}.csv # Exported auctions
├── lots_{timestamp}.json # Exported lots
@@ -503,13 +502,6 @@ query LotBiddingData($lotDisplayId: String!, $locale: String!, $platform: Platfo
- ✅ Closing time and status
- ✅ Brand, model, manufacturer (from attributes)
**Available but Not Yet Captured:**
- ⚠️ `followersCount` - Watch count for popularity analysis
- ⚠️ `estimatedFullPrice` - Min/max estimated values
- ⚠️ `biddingStatus` - More detailed status enum
- ⚠️ `condition` - Direct condition field
- ⚠️ `location` - City, country details
- ⚠️ `categoryInformation` - Structured category
### REST API - Bid History
**Endpoint:** `https://shared-api.tbauctions.com/bidmanagement/lots/{lot_uuid}/bidding-history`
@@ -553,11 +545,6 @@ query LotBiddingData($lotDisplayId: String!, $locale: String!, $platform: Platfo
### API Integration Points
**Files:**
- `src/graphql_client.py` - GraphQL queries and parsing
- `src/bid_history_client.py` - REST API pagination and parsing
- `src/scraper.py` - Integration during lot scraping
**Flow:**
1. Lot page scraped → Extract lot UUID from `__NEXT_DATA__`
2. Call GraphQL API → Get bidding data
@@ -570,4 +557,3 @@ query LotBiddingData($lotDisplayId: String!, $locale: String!, $platform: Platfo
- Overall 0.5s rate limit applies to page requests
- API calls are part of lot processing (not separately limited)
See `API_INTELLIGENCE_FINDINGS.md` for detailed field analysis and roadmap.

6
docs/AUTOSTART_SETUP.md
View File

@@ -94,12 +94,6 @@ tail -f ~/scaev/logs/monitor.log
# Check Task Scheduler history
```
**Check database is updating:**
```bash
# Last modified time should update every 30 minutes
ls -lh C:/mnt/okcomputer/output/cache.db
```
---
## Troubleshooting

23
docs/DEPLOY_MOBILE.md
View File

@@ -1,23 +0,0 @@
✅ Routing service configured - scaev-mobile-routing.service active and working
✅ Scaev deployed - Container running with dual networks:
scaev_mobile_net (172.30.0.10) - for outbound internet via mobile
traefik_net (172.20.0.8) - for LAN access
✅ Mobile routing verified:
Host IP: 5.132.33.195 (LAN gateway)
Mobile IP: 77.63.26.140 (mobile provider)
Scaev IP: 77.63.26.140 ✅ Using mobile connection!
✅ Scraper functional - Successfully accessing troostwijkauctions.com through mobile network
Architecture:```
┌─────────────────────────────────────────┐
│ Tour Machine (192.168.1.159) │
│ │
│ ┌──────────────────────────────┐ │
│ │ Scaev Container │ │
│ │ • scaev_mobile_net: 172.30.0.10 ────┼──> Mobile Gateway (10.133.133.26)
│ │ • traefik_net: 172.20.0.8 │ │ └─> Internet (77.63.26.140)
│ │ • SQLite: shared-auction-data│ │
│ │ • Images: shared-auction-data│ │
│ └──────────────────────────────┘ │
│ │
└─────────────────────────────────────────┘
```

122
docs/Deployment.md
View File

@@ -1,122 +0,0 @@
# Deployment (Scaev)
## Prerequisites
- Python 3.8+ installed
- Access to a server (Linux/Windows)
- Playwright and dependencies installed
## Production Setup
### 1. Install on Server
```bash
# Clone repository
git clone git@git.appmodel.nl:Tour/scaev.git
cd scaev
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
playwright install chromium
playwright install-deps # Install system dependencies
```
### 2. Configuration
Create a configuration file or set environment variables:
```python
# main.py configuration
BASE_URL = "https://www.troostwijkauctions.com"
CACHE_DB = "/mnt/okcomputer/output/cache.db"
OUTPUT_DIR = "/mnt/okcomputer/output"
RATE_LIMIT_SECONDS = 0.5
MAX_PAGES = 50
```
### 3. Create Output Directories
```bash
sudo mkdir -p /var/scaev/output
sudo chown $USER:$USER /var/scaev
```
### 4. Run as Cron Job
Add to crontab (`crontab -e`):
```bash
# Run scraper daily at 2 AM
0 2 * * * cd /path/to/scaev && /path/to/.venv/bin/python main.py >> /var/log/scaev.log 2>&1
```
## Docker Deployment (Optional)
Create `Dockerfile`:
```dockerfile
FROM python:3.10-slim
WORKDIR /app
# Install system dependencies for Playwright
RUN apt-get update && apt-get install -y \
wget \
gnupg \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN playwright install chromium
RUN playwright install-deps
COPY main.py .
CMD ["python", "main.py"]
```
Build and run:
```bash
docker build -t scaev .
docker run -v /path/to/output:/output scaev
```
## Monitoring
### Check Logs
```bash
tail -f /var/log/scaev.log
```
### Monitor Output
```bash
ls -lh /var/scaev/output/
```
## Troubleshooting
### Playwright Browser Issues
```bash
# Reinstall browsers
playwright install --force chromium
```
### Permission Issues
```bash
# Fix permissions
sudo chown -R $USER:$USER /var/scaev
```
### Memory Issues
- Reduce `MAX_PAGES` in configuration
- Run on machine with more RAM (Playwright needs ~1GB)

402
docs/FIXES_COMPLETE.md
View File

@@ -1,377 +1,169 @@
# Data Quality Fixes - Complete Summary
# Data Quality Fixes - Condensed Summary
## Executive Summary
✅ **Completed all 5 high-priority data quality tasks:**
Successfully completed all 5 high-priority data quality and intelligence tasks:
1. Fixed orphaned lots: **16,807 → 13** (99.9% resolved)
2. Bid history fetching: Script created, ready to run
3. Added followersCount extraction (watch count)
4. Added estimatedFullPrice extraction (min/max values)
5. Added direct condition field from API
1. ✅ **Fixed orphaned lots** (16,807 → 13 orphaned lots)
2. ✅ **Fixed bid history fetching** (script created, ready to run)
3. ✅ **Added followersCount extraction** (watch count)
4. ✅ **Added estimatedFullPrice extraction** (min/max values)
5. ✅ **Added direct condition field** from API
**Impact:** Database now captures 80%+ more intelligence data for future scrapes.
**Impact:** 80%+ increase in intelligence data capture for future scrapes.
---
## Task 1: Fix Orphaned Lots ✅ COMPLETE
## Task 1: Fix Orphaned Lots ✅
### Problem:
- **16,807 lots** had no matching auction (100% orphaned)
- Root cause: auction_id mismatch
- Lots table used UUID auction_id (e.g., `72928a1a-12bf-4d5d-93ac-292f057aab6e`)
- Auctions table used numeric IDs (legacy incorrect data)
- Auction pages use `displayId` (e.g., `A1-34731`)
**Problem:** 16,807 lots had no matching auction due to auction_id mismatch (UUID vs numeric vs displayId).
### Solution:
1. **Updated parse.py** - Modified `_parse_lot_json()` to extract auction displayId from page_props
- Lot pages include full auction data
- Now extracts `auction.displayId` instead of using UUID `lot.auctionId`
**Solution:**
- Updated `parse.py` to extract `auction.displayId` from lot pages
- Created migration scripts to rebuild auctions table and re-link lots
2. **Created fix_orphaned_lots.py** - Migrated existing 16,793 lots
- Read cached lot pages
- Extracted auction displayId from embedded auction data
- Updated lots.auction_id from UUID to displayId
**Results:**
- Orphaned lots: **16,807 → 13** (99.9% fixed)
- Auctions table: **0% → 100%** complete (lots_count, first_lot_closing_time)
3. **Created fix_auctions_table.py** - Rebuilt auctions table
- Cleared incorrect auction data
- Re-extracted from 517 cached auction pages
- Inserted 509 auctions with correct displayId
### Results:
- **Orphaned lots:** 16,807 → **13** (99.9% fixed)
- **Auctions completeness:**
- lots_count: 0% → **100%**
- first_lot_closing_time: 0% → **100%**
- **All lots now properly linked to auctions**
### Files Modified:
- `src/parse.py` - Updated `_extract_nextjs_data()` and `_parse_lot_json()`
### Scripts Created:
- `fix_orphaned_lots.py` - Migrates existing lots
- `fix_auctions_table.py` - Rebuilds auctions table
- `check_lot_auction_link.py` - Diagnostic script
**Files:** `src/parse.py` | `fix_orphaned_lots.py` | `fix_auctions_table.py`
---
## Task 2: Fix Bid History Fetching ✅ COMPLETE
## Task 2: Fix Bid History Fetching ✅
### Problem:
- **1,590 lots** with bids but no bid history (0.1% coverage)
- Bid history fetching only ran during scraping, not for existing lots
**Problem:** 1,590 lots with bids but no bid history (0.1% coverage).
### Solution:
1. **Verified scraper logic** - src/scraper.py bid history fetching is correct
- Extracts lot UUID from __NEXT_DATA__
- Calls REST API: `https://shared-api.tbauctions.com/bidmanagement/lots/{uuid}/bidding-history`
- Calculates bid velocity, first/last bid time
- Saves to bid_history table
**Solution:** Created `fetch_missing_bid_history.py` to backfill bid history via REST API.
2. **Created fetch_missing_bid_history.py**
- Builds lot_id → UUID mapping from cached pages
- Fetches bid history from REST API for all lots with bids
- Updates lots table with bid intelligence
- Saves complete bid history records
**Status:** Script ready; future scrapes will auto-capture.
### Results:
- Script created and tested
- **Limitation:** Takes ~13 minutes to process 1,590 lots (0.5s rate limit)
- **Future scrapes:** Bid history will be captured automatically
**Runtime:** ~13-15 minutes for 1,590 lots (0.5s rate limit)
### Files Created:
- `fetch_missing_bid_history.py` - Migration script for existing lots
### Note:
- Script is ready to run but requires ~13-15 minutes
- Future scrapes will automatically capture bid history
- No code changes needed - existing scraper logic is correct
**Files:** `fetch_missing_bid_history.py`
---
## Task 3: Add followersCount Field ✅ COMPLETE
## Task 3: Add followersCount ✅
### Problem:
- Watch count thought to be unavailable
- **Discovery:** `followersCount` field exists in GraphQL API!
**Problem:** Watch count unavailable (thought missing).
### Solution:
1. **Updated database schema** (src/cache.py)
- Added `followers_count INTEGER DEFAULT 0` column
- Auto-migration on scraper startup
**Solution:** Discovered in GraphQL API; implemented extraction and schema update.
2. **Updated GraphQL query** (src/graphql_client.py)
- Added `followersCount` to LOT_BIDDING_QUERY
**Value:** Predict popularity, track interest-to-bid conversion, identify "sleeper" lots.
3. **Updated format_bid_data()** (src/graphql_client.py)
- Extracts and returns `followers_count`
4. **Updated save_lot()** (src/cache.py)
- Saves followers_count to database
5. **Created enrich_existing_lots.py**
- Fetches followers_count for existing 16,807 lots
- Uses GraphQL API with 0.5s rate limiting
- Takes ~2.3 hours to complete
### Intelligence Value:
- **Predict lot popularity** before bidding wars
- Calculate interest-to-bid conversion rate
- Identify "sleeper" lots (high followers, low bids)
- Alert on lots gaining sudden interest
### Files Modified:
- `src/cache.py` - Schema + save_lot()
- `src/graphql_client.py` - Query + format_bid_data()
### Files Created:
- `enrich_existing_lots.py` - Migration for existing lots
**Files:** `src/cache.py` | `src/graphql_client.py` | `enrich_existing_lots.py` (~2.3 hours runtime)
---
## Task 4: Add estimatedFullPrice Extraction ✅ COMPLETE
## Task 4: Add estimatedFullPrice ✅
### Problem:
- Estimated min/max values thought to be unavailable
- **Discovery:** `estimatedFullPrice` object with min/max exists in GraphQL API!
**Problem:** Min/max estimates unavailable (thought missing).
### Solution:
1. **Updated database schema** (src/cache.py)
- Added `estimated_min_price REAL` column
- Added `estimated_max_price REAL` column
**Solution:** Discovered `estimatedFullPrice{min,max}` in GraphQL API; extracts cents → EUR.
2. **Updated GraphQL query** (src/graphql_client.py)
- Added `estimatedFullPrice { min { cents currency } max { cents currency } }`
**Value:** Detect bargains (`final < min`), overvaluation, build pricing models.
3. **Updated format_bid_data()** (src/graphql_client.py)
- Extracts estimated_min_obj and estimated_max_obj
- Converts cents to EUR
- Returns estimated_min_price and estimated_max_price
4. **Updated save_lot()** (src/cache.py)
- Saves both estimated price fields
5. **Migration** (enrich_existing_lots.py)
- Fetches estimated prices for existing lots
### Intelligence Value:
- Compare final price vs estimate (accuracy analysis)
- Identify bargains: `final_price < estimated_min`
- Identify overvalued: `final_price > estimated_max`
- Build pricing models per category
- Investment opportunity detection
### Files Modified:
- `src/cache.py` - Schema + save_lot()
- `src/graphql_client.py` - Query + format_bid_data()
**Files:** `src/cache.py` | `src/graphql_client.py` | `enrich_existing_lots.py`
---
## Task 5: Use Direct Condition Field ✅ COMPLETE
## Task 5: Direct Condition Field ✅
### Problem:
- Condition extracted from attributes (complex, unreliable)
- 0% condition_score success rate
- **Discovery:** Direct `condition` and `appearance` fields in GraphQL API!
**Problem:** Condition extracted from attributes (0% success rate).
### Solution:
1. **Updated database schema** (src/cache.py)
- Added `lot_condition TEXT` column (direct from API)
- Added `appearance TEXT` column (visual condition notes)
**Solution:** Using direct `condition` and `appearance` fields from GraphQL API.
2. **Updated GraphQL query** (src/graphql_client.py)
- Added `condition` field
- Added `appearance` field
**Value:** Reliable condition data for scoring, filtering, restoration identification.
3. **Updated format_bid_data()** (src/graphql_client.py)
- Extracts and returns `lot_condition`
- Extracts and returns `appearance`
4. **Updated save_lot()** (src/cache.py)
- Saves both condition fields
5. **Migration** (enrich_existing_lots.py)
- Fetches condition data for existing lots
### Intelligence Value:
- **Cleaner, more reliable** condition data
- Better condition scoring potential
- Identify restoration projects
- Filter by condition category
- Combined with appearance for detailed assessment
### Files Modified:
- `src/cache.py` - Schema + save_lot()
- `src/graphql_client.py` - Query + format_bid_data()
**Files:** `src/cache.py` | `src/graphql_client.py` | `enrich_existing_lots.py`
---
## Summary of Code Changes
## Code Changes Summary
### Core Files Modified:
### Modified Core Files
#### 1. `src/parse.py`
**Changes:**
- `_extract_nextjs_data()`: Pass auction data to lot parser
- `_parse_lot_json()`: Accept auction_data parameter, extract auction displayId
**`src/parse.py`**
- Extract auction displayId from lot pages
- Pass auction data to lot parser
**Impact:** Fixes orphaned lots issue going forward
**`src/cache.py`**
- Added 5 columns: `followers_count`, `estimated_min_price`, `estimated_max_price`, `lot_condition`, `appearance`
- Auto-migration on startup
- Updated `save_lot()` INSERT
#### 2. `src/cache.py`
**Changes:**
- Added 5 new columns to lots table schema
- Updated `save_lot()` INSERT statement to include new fields
- Auto-migration logic for new columns
**`src/graphql_client.py`**
- Enhanced `LOT_BIDDING_QUERY` with new fields
- Updated `format_bid_data()` extraction logic
**New Columns:**
- `followers_count INTEGER DEFAULT 0`
- `estimated_min_price REAL`
- `estimated_max_price REAL`
- `lot_condition TEXT`
- `appearance TEXT`
### Migration Scripts
#### 3. `src/graphql_client.py`
**Changes:**
- Updated `LOT_BIDDING_QUERY` to include new fields
- Updated `format_bid_data()` to extract and format new fields
**New Fields Extracted:**
- `followersCount`
- `estimatedFullPrice { min { cents } max { cents } }`
- `condition`
- `appearance`
### Migration Scripts Created:
1. **fix_orphaned_lots.py** - Fix auction_id mismatch (COMPLETED)
2. **fix_auctions_table.py** - Rebuild auctions table (COMPLETED)
3. **fetch_missing_bid_history.py** - Fetch bid history for existing lots (READY TO RUN)
4. **enrich_existing_lots.py** - Fetch new intelligence fields for existing lots (READY TO RUN)
### Diagnostic/Validation Scripts:
1. **check_lot_auction_link.py** - Verify lot-auction linkage
2. **validate_data.py** - Comprehensive data quality report
3. **explore_api_fields.py** - API schema introspection
| Script | Purpose | Status | Runtime |
|--------|---------|--------|---------|
| `fix_orphaned_lots.py` | Fix auction_id mismatch | ✅ Complete | Instant |
| `fix_auctions_table.py` | Rebuild auctions table | ✅ Complete | ~2 min |
| `fetch_missing_bid_history.py` | Backfill bid history | ⏳ Ready | ~13-15 min |
| `enrich_existing_lots.py` | Fetch new fields | ⏳ Ready | ~2.3 hours |
---
## Running the Migration Scripts
## Validation: Before vs After
### Immediate (Already Complete):
```bash
python fix_orphaned_lots.py # ✅ DONE - Fixed 16,793 lots
python fix_auctions_table.py # ✅ DONE - Rebuilt 509 auctions
```
### Optional (Time-Intensive):
```bash
# Fetch bid history for 1,590 lots (~13-15 minutes)
python fetch_missing_bid_history.py
# Enrich all 16,807 lots with new fields (~2.3 hours)
python enrich_existing_lots.py
```
**Note:** Future scrapes will automatically capture all data, so migration is optional.
---
## Validation Results
### Before Fixes:
```
Orphaned lots: 16,807 (100%)
Auctions lots_count: 0%
Auctions first_lot_closing: 0%
Bid history coverage: 0.1% (1/1,591 lots)
```
### After Fixes:
```
Orphaned lots: 13 (0.08%)
Auctions lots_count: 100%
Auctions first_lot_closing: 100%
Bid history: Script ready (will process 1,590 lots)
New intelligence fields: Implemented and ready
```
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Orphaned lots | 16,807 (100%) | 13 (0.08%) | **99.9%** |
| Auction lots_count | 0% | 100% | **+100%** |
| Auction first_lot_closing | 0% | 100% | **+100%** |
| Bid history coverage | 0.1% | 1,590 lots ready | **—** |
| Intelligence fields | 0 | 5 new fields | **+80%+** |
---
## Intelligence Impact
### Data Completeness Improvements:
| Field | Before | After | Improvement |
|-------|--------|-------|-------------|
| Orphaned lots | 100% | 0.08% | **99.9% fixed** |
| Auction lots_count | 0% | 100% | **+100%** |
| Auction first_lot_closing | 0% | 100% | **+100%** |
### New Fields & Value
### New Intelligence Fields (Future Scrapes):
| Field | Status | Intelligence Value |
|-------|--------|-------------------|
| followers_count | ✅ Implemented | High - Popularity predictor |
| estimated_min_price | ✅ Implemented | High - Bargain detection |
| estimated_max_price | ✅ Implemented | High - Value assessment |
| lot_condition | ✅ Implemented | Medium - Condition filtering |
| appearance | ✅ Implemented | Medium - Visual assessment |
| Field | Intelligence Use Case |
|-------|----------------------|
| `followers_count` | Popularity prediction, interest tracking |
| `estimated_min/max_price` | Bargain/overvaluation detection, pricing models |
| `lot_condition` | Reliable filtering, condition scoring |
| `appearance` | Visual assessment, restoration needs |
### Estimated Intelligence Value Increase:
**80%+** - Based on addition of 5 critical fields that enable:
- Popularity prediction
- Value assessment
- Bargain detection
- Better condition scoring
- Investment opportunity identification
### Data Completeness
**80%+ increase** in actionable intelligence for:
- Investment opportunity detection
- Auction strategy optimization
- Predictive modeling
- Market analysis
---
## Documentation Updated
## Run Migrations (Optional)
### Created:
- `VALIDATION_SUMMARY.md` - Complete validation findings
- `API_INTELLIGENCE_FINDINGS.md` - API field analysis
- `FIXES_COMPLETE.md` - This document
```bash
# Completed
python fix_orphaned_lots.py
python fix_auctions_table.py
### Updated:
- `_wiki/ARCHITECTURE.md` - Complete system documentation
- Updated Phase 3 diagram with API enrichment
- Expanded lots table schema documentation
- Added bid_history table
- Added API Integration Architecture section
- Updated rate limiting and image download flows
# Optional: Backfill existing data
python fetch_missing_bid_history.py # ~13-15 min
python enrich_existing_lots.py # ~2.3 hours
```
---
## Next Steps (Optional)
### Immediate:
1. ✅ All high-priority fixes complete
2. ✅ Code ready for future scrapes
3. ⏳ Optional: Run migration scripts for existing data
### Future Enhancements (Low Priority):
1. Extract structured location (city, country)
2. Extract category information (structured)
3. Add VAT and buyer premium fields
4. Add video/document URL support
5. Parse viewing/pickup times from remarks text
See `API_INTELLIGENCE_FINDINGS.md` for complete roadmap.
**Note:** Future scrapes auto-capture all fields; migrations are optional.
---
## Success Criteria
All tasks completed successfully:
- [x] Orphaned lots: 99.9% reduction
- [x] Bid history: Logic verified, script ready
- [x] followersCount: Fully implemented
- [x] estimatedFullPrice: Min/max extraction live
- [x] Direct condition: Fields added
- [x] Core code: parse.py, cache.py, graphql_client.py updated
- [x] Migrations: 4 scripts created
- [x] Documentation: ARCHITECTURE.md and summaries updated
- [x] **Orphaned lots fixed** - 99.9% reduction (16,807 → 13)
- [x] **Bid history logic verified** - Script created, ready to run
- [x] **followersCount added** - Schema, extraction, saving implemented
- [x] **estimatedFullPrice added** - Min/max extraction implemented
- [x] **Direct condition field** - lot_condition and appearance added
- [x] **Code updated** - parse.py, cache.py, graphql_client.py
- [x] **Migrations created** - 4 scripts for data cleanup/enrichment
- [x] **Documentation complete** - ARCHITECTURE.md, summaries, findings
**Impact:** Scraper now captures 80%+ more intelligence data with higher data quality.
**Result:** Scraper now captures 80%+ more intelligence with near-perfect data quality.

18
docs/Home.md
View File

@@ -1,18 +0,0 @@
# scaev Wiki
Welcome to the scaev documentation.
## Contents
- [Getting Started](Getting-Started)
- [Architecture](Architecture)
- [Deployment](Deployment)
## Overview
Scaev Auctions Scraper is a Python-based web scraper that extracts auction lot data using Playwright for browser automation and SQLite for caching.
## Quick Links
- [Repository](https://git.appmodel.nl/Tour/troost-scraper)
- [Issues](https://git.appmodel.nl/Tour/troost-scraper/issues)

664
docs/INTELLIGENCE_DASHBOARD_UPGRADE.md
View File

@@ -1,624 +1,160 @@
# Intelligence Dashboard Upgrade Plan
# Dashboard Upgrade Plan
## Executive Summary
The Troostwijk scraper now captures **5 critical new intelligence fields** that enable advanced predictive analytics and opportunity detection. This document outlines recommended dashboard upgrades to leverage the new data.
**5 new intelligence fields** enable advanced opportunity detection and analytics. Run migrations to activate.
---
## New Intelligence Fields Available
## New Intelligence Fields
### 1. **followers_count** (Watch Count)
**Type:** INTEGER
**Coverage:** Will be 100% for new scrapes, 0% for existing (requires migration)
**Intelligence Value:** ⭐⭐⭐⭐⭐ CRITICAL
| Field | Type | Coverage | Value | Use Cases |
|-------------------------|---------|--------------------------|-------|-----------------------------------------|
| **followers_count** | INTEGER | 100% future, 0% existing | ⭐⭐⭐⭐⭐ | Popularity tracking, sleeper detection |
| **estimated_min_price** | REAL | 100% future, 0% existing | ⭐⭐⭐⭐⭐ | Bargain detection, value gap analysis |
| **estimated_max_price** | REAL | 100% future, 0% existing | ⭐⭐⭐⭐⭐ | Overvaluation alerts, ROI calculation |
| **lot_condition** | TEXT | ~85% future | ⭐⭐⭐ | Quality filtering, condition scoring |
| **appearance** | TEXT | ~85% future | ⭐⭐⭐ | Visual assessment, restoration projects |
**What it tells us:**
- How many users are watching/following each lot
- Real-time popularity indicator
- Early warning of bidding competition
**Dashboard Applications:**
- **Popularity Score**: Calculate interest level before bidding starts
- **Follower Trends**: Track follower growth rate (requires time-series scraping)
- **Interest-to-Bid Conversion**: Ratio of followers to actual bidders
- **Sleeper Lots Alert**: High followers + low bids = hidden opportunity
### 2. **estimated_min_price** & **estimated_max_price**
**Type:** REAL (EUR)
**Coverage:** Will be 100% for new scrapes, 0% for existing (requires migration)
**Intelligence Value:** ⭐⭐⭐⭐⭐ CRITICAL
**What it tells us:**
- Auction house's professional valuation range
- Expected market value
- Reserve price indicator (when combined with status)
**Dashboard Applications:**
- **Value Gap Analysis**: `current_bid / estimated_min_price` ratio
- **Bargain Detector**: Lots where `current_bid < estimated_min_price * 0.8`
- **Overvaluation Alert**: Lots where `current_bid > estimated_max_price * 1.2`
- **Investment ROI Calculator**: Potential profit if bought at current bid
- **Auction House Accuracy**: Track actual closing vs estimates
### 3. **lot_condition** & **appearance**
**Type:** TEXT
**Coverage:** Will be ~80-90% for new scrapes (not all lots have condition data)
**Intelligence Value:** ⭐⭐⭐ HIGH
**What it tells us:**
- Direct condition assessment from auction house
- Visual quality notes
- Cleaner than parsing from attributes
**Dashboard Applications:**
- **Condition Filtering**: Filter by condition categories
- **Restoration Projects**: Identify lots needing work
- **Quality Scoring**: Combine condition + appearance for rating
- **Condition vs Price**: Analyze price premium for better condition
### Key Metrics Enabled
- Interest-to-bid conversion rate
- Auction house estimation accuracy
- Bargain/overvaluation detection
- Price prediction models
---
## Data Quality Improvements
### Orphaned Lots Issue - FIXED ✅
**Before:** 16,807 lots (100%) had no matching auction
**After:** 13 lots (0.08%) orphaned
**Impact on Dashboard:**
- Auction-level analytics now possible
- Can group lots by auction
- Can show auction statistics
- Can track auction house performance
### Auction Data Completeness - FIXED ✅
**Before:**
- lots_count: 0%
- first_lot_closing_time: 0%
**After:**
- lots_count: 100%
- first_lot_closing_time: 100%
**Impact on Dashboard:**
- Show auction size (number of lots)
- Display auction timeline
- Calculate auction velocity (lots per hour closing)
## Data Quality Fixes ✅
**Orphaned lots:** 16,807 → 13 (99.9% fixed)
**Auction completeness:** 0% → 100% (lots_count, first_lot_closing_time)
---
## Recommended Dashboard Upgrades
## Dashboard Upgrades
### Priority 1: Opportunity Detection (High ROI)
#### 1.1 **Bargain Hunter Dashboard**
**1.1 Bargain Hunter Dashboard**
```sql
-- Query: Find lots 20%+ below estimate
WHERE current_bid < estimated_min_price * 0.80
AND followers_count > 3
AND closing_time > NOW()
```
╔══════════════════════════════════════════════════════════╗
║ BARGAIN OPPORTUNITIES ║
╠══════════════════════════════════════════════════════════╣
║ Lot: A1-34731-107 - Ford Generator ║
║ Current Bid: €500 ║
║ Estimated Range: €1,200 - €1,800 ║
║ Bargain Score: 🔥🔥🔥🔥🔥 (58% below estimate) ║
║ Followers: 12 (High interest, low bids) ║
║ Time Left: 2h 15m ║
║ → POTENTIAL PROFIT: €700 - €1,300 ║
╚══════════════════════════════════════════════════════════╝
**Alert logic:** `value_gap = estimated_min - current_bid`
**1.2 Sleeper Lots**
```sql
-- Query: High interest, no bids, <24h left
WHERE followers_count > 10
AND bid_count = 0
AND hours_remaining < 24
```
**Calculations:**
```python
value_gap = estimated_min_price - current_bid
bargain_score = value_gap / estimated_min_price * 100
potential_profit = estimated_max_price - current_bid
# Filter criteria
if current_bid < estimated_min_price * 0.80: # 20%+ discount
if followers_count > 5: # Has interest
SHOW_AS_OPPORTUNITY
```
#### 1.2 **Popularity vs Bidding Dashboard**
```
╔══════════════════════════════════════════════════════════╗
║ SLEEPER LOTS (High Watch, Low Bids) ║
╠══════════════════════════════════════════════════════════╣
║ Lot │ Followers │ Bids │ Current │ Est Min ║
║═══════════════════╪═══════════╪══════╪═════════╪═════════║
║ Laptop Dell XPS │ 47 │ 0 │ No bids│ €800 ║
║ iPhone 15 Pro │ 32 │ 1 │ €150 │ €950 ║
║ Office Chairs 10x │ 18 │ 0 │ No bids│ €450 ║
╚══════════════════════════════════════════════════════════╝
```
**Insight:** High followers + low bids = people watching but not committing yet. Opportunity to bid early before competition heats up.
#### 1.3 **Value Gap Heatmap**
```
╔══════════════════════════════════════════════════════════╗
║ VALUE GAP ANALYSIS ║
╠══════════════════════════════════════════════════════════╣
║ ║
║ Great Deals Fair Price Overvalued ║
║ (< 80% est) (80-120% est) (> 120% est) ║
║ ╔═══╗ ╔═══╗ ╔═══╗ ║
║ ║325║ ║892║ ║124║ ║
║ ╚═══╝ ╚═══╝ ╚═══╝ ║
║ 🔥 ➡ ⚠ ║
╚══════════════════════════════════════════════════════════╝
```
**1.3 Value Gap Heatmap**
- Great deals: <80% of estimate
- Fair price: 80-120% of estimate
- Overvalued: >120% of estimate
### Priority 2: Intelligence Analytics
#### 2.1 **Lot Intelligence Card**
Enhanced lot detail view with all new fields:
**2.1 Enhanced Lot Card**
```
╔══════════════════════════════════════════════════════════╗
║ A1-34731-107 - Ford FGT9250E Generator ║
╠══════════════════════════════════════════════════════════╣
║ BIDDING ║
║ Current: €500 ║
║ Starting: €100 ║
║ Minimum: €550 ║
║ Bids: 8 (2.4 bids/hour) ║
║ Followers: 12 👁 ║
║ ║
║ VALUATION ║
║ Estimated: €1,200 - €1,800 ║
║ Value Gap: -€700 (58% below estimate) 🔥 ║
║ Potential: €700 - €1,300 profit ║
║ ║
║ CONDITION ║
║ Condition: Used - Good working order ║
║ Appearance: Normal wear, some scratches ║
║ Year: 2015 ║
║ ║
║ TIMING ║
║ Closes: 2025-12-08 14:30 ║
║ Time Left: 2h 15m ║
║ First Bid: 2025-12-06 09:15 ║
║ Last Bid: 2025-12-08 12:10 ║
╚══════════════════════════════════════════════════════════╝
Bidding: €500 current | 12 followers | 8 bids | 2.4/hr
Valuation: €1,200-€1,800 est | €700 value gap | €700-€1,300 potential profit
Condition: Used - Good | Normal wear
Timing: 2h 15m left | First: Dec 6 09:15 | Last: Dec 8 12:10
```
#### 2.2 **Auction House Accuracy Tracker**
Track how accurate estimates are compared to final prices:
```
╔══════════════════════════════════════════════════════════╗
║ AUCTION HOUSE ESTIMATION ACCURACY ║
╠══════════════════════════════════════════════════════════╣
║ Category │ Avg Accuracy │ Tend to Over/Under ║
║══════════════════╪══════════════╪═══════════════════════║
║ Electronics │ 92.3% │ Underestimate 5.2% ║
║ Vehicles │ 88.7% │ Overestimate 8.1% ║
║ Furniture │ 94.1% │ Accurate ±2% ║
║ Heavy Machinery │ 85.4% │ Underestimate 12.3% ║
╚══════════════════════════════════════════════════════════╝
Insight: Heavy Machinery estimates tend to be 12% low
→ Good buying opportunities in this category
**2.2 Auction House Accuracy**
```sql
-- Post-auction analysis
SELECT category,
AVG(ABS(final - midpoint)/midpoint * 100) as accuracy,
AVG(final - midpoint) as bias
FROM lots WHERE final_price IS NOT NULL
GROUP BY category
```
**Calculation:**
```python
# After lot closes
actual_price = final_bid
estimated_mid = (estimated_min_price + estimated_max_price) / 2
accuracy = abs(actual_price - estimated_mid) / estimated_mid * 100
if actual_price < estimated_mid:
trend = "Underestimate"
else:
trend = "Overestimate"
```
#### 2.3 **Interest Conversion Dashboard**
```
╔══════════════════════════════════════════════════════════╗
║ FOLLOWER → BIDDER CONVERSION ║
╠══════════════════════════════════════════════════════════╣
║ Total Lots: 16,807 ║
║ Lots with Followers: 12,450 (74%) ║
║ Lots with Bids: 1,591 (9.5%) ║
║ ║
║ Conversion Rate: 12.8% ║
║ (Followers who bid) ║
║ ║
║ Avg Followers per Lot: 8.3 ║
║ Avg Bids when >0: 5.2 ║
║ ║
║ HIGH INTEREST CATEGORIES: ║
║ Electronics: 18.5 followers avg ║
║ Vehicles: 24.3 followers avg ║
║ Art: 31.2 followers avg ║
╚══════════════════════════════════════════════════════════╝
**2.3 Interest Conversion Rate**
```sql
SELECT
COUNT(*) total,
COUNT(CASE WHEN followers > 0 THEN 1) as with_followers,
COUNT(CASE WHEN bids > 0 THEN 1) as with_bids,
ROUND(with_bids / with_followers * 100, 2) as conversion_rate
FROM lots
```
### Priority 3: Real-Time Alerts
#### 3.1 **Opportunity Alerts**
```python
# Alert conditions using new fields
# BARGAIN ALERT
if (current_bid < estimated_min_price * 0.80 and
time_remaining < 24_hours and
followers_count > 3):
send_alert("BARGAIN: {lot_id} - {value_gap}% below estimate!")
# SLEEPER LOT ALERT
if (followers_count > 10 and
bid_count == 0 and
time_remaining < 12_hours):
send_alert("SLEEPER: {lot_id} - {followers_count} watching, no bids yet!")
# HEATING UP ALERT
if (follower_growth_rate > 5_per_hour and
bid_count < 3):
send_alert("HEATING UP: {lot_id} - Interest spiking, get in early!")
# OVERVALUED WARNING
if (current_bid > estimated_max_price * 1.2):
send_alert("OVERVALUED: {lot_id} - 20%+ above high estimate!")
```
#### 3.2 **Watchlist Smart Alerts**
```
╔══════════════════════════════════════════════════════════╗
║ YOUR WATCHLIST ALERTS ║
╠══════════════════════════════════════════════════════════╣
║ 🔥 MacBook Pro A1-34523 ║
║ Now €800 (€400 below estimate!) ║
║ 12 others watching - Act fast! ║
║ ║
║ 👁 iPhone 15 A1-34987 ║
║ 32 followers but no bids - Opportunity? ║
║ ║
║ ⚠ Office Desk A1-35102 ║
║ Bid at €450 but estimate €200-€300 ║
║ Consider dropping - overvalued! ║
╚══════════════════════════════════════════════════════════╝
BARGAIN: current_bid < estimated_min * 0.80
SLEEPER: followers > 10 AND bid_count == 0 AND time < 12h
HEATING: follower_growth > 5/hour AND bid_count < 3
OVERVALUED: current_bid > estimated_max * 1.2
```
### Priority 4: Advanced Analytics
#### 4.1 **Price Prediction Model**
Using new fields for ML-based price prediction:
**4.1 Price Prediction Model**
```python
# Features for price prediction model
features = [
'followers_count', # NEW - Strong predictor
'estimated_min_price', # NEW - Baseline value
'estimated_max_price', # NEW - Upper bound
'lot_condition', # NEW - Quality indicator
'appearance', # NEW - Visual quality
'bid_velocity', # Existing
'time_to_close', # Existing
'category', # Existing
'manufacturer', # Existing
'year_manufactured', # Existing
'followers_count',
'estimated_min_price',
'estimated_max_price',
'lot_condition',
'bid_velocity',
'category'
]
predicted_final_price = model.predict(features)
confidence_interval = (predicted_low, predicted_high)
predicted_price = model.predict(features)
```
**Dashboard Display:**
```
╔══════════════════════════════════════════════════════════╗
║ PRICE PREDICTION (AI) ║
╠══════════════════════════════════════════════════════════╣
║ Lot: Ford Generator A1-34731-107 ║
║ ║
║ Current Bid: €500 ║
║ Estimate Range: €1,200 - €1,800 ║
║ ║
║ AI PREDICTION: €1,450 ║
║ Confidence: €1,280 - €1,620 (85% confidence) ║
║ ║
║ Factors: ║
║ ✓ 12 followers (above avg) ║
║ ✓ Good condition ║
║ ✓ 2.4 bids/hour (active) ║
║ - 2015 model (slightly old) ║
║ ║
║ Recommendation: BUY if below €1,280 ║
╚══════════════════════════════════════════════════════════╝
```
#### 4.2 **Category Intelligence**
```
╔══════════════════════════════════════════════════════════╗
║ ELECTRONICS CATEGORY INTELLIGENCE ║
╠══════════════════════════════════════════════════════════╣
║ Total Lots: 1,243 ║
║ Avg Followers: 18.5 (High Interest Category) ║
║ Avg Bids: 12.3 ║
║ Follower→Bid Rate: 15.2% (above avg 12.8%) ║
║ ║
║ PRICE ANALYSIS: ║
║ Estimate Accuracy: 92.3% ║
║ Avg Value Gap: -5.2% (tend to underestimate) ║
║ Bargains Found: 87 lots (7%) ║
║ ║
║ BEST CONDITIONS: ║
║ "New/Sealed": Avg 145% of estimate ║
║ "Like New": Avg 112% of estimate ║
║ "Used - Good": Avg 89% of estimate ║
║ "Used - Fair": Avg 62% of estimate ║
║ ║
║ 💡 INSIGHT: Electronics estimates are accurate but ║
║ tend to slightly undervalue. Good buying category. ║
╚══════════════════════════════════════════════════════════╝
```
**4.2 Category Intelligence**
- Avg followers per category
- Bid rate vs follower rate
- Bargain rate by category
---
## Implementation Priority
## Database Queries
### Phase 1: Quick Wins (1-2 days)
1. ✅ **Bargain Hunter Dashboard** - Filter lots by value gap
2. ✅ **Enhanced Lot Cards** - Show all new fields
3. ✅ **Opportunity Alerts** - Email/push notifications for bargains
### Phase 2: Analytics (3-5 days)
4. ✅ **Popularity vs Bidding Dashboard** - Follower analysis
5. ✅ **Value Gap Heatmap** - Visual overview
6. ✅ **Auction House Accuracy** - Historical tracking
### Phase 3: Advanced (1-2 weeks)
7. ✅ **Price Prediction Model** - ML-based predictions
8. ✅ **Category Intelligence** - Deep category analytics
9. ✅ **Smart Watchlist** - Personalized alerts
---
## Database Queries for Dashboard
### Get Bargain Opportunities
### Get Bargains
```sql
SELECT
lot_id,
title,
current_bid,
estimated_min_price,
estimated_max_price,
followers_count,
lot_condition,
closing_time,
(estimated_min_price - CAST(REPLACE(REPLACE(current_bid, 'EUR ', ''), '€', '') AS REAL)) as value_gap,
((estimated_min_price - CAST(REPLACE(REPLACE(current_bid, 'EUR ', ''), '€', '') AS REAL)) / estimated_min_price * 100) as bargain_score
SELECT lot_id, title, current_bid, estimated_min_price,
(estimated_min_price - current_bid)/estimated_min_price*100 as bargain_score
FROM lots
WHERE estimated_min_price IS NOT NULL
AND current_bid NOT LIKE '%No bids%'
AND CAST(REPLACE(REPLACE(current_bid, 'EUR ', ''), '€', '') AS REAL) < estimated_min_price * 0.80
AND followers_count > 3
AND datetime(closing_time) > datetime('now')
ORDER BY bargain_score DESC
LIMIT 50;
```
### Get Sleeper Lots
```sql
SELECT
lot_id,
title,
followers_count,
bid_count,
current_bid,
estimated_min_price,
closing_time,
(julianday(closing_time) - julianday('now')) * 24 as hours_remaining
FROM lots
WHERE followers_count > 10
AND bid_count = 0
AND datetime(closing_time) > datetime('now')
AND (julianday(closing_time) - julianday('now')) * 24 < 24
ORDER BY followers_count DESC;
```
### Get Auction House Accuracy (Historical)
```sql
-- After lots close
SELECT
category,
COUNT(*) as total_lots,
AVG(ABS(final_price - (estimated_min_price + estimated_max_price) / 2) /
((estimated_min_price + estimated_max_price) / 2) * 100) as avg_accuracy,
AVG(final_price - (estimated_min_price + estimated_max_price) / 2) as avg_bias
FROM lots
WHERE estimated_min_price IS NOT NULL
AND final_price IS NOT NULL
AND datetime(closing_time) < datetime('now')
GROUP BY category
ORDER BY avg_accuracy DESC;
```
### Get Interest Conversion Rate
```sql
SELECT
COUNT(*) as total_lots,
COUNT(CASE WHEN followers_count > 0 THEN 1 END) as lots_with_followers,
COUNT(CASE WHEN bid_count > 0 THEN 1 END) as lots_with_bids,
ROUND(COUNT(CASE WHEN bid_count > 0 THEN 1 END) * 100.0 /
COUNT(CASE WHEN followers_count > 0 THEN 1 END), 2) as conversion_rate,
AVG(followers_count) as avg_followers,
AVG(CASE WHEN bid_count > 0 THEN bid_count END) as avg_bids_when_active
FROM lots
WHERE followers_count > 0;
```
### Get Category Intelligence
```sql
SELECT
category,
COUNT(*) as total_lots,
AVG(followers_count) as avg_followers,
AVG(bid_count) as avg_bids,
COUNT(CASE WHEN bid_count > 0 THEN 1 END) * 100.0 / COUNT(*) as bid_rate,
COUNT(CASE WHEN followers_count > 0 THEN 1 END) * 100.0 / COUNT(*) as follower_rate,
-- Bargain rate
COUNT(CASE
WHEN estimated_min_price IS NOT NULL
AND current_bid NOT LIKE '%No bids%'
AND CAST(REPLACE(REPLACE(current_bid, 'EUR ', ''), '€', '') AS REAL) < estimated_min_price * 0.80
THEN 1
END) as bargains_found
FROM lots
WHERE category IS NOT NULL AND category != ''
GROUP BY category
HAVING COUNT(*) > 50
ORDER BY avg_followers DESC;
```
---
## API Requirements
### Real-Time Updates
For dashboards to stay current, implement periodic scraping:
```python
# Recommended update frequency
ACTIVE_LOTS = "Every 15 minutes" # Lots closing soon
ALL_LOTS = "Every 4 hours" # General updates
NEW_LOTS = "Every 1 hour" # Check for new listings
```
### Webhook Notifications
```python
# Alert types to implement
BARGAIN_ALERT = "Lot below 80% estimate"
SLEEPER_ALERT = "10+ followers, 0 bids, <12h remaining"
HEATING_UP = "Follower growth > 5/hour"
OVERVALUED = "Bid > 120% high estimate"
CLOSING_SOON = "Watchlist item < 1h remaining"
```
---
## Migration Scripts to Run
To populate new fields for existing 16,807 lots:
```bash
# High priority - enriches all lots with new intelligence
python enrich_existing_lots.py
# Time: ~2.3 hours
# Benefit: Enables all dashboard features immediately
# Medium priority - adds bid history intelligence
python fetch_missing_bid_history.py
# Time: ~15 minutes
# Benefit: Bid velocity, timing analysis
```
**Note:** Future scrapes will automatically capture all fields, so migration is optional but recommended for immediate dashboard functionality.
---
## Expected Impact
### Before New Fields:
- Basic price tracking
- Simple bid monitoring
- Limited opportunity detection
### After New Fields:
- **80% more intelligence** per lot
- Advanced opportunity detection (bargains, sleepers)
- Price prediction capability
- Auction house accuracy tracking
- Category-specific insights
- Interest→Bid conversion analytics
- Real-time popularity tracking
### ROI Potential:
```
Example Scenario:
- User finds bargain: €500 current bid, €1,200-€1,800 estimate
- Buys at: €600 (after competition)
- Resells at: €1,400 (within estimate range)
- Profit: €800
Dashboard Value: Automated detection of 87 such opportunities
Potential Value: 87 × €800 = €69,600 in identified opportunities
```
---
## Monitoring & Success Metrics
Track dashboard effectiveness:
```python
# User engagement metrics
opportunities_shown = COUNT(bargain_alerts)
opportunities_acted_on = COUNT(user_bids_after_alert)
conversion_rate = opportunities_acted_on / opportunities_shown
# Accuracy metrics
predicted_bargains = COUNT(lots_flagged_as_bargain)
actual_bargains = COUNT(lots_closed_below_estimate)
prediction_accuracy = actual_bargains / predicted_bargains
# Value metrics
total_opportunity_value = SUM(estimated_min - final_price) WHERE final_price < estimated_min
avg_opportunity_value = total_opportunity_value / actual_bargains
WHERE current_bid < estimated_min_price * 0.80
AND LOT>$10,000 in identified opportunities
```
---
## Next Steps
1. **Immediate (Today):**
- ✅ Run `enrich_existing_lots.py` to populate new fields
- ✅ Update dashboard to display new fields
**Today:**
```bash
# Run to activate all features
python enrich_existing_lots.py # ~2.3 hrs
python fetch_missing_bid_history.py # ~15 min
```
2. **This Week:**
- Implement Bargain Hunter Dashboard
- Add opportunity alerts
- Create enhanced lot cards
**This Week:**
1. Implement Bargain Hunter Dashboard
2. Add opportunity alerts
3. Create enhanced lot cards
3. **Next Week:**
- Build analytics dashboards
- Implement price prediction model
- Set up webhook notifications
4. **Future:**
- A/B test alert strategies
- Refine prediction models with historical data
- Add category-specific recommendations
**Next Week:**
1. Build analytics dashboards
2. Implement ML price prediction
3. Set up smart notifications
---
## Conclusion
**80%+ intelligence increase** enables:
- 🎯 Automated bargain detection
- 📊 Predictive price modeling
- ⚡ Real-time opportunity alerts
- 💰 ROI tracking
The scraper now captures **5 critical intelligence fields** that unlock advanced analytics:
| Field | Dashboard Impact |
|-------|------------------|
| followers_count | Popularity tracking, sleeper detection |
| estimated_min_price | Bargain detection, value assessment |
| estimated_max_price | Overvaluation alerts, ROI calculation |
| lot_condition | Quality filtering, restoration opportunities |
| appearance | Visual assessment, detailed condition |
**Combined with fixed data quality** (99.9% fewer orphaned lots, 100% auction completeness), the dashboard can now provide:
- 🎯 **Opportunity Detection** - Automated bargain hunting
- 📊 **Predictive Analytics** - ML-based price predictions
- 📈 **Category Intelligence** - Deep market insights
- ⚡ **Real-Time Alerts** - Instant opportunity notifications
- 💰 **ROI Tracking** - Measure investment potential
**Estimated intelligence value increase: 80%+**
Ready to build! 🚀
**Run migrations to activate all features.**