141 lines
4.7 KiB
Markdown
141 lines
4.7 KiB
Markdown
# Scaev Scraper Refactoring Summary
|
|
|
|
## Date: 2025-12-07
|
|
|
|
## Objectives Completed
|
|
|
|
### 1. Image Download Integration ✅
|
|
- **Changed**: Enabled `DOWNLOAD_IMAGES = True` in `config.py` and `docker-compose.yml`
|
|
- **Added**: Unique constraint on `images(lot_id, url)` to prevent duplicates
|
|
- **Added**: Automatic duplicate cleanup migration in `cache.py`
|
|
- **Result**: Images are now downloaded to `/mnt/okcomputer/output/images/{lot_id}/` and marked as `downloaded=1`
|
|
- **Impact**: Eliminates 57M+ duplicate image downloads by monitor app
|
|
|
|
### 2. Data Completeness Fix ✅
|
|
- **Problem**: 99.9% of lots missing closing_time, 100% missing bid data
|
|
- **Root Cause**: Troostwijk loads bid/timing data dynamically via GraphQL API, not in HTML
|
|
- **Solution**: Added GraphQL client to fetch real-time bidding data
|
|
|
|
## Key Changes
|
|
|
|
### New Files
|
|
1. **src/graphql_client.py** - GraphQL API client for fetching lot bidding data
|
|
- Endpoint: `https://storefront.tbauctions.com/storefront/graphql`
|
|
- Fetches: current_bid, starting_bid, minimum_bid, bid_count, closing_time
|
|
|
|
### Modified Files
|
|
1. **src/config.py:22** - `DOWNLOAD_IMAGES = True`
|
|
2. **docker-compose.yml:13** - `DOWNLOAD_IMAGES: "True"`
|
|
3. **src/cache.py**
|
|
- Added unique index on `images(lot_id, url)`
|
|
- Added columns `starting_bid`, `minimum_bid` to `lots` table
|
|
- Added migration to clean duplicates and add missing columns
|
|
4. **src/scraper.py**
|
|
- Integrated GraphQL API calls for each lot
|
|
- Fetches real-time bidding data after parsing HTML
|
|
- Removed unicode characters causing Windows encoding issues
|
|
|
|
## Database Schema Updates
|
|
|
|
### lots table - New Columns
|
|
```sql
|
|
ALTER TABLE lots ADD COLUMN starting_bid TEXT;
|
|
ALTER TABLE lots ADD COLUMN minimum_bid TEXT;
|
|
```
|
|
|
|
### images table - New Index
|
|
```sql
|
|
CREATE UNIQUE INDEX idx_unique_lot_url ON images(lot_id, url);
|
|
```
|
|
|
|
## Data Flow (New Architecture)
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────┐
|
|
│ Phase 3: Scrape Lot Page │
|
|
└────────────────────────────────────────────────────┘
|
|
│
|
|
├─▶ Parse HTML (__NEXT_DATA__)
|
|
│ └─▶ Extract: title, location, images, description
|
|
│
|
|
├─▶ Fetch GraphQL API
|
|
│ └─▶ Query: LotBiddingData(lot_display_id)
|
|
│ └─▶ Returns:
|
|
│ - currentBidAmount (cents)
|
|
│ - initialAmount (starting_bid)
|
|
│ - nextMinimalBid (minimum_bid)
|
|
│ - bidsCount
|
|
│ - endDate (Unix timestamp)
|
|
│ - startDate
|
|
│ - biddingStatus
|
|
│
|
|
└─▶ Save to Database
|
|
- lots table: complete bid & timing data
|
|
- images table: deduplicated URLs
|
|
- Download images immediately
|
|
```
|
|
|
|
## Testing Results
|
|
|
|
### Test Lot: A1-28505-5
|
|
```
|
|
Current Bid: EUR 50.00 ✅
|
|
Starting Bid: EUR 50.00 ✅
|
|
Minimum Bid: EUR 55.00 ✅
|
|
Bid Count: 1 ✅
|
|
Closing Time: 2025-12-16 19:10:00 ✅
|
|
Images: Downloaded 2 ✅
|
|
```
|
|
|
|
## Deployment Checklist
|
|
|
|
- [x] Enable DOWNLOAD_IMAGES in config
|
|
- [x] Update docker-compose environment
|
|
- [x] Add GraphQL client
|
|
- [x] Update scraper integration
|
|
- [x] Add database migrations
|
|
- [x] Test with live lot
|
|
- [ ] Deploy to production
|
|
- [ ] Run full scrape to populate data
|
|
- [ ] Verify monitor app sees downloaded images
|
|
|
|
## Post-Deployment Verification
|
|
|
|
### Check Data Quality
|
|
```sql
|
|
-- Bid data completeness
|
|
SELECT
|
|
COUNT(*) as total,
|
|
SUM(CASE WHEN closing_time != '' THEN 1 ELSE 0 END) as has_closing,
|
|
SUM(CASE WHEN bid_count > 0 THEN 1 ELSE 0 END) as has_bids,
|
|
SUM(CASE WHEN starting_bid IS NOT NULL THEN 1 ELSE 0 END) as has_starting_bid
|
|
FROM lots
|
|
WHERE scraped_at > datetime('now', '-1 hour');
|
|
|
|
-- Image download rate
|
|
SELECT
|
|
COUNT(*) as total,
|
|
SUM(downloaded) as downloaded,
|
|
ROUND(100.0 * SUM(downloaded) / COUNT(*), 2) as success_rate
|
|
FROM images
|
|
WHERE id IN (
|
|
SELECT i.id FROM images i
|
|
JOIN lots l ON i.lot_id = l.lot_id
|
|
WHERE l.scraped_at > datetime('now', '-1 hour')
|
|
);
|
|
|
|
-- Duplicate check (should be 0)
|
|
SELECT lot_id, url, COUNT(*) as dup_count
|
|
FROM images
|
|
GROUP BY lot_id, url
|
|
HAVING COUNT(*) > 1;
|
|
```
|
|
|
|
## Notes
|
|
|
|
- GraphQL API requires no authentication
|
|
- API rate limits: handled by existing `RATE_LIMIT_SECONDS = 0.5`
|
|
- Currency format: Changed from € to EUR for Windows compatibility
|
|
- Timestamps: API returns Unix timestamps in seconds (not milliseconds)
|
|
- Existing data: Old lots still have missing data; re-scrape required to populate
|