This commit is contained in:
Tour
2025-12-08 09:08:43 +01:00
commit 067abd1280
30 changed files with 5348 additions and 0 deletions

378
docs/Home.md Normal file
View File

@@ -0,0 +1,378 @@
# Diagram Nex Wiki
Welcome to the **Diagram Nex** project documentation.
This project provides a static web interface for viewing and managing network architecture diagrams generated with Python's `diagrams` library.
## 📚 Contents
- [Getting Started](#getting-started)
- [Development Guide](#development-guide)
- [Deployment Guide](#deployment-guide)
- [Architecture Overview](#architecture-overview)
---
## Getting Started
### Prerequisites
- Python 3.8+
- Graphviz
- Docker (for deployment)
### Quick Start
```bash
# Clone repository
git clone git@git.appmodel.nl:Tour/nex.git
cd nex
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Generate diagrams
python lan_architecture.py
python main.py
```
---
## Development Guide
### Project Structure
```
nex/
├── Dockerfile # Multi-stage build for production
├── docker-compose.yml # Docker Compose configuration
├── requirements.txt # Python dependencies
├── lan_architecture.py # LAN architecture diagram generator
├── main.py # Main diagram generator
├── public/
│ └── index.html # Live Mermaid diagram editor
└── wiki/ # Documentation (this wiki)
```
### Adding New Diagrams
1. Create a new Python file (e.g., `network_diagram.py`)
2. Import required components from `diagrams` library
3. Define your architecture using context managers
4. Run the script to generate PNG output
Example:
```python
from diagrams import Diagram, Cluster
from diagrams.onprem.network import Router
from diagrams.generic.device import Mobile
with Diagram("My Network", show=False, filename="my_network"):
router = Router("Gateway")
device = Mobile("Phone")
device >> router
```
### Testing Locally
```bash
# Generate diagrams
python lan_architecture.py
# View output
ls -la *.png
# Test with local web server
python -m http.server 8000 --directory public/
# Open: http://localhost:8000
```
### Available Icons
The `diagrams` library provides icons from multiple providers:
- **OnPrem**: Network, compute, storage, etc.
- **Cloud**: AWS, Azure, GCP services
- **Generic**: Network devices, blank nodes
- **Custom**: Use your own images
See: [Diagrams Documentation](https://diagrams.mingrammer.com/docs/nodes/overview)
---
## Deployment Guide
### Overview
The project uses a fully automated Docker-based deployment pipeline:
1. **Develop** on local machine
2. **Commit & Push** to Gitea
3. **Auto-Deploy** via post-receive hook
4. **Build** multi-stage Docker image
5. **Serve** via Traefik reverse proxy
### Deployment Steps
#### 1. Initial Setup
```bash
# On server: Create app structure
apps-create nex static-fe
# This creates:
# - /opt/apps/nex (git repository)
# - /home/tour/infra/nex/docker-compose.yml
```
#### 2. Configure Gitea Hook
In repository **Settings → Git Hooks → post-receive**:
```bash
#!/usr/bin/env bash
/usr/local/bin/app-deploy nex
```
#### 3. Deploy
Simply push to Gitea:
```bash
git push origin main
```
The server automatically:
- Pulls latest code
- Rebuilds Docker image
- Restarts container
- Updates live site at https://nex.appmodel.nl
### Manual Deployment
If needed:
```bash
# On server
app-deploy nex
# Or step-by-step
cd /opt/apps/nex
git pull
cd /home/tour/infra/nex
docker compose up -d --build nex
```
### Monitoring
```bash
# View deployment logs
tail -f /var/log/app-deploy-nex.log
# View container logs
cd /home/tour/infra/nex
docker compose logs -f nex
# Check container status
docker compose ps
```
---
## Architecture Overview
### Infrastructure
```
┌─────────────────────────────────────────────┐
│ Production Architecture │
├─────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ │
│ │ Gitea │ (Source of Truth) │
│ │ Tour/nex │ │
│ └──────┬───────┘ │
│ │ git push │
│ ↓ │
│ ┌──────────────┐ │
│ │ Post-Receive │ (Auto-trigger) │
│ │ Hook │ │
│ └──────┬───────┘ │
│ │ app-deploy nex │
│ ↓ │
│ ┌──────────────────────┐ │
│ │ /opt/apps/nex/ │ │
│ │ (Git Repository) │ │
│ └──────┬───────────────┘ │
│ │ git pull │
│ ↓ │
│ ┌──────────────────────┐ │
│ │ Docker Build Process │ │
│ │ ┌─────────────────┐ │ │
│ │ │ Stage 1: Build │ │ │
│ │ │ - Python │ │ │
│ │ │ - Graphviz │ │ │
│ │ │ - Generate PNGs │ │ │
│ │ └─────────────────┘ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ Stage 2: Serve │ │ │
│ │ │ - Nginx │ │ │
│ │ │ - Static files │ │ │
│ │ └─────────────────┘ │ │
│ └──────┬───────────────┘ │
│ │ container starts │
│ ↓ │
│ ┌──────────────────────┐ │
│ │ diagram-nex │ │
│ │ Container :80 │ │
│ └──────┬───────────────┘ │
│ │ traefik_net │
│ ↓ │
│ ┌──────────────────────┐ │
│ │ Traefik │ │
│ │ Reverse Proxy │ │
│ │ + Let's Encrypt │ │
│ └──────┬───────────────┘ │
│ │ HTTPS │
│ ↓ │
│ nex.appmodel.nl │
│ │
└─────────────────────────────────────────────┘
```
### Network Topology
The diagram nex documents our home lab infrastructure:
- **LAN 192.168.1.0/24**: Main network
- Core server (192.168.1.159): Traefik, Gitea, Dokku, Auction stack, MI50/Ollama
- DNS server (192.168.1.163): AdGuard, Artifactory, CI runner
- Home Assistant (192.168.1.193)
- Atlas worker (192.168.1.100)
- IoT devices
- **Tether 192.168.137.0/24**: Isolated subnet
- Hermes worker (192.168.137.239)
- Plato worker (192.168.137.163)
### Services
| Service | URL | Description |
|------------------|-----------------------------|------------------|
| Diagram nex | https://nex.appmodel.nl | This application |
| Gitea | https://git.appmodel.nl | Git hosting |
| Auction Frontend | https://auction.appmodel.nl | Auction platform |
| Aupi API | https://aupi.appmodel.nl | Backend API |
| Dokku | https://dokku.lan | PaaS platform |
---
## Additional Resources
### Related Documentation
- [DEPLOY_SERVER_SETUP.md](../DEPLOY_SERVER_SETUP.md) - Detailed deployment guide
- [DEPLOYMENT_GUIDE.md](../DEPLOYMENT_GUIDE.md) - General deployment pipeline
- [README.md](../README.md) - Project overview
### External Links
- [Diagrams Library](https://diagrams.mingrammer.com/)
- [Graphviz](https://graphviz.org/)
- [Mermaid.js](https://mermaid.js.org/) (used in live editor)
- [Traefik Documentation](https://doc.traefik.io/traefik/)
- [Docker Compose](https://docs.docker.com/compose/)
### Contributing
To contribute to this project:
1. Create a feature branch
2. Make your changes
3. Test locally
4. Submit a pull request in Gitea
5. Wait for automatic deployment after merge
---
## Troubleshooting
### Common Issues
#### Diagrams Not Generating
**Problem**: Python script fails to generate PNG files
**Solution**:
```bash
# Check Graphviz installation
dot -V
# Reinstall if needed
# macOS: brew install graphviz
# Ubuntu: sudo apt-get install graphviz
# Windows: choco install graphviz
# Verify Python dependencies
pip list | grep diagrams
pip install --upgrade diagrams
```
#### Container Build Fails
**Problem**: Docker build fails during deployment
**Solution**:
```bash
# Check deployment logs
tail -f /var/log/app-deploy-nex.log
# Rebuild manually with verbose output
cd /home/tour/infra/nex
docker compose build --no-cache --progress=plain nex
```
#### Site Not Accessible
**Problem**: Container runs but site not reachable
**Solution**:
1. Check DNS: `nslookup nex.appmodel.nl`
2. Verify Traefik labels: `docker inspect nex-nex | grep traefik`
3. Check Traefik logs: `docker logs traefik`
4. Test container directly: `curl http://localhost:PORT`
#### Changes Not Reflected
**Problem**: Pushed changes but site unchanged
**Solution**:
```bash
# Force rebuild on server
cd /home/tour/infra/nex
docker compose down
docker compose up -d --build --force-recreate nex
# Clear browser cache
# Ctrl+Shift+R (Windows/Linux) or Cmd+Shift+R (macOS)
```
---
## Contact & Support
For issues, questions, or suggestions:
1. Check this wiki
2. Review [DEPLOY_SERVER_SETUP.md](../DEPLOY_SERVER_SETUP.md)
3. Check container logs
4. Contact the infrastructure team
---
**Last Updated**: 2025-12-02
**Version**: 1.0.0
**Maintainer**: Tour

31
docs/build-pipeline.md Normal file
View File

@@ -0,0 +1,31 @@
# Appmodel Home Lab Build & Deploy Pipeline
Dit document beschrijft hoe een nieuwe applicatie in het home lab wordt aangemaakt en hoe de build- & deploy-pipeline werkt.
## Overzicht
- **Broncode**: Gitea (`https://git.appmodel.nl`)
- **Build & runtime**: Docker containers op netwerk `traefik_net`
- **Routing & TLS**: Traefik (`https://traefik.appmodel.nl`)
- **Automatische deploy**: Gitea `post-receive` hooks → `app-deploy <app>`
## Pipeline in één diagram
```mermaid
flowchart LR
Dev[💻 Dev machine\nVS Code / Git] -->|git push| Gitea[📚 Gitea\nTour/<app>]
subgraph Server[🏠 Home lab server\n192.168.1.159]
Gitea --> Hook[🔔 post-receive hook\n/app-deploy <app>]
Hook --> Deploy[⚙️ app-deploy <app>\n/git pull + docker compose up -d --build <app>]
subgraph Docker[🐳 Docker / traefik_net]
AppC[🧱 App container\n<app>.appmodel.nl]
Traefik[🚦 Traefik\nReverse Proxy]
end
end
Deploy --> AppC
Traefik --> AppC
Client[🌐 Browser / API client] -->|https://<app>.appmodel.nl| Traefik

85
docs/deployment.md Normal file
View File

@@ -0,0 +1,85 @@
```mermaid
flowchart LR
%% ============ Internet ============
subgraph WAN[🌐 Internet / Cloud]
extDNS[(📡 Public DNS)]
extGit[(☁️ Externe registries / Git)]
end
%% ============ LAN 192.168.1.x ============
subgraph LAN[🏠 LAN 192.168.1.0/24]
hub[🛜 Router / Gateway\nhub.lan\n192.168.1.1]
subgraph core[💻 Hoofdserver / Desktop\nTour / hephaestus / ollama / dokku.lan\n192.168.1.159]
traefik[🚦 Traefik\nReverse Proxy]
gitea[📚 Gitea\n git.appmodel.nl]
dokku[🐳 Dokku\nPaaS / build]
auctionFE[🧱 Auction Frontend\n auction.appmodel.nl]
aupiAPI[🧱 Auction Backend API\n aupi.appmodel.nl]
mi50[🧠 MI50 / Ollama\nAI workloads]
end
subgraph infraDNS[🧭 Infra & DNS\nodroid / dns.lan\n192.168.1.163]
adguard[🧭 AdGuard Home\nDNS / *.lan / *.appmodel.nl]
artifactory[📦 Artifactory]
runner[⚙️ Build runners]
end
subgraph ha[🏡 Home Automation\nha.lan\n192.168.1.193]
hass[🏠 Home Assistant]
end
atlas[🧱 atlas.lan\n192.168.1.100\n]
iot1[📺 hof-E402NA\n192.168.1.214]
iot2[🎧 S380HB\n192.168.1.59]
iot3[📟 ecb5faa56c90\n192.168.1.49]
iot4[❓ Unknown\n192.168.1.240]
end
%% ============ Tether subnet ============
subgraph TETHER[📶 Tether subnet 192.168.137.0/24]
hermes[🛰️ hermes.lan\n192.168.137.239\nworker / node]
plato[🛰️ plato.lan\n192.168.137.163\nworker / node]
end
%% ============ Verkeer ============
%% Basis LAN connecties
hub --- core
hub --- infraDNS
hub --- ha
hub --- atlas
hub --- iot1
hub --- iot2
hub --- iot3
hub --- iot4
%% WAN koppeling
hub --> WAN
infraDNS --> WAN
%% DNS-resolutie
core --> adguard
ha --> adguard
atlas --> adguard
TETHER --> adguard
%% Websites / reverse proxy
extDNS --> traefik
traefik --> gitea
traefik --> auctionFE
traefik --> aupiAPI
traefik --> dokku
%% App flow
auctionFE --> aupiAPI
aupiAPI --> adguard
%% AI workloads
core --> mi50
%% Tether workers
core --- TETHER
```

94
docs/integration.md Normal file
View File

@@ -0,0 +1,94 @@
```mermaid
flowchart TD
subgraph P1["PHASE 1: EXTERNAL SCRAPER (Python/Playwright)"]
direction LR
A1[Listing Pages<br/>/auctions?page=N] --> A2[Extract URLs]
B1[Auction Pages<br/>/a/auction-id] --> B2[Parse __NEXT_DATA__ JSON]
C1[Lot Pages<br/>/l/lot-id] --> C2[Parse __NEXT_DATA__ JSON]
A2 --> D1[INSERT auctions to SQLite]
B2 --> D1
C2 --> D2[INSERT lots & image URLs]
D1 --> DB[(SQLite Database<br/>output/cache.db)]
D2 --> DB
end
DB --> P2_Entry
subgraph P2["PHASE 2: MONITORING & PROCESSING (Java)"]
direction TB
P2_Entry[Data Ready] --> Monitor[TroostwijkMonitor<br/>Read lots every hour]
P2_Entry --> DBService[DatabaseService<br/>Query & Import]
P2_Entry --> Adapter[ScraperDataAdapter<br/>Transform TEXT → INTEGER]
Monitor --> BM[Bid Monitoring<br/>Check API every 1h]
DBService --> IP[Image Processing<br/>Download & Analyze]
Adapter --> DataForNotify[Formatted Data]
BM --> BidUpdate{New bid?}
BidUpdate -->|Yes| UpdateDB[Update current_bid in DB]
UpdateDB --> NotifyTrigger1
IP --> Detection[Object Detection<br/>YOLO/OpenCV DNN]
Detection --> ObjectCheck{Detect objects?}
ObjectCheck -->|Vehicle| Save1[Save labels & estimate value]
ObjectCheck -->|Furniture| Save2[Save labels & estimate value]
ObjectCheck -->|Machinery| Save3[Save labels & estimate value]
Save1 --> NotifyTrigger2
Save2 --> NotifyTrigger2
Save3 --> NotifyTrigger2
CA[Closing Alerts<br/>Check < 5 min] --> TimeCheck{Time critical?}
TimeCheck -->|Yes| NotifyTrigger3
end
NotifyTrigger1 --> NS
NotifyTrigger2 --> NS
NotifyTrigger3 --> NS
subgraph P3["PHASE 3: NOTIFICATION SYSTEM"]
NS[NotificationService] --> DN[Desktop Notify<br/>Windows/macOS/Linux]
NS --> EN[Email Notify<br/>Gmail SMTP]
NS --> PL[Set Priority Level<br/>0=Normal, 1=High]
end
DN --> UI[User Interaction & Decisions]
EN --> UI
PL --> UI
subgraph UI_Details[User Decision Points / Trigger Events]
direction LR
E1["1. BID CHANGE<br/>'Nieuw bod op kavel 12345...'<br/>Actions: Place bid? Monitor? Ignore?"]
E2["2. OBJECT DETECTED<br/>'Lot contains: Vehicle...'<br/>Actions: Review? Confirm value?"]
E3["3. CLOSING ALERT<br/>'Kavel 12345 sluit binnen 5 min.'<br/>Actions: Place final bid? Let expire?"]
E4["4. VIEWING DAY QUESTIONS<br/>'Bezichtiging op [date]...'"]
E5["5. ITEM RECOGNITION CONFIRMATION<br/>'Detected: [object]...'"]
E6["6. VALUE ESTIMATE APPROVAL<br/>'Geschatte waarde: €X...'"]
E7["7. EXCEPTION HANDLING<br/>'Afwijkende sluitingstijd...'"]
end
UI --> UI_Details
%% Object Detection Sub-Flow Detail
subgraph P2_Detail["Object Detection & Value Estimation Pipeline"]
direction LR
DI[Downloaded Image] --> IPS[ImageProcessingService]
IPS --> ODS[ObjectDetectionService]
ODS --> Load[Load YOLO model]
ODS --> Run[Run inference]
ODS --> Post[Post-process detections<br/>confidence > 0.5]
Post --> ObjList["Detected Objects List<br/>(80 COCO classes)"]
ObjList --> VEL[Value Estimation Logic<br/>Future enhancement]
VEL --> Match[Match to categories]
VEL --> History[Historical price analysis]
VEL --> Condition[Condition assessment]
VEL --> Market[Market trends]
Market --> ValueEst["Estimated Value Range<br/>Confidence: 75%"]
ValueEst --> SaveToDB[Save to Database]
SaveToDB --> TriggerNotify{Value > threshold?}
end
IP -.-> P2_Detail
TriggerNotify -.-> NotifyTrigger2
```