init
This commit is contained in:
378
docs/Home.md
Normal file
378
docs/Home.md
Normal file
@@ -0,0 +1,378 @@
|
||||
# Diagram Nex Wiki
|
||||
|
||||
Welcome to the **Diagram Nex** project documentation.
|
||||
|
||||
This project provides a static web interface for viewing and managing network architecture diagrams generated with Python's `diagrams` library.
|
||||
|
||||
## 📚 Contents
|
||||
|
||||
- [Getting Started](#getting-started)
|
||||
- [Development Guide](#development-guide)
|
||||
- [Deployment Guide](#deployment-guide)
|
||||
- [Architecture Overview](#architecture-overview)
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- Graphviz
|
||||
- Docker (for deployment)
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone git@git.appmodel.nl:Tour/nex.git
|
||||
cd nex
|
||||
|
||||
# Create virtual environment
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Generate diagrams
|
||||
python lan_architecture.py
|
||||
python main.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Guide
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
nex/
|
||||
├── Dockerfile # Multi-stage build for production
|
||||
├── docker-compose.yml # Docker Compose configuration
|
||||
├── requirements.txt # Python dependencies
|
||||
├── lan_architecture.py # LAN architecture diagram generator
|
||||
├── main.py # Main diagram generator
|
||||
├── public/
|
||||
│ └── index.html # Live Mermaid diagram editor
|
||||
└── wiki/ # Documentation (this wiki)
|
||||
```
|
||||
|
||||
### Adding New Diagrams
|
||||
|
||||
1. Create a new Python file (e.g., `network_diagram.py`)
|
||||
2. Import required components from `diagrams` library
|
||||
3. Define your architecture using context managers
|
||||
4. Run the script to generate PNG output
|
||||
|
||||
Example:
|
||||
```python
|
||||
from diagrams import Diagram, Cluster
|
||||
from diagrams.onprem.network import Router
|
||||
from diagrams.generic.device import Mobile
|
||||
|
||||
with Diagram("My Network", show=False, filename="my_network"):
|
||||
router = Router("Gateway")
|
||||
device = Mobile("Phone")
|
||||
device >> router
|
||||
```
|
||||
|
||||
### Testing Locally
|
||||
|
||||
```bash
|
||||
# Generate diagrams
|
||||
python lan_architecture.py
|
||||
|
||||
# View output
|
||||
ls -la *.png
|
||||
|
||||
# Test with local web server
|
||||
python -m http.server 8000 --directory public/
|
||||
# Open: http://localhost:8000
|
||||
```
|
||||
|
||||
### Available Icons
|
||||
|
||||
The `diagrams` library provides icons from multiple providers:
|
||||
- **OnPrem**: Network, compute, storage, etc.
|
||||
- **Cloud**: AWS, Azure, GCP services
|
||||
- **Generic**: Network devices, blank nodes
|
||||
- **Custom**: Use your own images
|
||||
|
||||
See: [Diagrams Documentation](https://diagrams.mingrammer.com/docs/nodes/overview)
|
||||
|
||||
---
|
||||
|
||||
## Deployment Guide
|
||||
|
||||
### Overview
|
||||
|
||||
The project uses a fully automated Docker-based deployment pipeline:
|
||||
|
||||
1. **Develop** on local machine
|
||||
2. **Commit & Push** to Gitea
|
||||
3. **Auto-Deploy** via post-receive hook
|
||||
4. **Build** multi-stage Docker image
|
||||
5. **Serve** via Traefik reverse proxy
|
||||
|
||||
### Deployment Steps
|
||||
|
||||
#### 1. Initial Setup
|
||||
|
||||
```bash
|
||||
# On server: Create app structure
|
||||
apps-create nex static-fe
|
||||
|
||||
# This creates:
|
||||
# - /opt/apps/nex (git repository)
|
||||
# - /home/tour/infra/nex/docker-compose.yml
|
||||
```
|
||||
|
||||
#### 2. Configure Gitea Hook
|
||||
|
||||
In repository **Settings → Git Hooks → post-receive**:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
/usr/local/bin/app-deploy nex
|
||||
```
|
||||
|
||||
#### 3. Deploy
|
||||
|
||||
Simply push to Gitea:
|
||||
|
||||
```bash
|
||||
git push origin main
|
||||
```
|
||||
|
||||
The server automatically:
|
||||
- Pulls latest code
|
||||
- Rebuilds Docker image
|
||||
- Restarts container
|
||||
- Updates live site at https://nex.appmodel.nl
|
||||
|
||||
### Manual Deployment
|
||||
|
||||
If needed:
|
||||
|
||||
```bash
|
||||
# On server
|
||||
app-deploy nex
|
||||
|
||||
# Or step-by-step
|
||||
cd /opt/apps/nex
|
||||
git pull
|
||||
cd /home/tour/infra/nex
|
||||
docker compose up -d --build nex
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
```bash
|
||||
# View deployment logs
|
||||
tail -f /var/log/app-deploy-nex.log
|
||||
|
||||
# View container logs
|
||||
cd /home/tour/infra/nex
|
||||
docker compose logs -f nex
|
||||
|
||||
# Check container status
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Infrastructure
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Production Architecture │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Gitea │ (Source of Truth) │
|
||||
│ │ Tour/nex │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ git push │
|
||||
│ ↓ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Post-Receive │ (Auto-trigger) │
|
||||
│ │ Hook │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ app-deploy nex │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ /opt/apps/nex/ │ │
|
||||
│ │ (Git Repository) │ │
|
||||
│ └──────┬───────────────┘ │
|
||||
│ │ git pull │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ Docker Build Process │ │
|
||||
│ │ ┌─────────────────┐ │ │
|
||||
│ │ │ Stage 1: Build │ │ │
|
||||
│ │ │ - Python │ │ │
|
||||
│ │ │ - Graphviz │ │ │
|
||||
│ │ │ - Generate PNGs │ │ │
|
||||
│ │ └─────────────────┘ │ │
|
||||
│ │ ┌─────────────────┐ │ │
|
||||
│ │ │ Stage 2: Serve │ │ │
|
||||
│ │ │ - Nginx │ │ │
|
||||
│ │ │ - Static files │ │ │
|
||||
│ │ └─────────────────┘ │ │
|
||||
│ └──────┬───────────────┘ │
|
||||
│ │ container starts │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ diagram-nex │ │
|
||||
│ │ Container :80 │ │
|
||||
│ └──────┬───────────────┘ │
|
||||
│ │ traefik_net │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ Traefik │ │
|
||||
│ │ Reverse Proxy │ │
|
||||
│ │ + Let's Encrypt │ │
|
||||
│ └──────┬───────────────┘ │
|
||||
│ │ HTTPS │
|
||||
│ ↓ │
|
||||
│ nex.appmodel.nl │
|
||||
│ │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Network Topology
|
||||
|
||||
The diagram nex documents our home lab infrastructure:
|
||||
|
||||
- **LAN 192.168.1.0/24**: Main network
|
||||
- Core server (192.168.1.159): Traefik, Gitea, Dokku, Auction stack, MI50/Ollama
|
||||
- DNS server (192.168.1.163): AdGuard, Artifactory, CI runner
|
||||
- Home Assistant (192.168.1.193)
|
||||
- Atlas worker (192.168.1.100)
|
||||
- IoT devices
|
||||
|
||||
- **Tether 192.168.137.0/24**: Isolated subnet
|
||||
- Hermes worker (192.168.137.239)
|
||||
- Plato worker (192.168.137.163)
|
||||
|
||||
### Services
|
||||
|
||||
| Service | URL | Description |
|
||||
|------------------|-----------------------------|------------------|
|
||||
| Diagram nex | https://nex.appmodel.nl | This application |
|
||||
| Gitea | https://git.appmodel.nl | Git hosting |
|
||||
| Auction Frontend | https://auction.appmodel.nl | Auction platform |
|
||||
| Aupi API | https://aupi.appmodel.nl | Backend API |
|
||||
| Dokku | https://dokku.lan | PaaS platform |
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
### Related Documentation
|
||||
|
||||
- [DEPLOY_SERVER_SETUP.md](../DEPLOY_SERVER_SETUP.md) - Detailed deployment guide
|
||||
- [DEPLOYMENT_GUIDE.md](../DEPLOYMENT_GUIDE.md) - General deployment pipeline
|
||||
- [README.md](../README.md) - Project overview
|
||||
|
||||
### External Links
|
||||
|
||||
- [Diagrams Library](https://diagrams.mingrammer.com/)
|
||||
- [Graphviz](https://graphviz.org/)
|
||||
- [Mermaid.js](https://mermaid.js.org/) (used in live editor)
|
||||
- [Traefik Documentation](https://doc.traefik.io/traefik/)
|
||||
- [Docker Compose](https://docs.docker.com/compose/)
|
||||
|
||||
### Contributing
|
||||
|
||||
To contribute to this project:
|
||||
|
||||
1. Create a feature branch
|
||||
2. Make your changes
|
||||
3. Test locally
|
||||
4. Submit a pull request in Gitea
|
||||
5. Wait for automatic deployment after merge
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Diagrams Not Generating
|
||||
|
||||
**Problem**: Python script fails to generate PNG files
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check Graphviz installation
|
||||
dot -V
|
||||
|
||||
# Reinstall if needed
|
||||
# macOS: brew install graphviz
|
||||
# Ubuntu: sudo apt-get install graphviz
|
||||
# Windows: choco install graphviz
|
||||
|
||||
# Verify Python dependencies
|
||||
pip list | grep diagrams
|
||||
pip install --upgrade diagrams
|
||||
```
|
||||
|
||||
#### Container Build Fails
|
||||
|
||||
**Problem**: Docker build fails during deployment
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check deployment logs
|
||||
tail -f /var/log/app-deploy-nex.log
|
||||
|
||||
# Rebuild manually with verbose output
|
||||
cd /home/tour/infra/nex
|
||||
docker compose build --no-cache --progress=plain nex
|
||||
```
|
||||
|
||||
#### Site Not Accessible
|
||||
|
||||
**Problem**: Container runs but site not reachable
|
||||
|
||||
**Solution**:
|
||||
1. Check DNS: `nslookup nex.appmodel.nl`
|
||||
2. Verify Traefik labels: `docker inspect nex-nex | grep traefik`
|
||||
3. Check Traefik logs: `docker logs traefik`
|
||||
4. Test container directly: `curl http://localhost:PORT`
|
||||
|
||||
#### Changes Not Reflected
|
||||
|
||||
**Problem**: Pushed changes but site unchanged
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Force rebuild on server
|
||||
cd /home/tour/infra/nex
|
||||
docker compose down
|
||||
docker compose up -d --build --force-recreate nex
|
||||
|
||||
# Clear browser cache
|
||||
# Ctrl+Shift+R (Windows/Linux) or Cmd+Shift+R (macOS)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contact & Support
|
||||
|
||||
For issues, questions, or suggestions:
|
||||
|
||||
1. Check this wiki
|
||||
2. Review [DEPLOY_SERVER_SETUP.md](../DEPLOY_SERVER_SETUP.md)
|
||||
3. Check container logs
|
||||
4. Contact the infrastructure team
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-12-02
|
||||
**Version**: 1.0.0
|
||||
**Maintainer**: Tour
|
||||
31
docs/build-pipeline.md
Normal file
31
docs/build-pipeline.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Appmodel Home Lab – Build & Deploy Pipeline
|
||||
|
||||
Dit document beschrijft hoe een nieuwe applicatie in het home lab wordt aangemaakt en hoe de build- & deploy-pipeline werkt.
|
||||
|
||||
## Overzicht
|
||||
|
||||
- **Broncode**: Gitea (`https://git.appmodel.nl`)
|
||||
- **Build & runtime**: Docker containers op netwerk `traefik_net`
|
||||
- **Routing & TLS**: Traefik (`https://traefik.appmodel.nl`)
|
||||
- **Automatische deploy**: Gitea `post-receive` hooks → `app-deploy <app>`
|
||||
|
||||
## Pipeline in één diagram
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Dev[💻 Dev machine\nVS Code / Git] -->|git push| Gitea[📚 Gitea\nTour/<app>]
|
||||
|
||||
subgraph Server[🏠 Home lab server\n192.168.1.159]
|
||||
Gitea --> Hook[🔔 post-receive hook\n/app-deploy <app>]
|
||||
|
||||
Hook --> Deploy[⚙️ app-deploy <app>\n/git pull + docker compose up -d --build <app>]
|
||||
|
||||
subgraph Docker[🐳 Docker / traefik_net]
|
||||
AppC[🧱 App container\n<app>.appmodel.nl]
|
||||
Traefik[🚦 Traefik\nReverse Proxy]
|
||||
end
|
||||
end
|
||||
|
||||
Deploy --> AppC
|
||||
Traefik --> AppC
|
||||
Client[🌐 Browser / API client] -->|https://<app>.appmodel.nl| Traefik
|
||||
85
docs/deployment.md
Normal file
85
docs/deployment.md
Normal file
@@ -0,0 +1,85 @@
|
||||
```mermaid
|
||||
flowchart LR
|
||||
|
||||
%% ============ Internet ============
|
||||
subgraph WAN[🌐 Internet / Cloud]
|
||||
extDNS[(📡 Public DNS)]
|
||||
extGit[(☁️ Externe registries / Git)]
|
||||
end
|
||||
|
||||
%% ============ LAN 192.168.1.x ============
|
||||
subgraph LAN[🏠 LAN 192.168.1.0/24]
|
||||
hub[🛜 Router / Gateway\nhub.lan\n192.168.1.1]
|
||||
|
||||
subgraph core[💻 Hoofdserver / Desktop\nTour / hephaestus / ollama / dokku.lan\n192.168.1.159]
|
||||
traefik[🚦 Traefik\nReverse Proxy]
|
||||
gitea[📚 Gitea\n git.appmodel.nl]
|
||||
dokku[🐳 Dokku\nPaaS / build]
|
||||
auctionFE[🧱 Auction Frontend\n auction.appmodel.nl]
|
||||
aupiAPI[🧱 Auction Backend API\n aupi.appmodel.nl]
|
||||
mi50[🧠 MI50 / Ollama\nAI workloads]
|
||||
end
|
||||
|
||||
subgraph infraDNS[🧭 Infra & DNS\nodroid / dns.lan\n192.168.1.163]
|
||||
adguard[🧭 AdGuard Home\nDNS / *.lan / *.appmodel.nl]
|
||||
artifactory[📦 Artifactory]
|
||||
runner[⚙️ Build runners]
|
||||
end
|
||||
|
||||
subgraph ha[🏡 Home Automation\nha.lan\n192.168.1.193]
|
||||
hass[🏠 Home Assistant]
|
||||
end
|
||||
|
||||
atlas[🧱 atlas.lan\n192.168.1.100\n]
|
||||
|
||||
iot1[📺 hof-E402NA\n192.168.1.214]
|
||||
iot2[🎧 S380HB\n192.168.1.59]
|
||||
iot3[📟 ecb5faa56c90\n192.168.1.49]
|
||||
iot4[❓ Unknown\n192.168.1.240]
|
||||
end
|
||||
|
||||
%% ============ Tether subnet ============
|
||||
subgraph TETHER[📶 Tether subnet 192.168.137.0/24]
|
||||
hermes[🛰️ hermes.lan\n192.168.137.239\nworker / node]
|
||||
plato[🛰️ plato.lan\n192.168.137.163\nworker / node]
|
||||
end
|
||||
|
||||
%% ============ Verkeer ============
|
||||
|
||||
%% Basis LAN connecties
|
||||
hub --- core
|
||||
hub --- infraDNS
|
||||
hub --- ha
|
||||
hub --- atlas
|
||||
hub --- iot1
|
||||
hub --- iot2
|
||||
hub --- iot3
|
||||
hub --- iot4
|
||||
|
||||
%% WAN koppeling
|
||||
hub --> WAN
|
||||
infraDNS --> WAN
|
||||
|
||||
%% DNS-resolutie
|
||||
core --> adguard
|
||||
ha --> adguard
|
||||
atlas --> adguard
|
||||
TETHER --> adguard
|
||||
|
||||
%% Websites / reverse proxy
|
||||
extDNS --> traefik
|
||||
traefik --> gitea
|
||||
traefik --> auctionFE
|
||||
traefik --> aupiAPI
|
||||
traefik --> dokku
|
||||
|
||||
%% App flow
|
||||
auctionFE --> aupiAPI
|
||||
aupiAPI --> adguard
|
||||
|
||||
%% AI workloads
|
||||
core --> mi50
|
||||
|
||||
%% Tether workers
|
||||
core --- TETHER
|
||||
```
|
||||
94
docs/integration.md
Normal file
94
docs/integration.md
Normal file
@@ -0,0 +1,94 @@
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph P1["PHASE 1: EXTERNAL SCRAPER (Python/Playwright)"]
|
||||
direction LR
|
||||
A1[Listing Pages<br/>/auctions?page=N] --> A2[Extract URLs]
|
||||
B1[Auction Pages<br/>/a/auction-id] --> B2[Parse __NEXT_DATA__ JSON]
|
||||
C1[Lot Pages<br/>/l/lot-id] --> C2[Parse __NEXT_DATA__ JSON]
|
||||
|
||||
A2 --> D1[INSERT auctions to SQLite]
|
||||
B2 --> D1
|
||||
C2 --> D2[INSERT lots & image URLs]
|
||||
|
||||
D1 --> DB[(SQLite Database<br/>output/cache.db)]
|
||||
D2 --> DB
|
||||
end
|
||||
|
||||
DB --> P2_Entry
|
||||
|
||||
subgraph P2["PHASE 2: MONITORING & PROCESSING (Java)"]
|
||||
direction TB
|
||||
P2_Entry[Data Ready] --> Monitor[TroostwijkMonitor<br/>Read lots every hour]
|
||||
P2_Entry --> DBService[DatabaseService<br/>Query & Import]
|
||||
P2_Entry --> Adapter[ScraperDataAdapter<br/>Transform TEXT → INTEGER]
|
||||
|
||||
Monitor --> BM[Bid Monitoring<br/>Check API every 1h]
|
||||
DBService --> IP[Image Processing<br/>Download & Analyze]
|
||||
Adapter --> DataForNotify[Formatted Data]
|
||||
|
||||
BM --> BidUpdate{New bid?}
|
||||
BidUpdate -->|Yes| UpdateDB[Update current_bid in DB]
|
||||
UpdateDB --> NotifyTrigger1
|
||||
|
||||
IP --> Detection[Object Detection<br/>YOLO/OpenCV DNN]
|
||||
Detection --> ObjectCheck{Detect objects?}
|
||||
ObjectCheck -->|Vehicle| Save1[Save labels & estimate value]
|
||||
ObjectCheck -->|Furniture| Save2[Save labels & estimate value]
|
||||
ObjectCheck -->|Machinery| Save3[Save labels & estimate value]
|
||||
Save1 --> NotifyTrigger2
|
||||
Save2 --> NotifyTrigger2
|
||||
Save3 --> NotifyTrigger2
|
||||
|
||||
CA[Closing Alerts<br/>Check < 5 min] --> TimeCheck{Time critical?}
|
||||
TimeCheck -->|Yes| NotifyTrigger3
|
||||
end
|
||||
|
||||
NotifyTrigger1 --> NS
|
||||
NotifyTrigger2 --> NS
|
||||
NotifyTrigger3 --> NS
|
||||
|
||||
subgraph P3["PHASE 3: NOTIFICATION SYSTEM"]
|
||||
NS[NotificationService] --> DN[Desktop Notify<br/>Windows/macOS/Linux]
|
||||
NS --> EN[Email Notify<br/>Gmail SMTP]
|
||||
NS --> PL[Set Priority Level<br/>0=Normal, 1=High]
|
||||
end
|
||||
|
||||
DN --> UI[User Interaction & Decisions]
|
||||
EN --> UI
|
||||
PL --> UI
|
||||
|
||||
subgraph UI_Details[User Decision Points / Trigger Events]
|
||||
direction LR
|
||||
E1["1. BID CHANGE<br/>'Nieuw bod op kavel 12345...'<br/>Actions: Place bid? Monitor? Ignore?"]
|
||||
E2["2. OBJECT DETECTED<br/>'Lot contains: Vehicle...'<br/>Actions: Review? Confirm value?"]
|
||||
E3["3. CLOSING ALERT<br/>'Kavel 12345 sluit binnen 5 min.'<br/>Actions: Place final bid? Let expire?"]
|
||||
E4["4. VIEWING DAY QUESTIONS<br/>'Bezichtiging op [date]...'"]
|
||||
E5["5. ITEM RECOGNITION CONFIRMATION<br/>'Detected: [object]...'"]
|
||||
E6["6. VALUE ESTIMATE APPROVAL<br/>'Geschatte waarde: €X...'"]
|
||||
E7["7. EXCEPTION HANDLING<br/>'Afwijkende sluitingstijd...'"]
|
||||
end
|
||||
|
||||
UI --> UI_Details
|
||||
|
||||
%% Object Detection Sub-Flow Detail
|
||||
subgraph P2_Detail["Object Detection & Value Estimation Pipeline"]
|
||||
direction LR
|
||||
DI[Downloaded Image] --> IPS[ImageProcessingService]
|
||||
IPS --> ODS[ObjectDetectionService]
|
||||
ODS --> Load[Load YOLO model]
|
||||
ODS --> Run[Run inference]
|
||||
ODS --> Post[Post-process detections<br/>confidence > 0.5]
|
||||
Post --> ObjList["Detected Objects List<br/>(80 COCO classes)"]
|
||||
ObjList --> VEL[Value Estimation Logic<br/>Future enhancement]
|
||||
VEL --> Match[Match to categories]
|
||||
VEL --> History[Historical price analysis]
|
||||
VEL --> Condition[Condition assessment]
|
||||
VEL --> Market[Market trends]
|
||||
Market --> ValueEst["Estimated Value Range<br/>Confidence: 75%"]
|
||||
ValueEst --> SaveToDB[Save to Database]
|
||||
SaveToDB --> TriggerNotify{Value > threshold?}
|
||||
end
|
||||
|
||||
IP -.-> P2_Detail
|
||||
TriggerNotify -.-> NotifyTrigger2
|
||||
```
|
||||
Reference in New Issue
Block a user