341 lines
14 KiB
Markdown
341 lines
14 KiB
Markdown
# Qwen-Image-Edit Rapid-AIO v23 — System Documentation
|
||
|
||
## Overview
|
||
|
||
A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the **Qwen 2.5 VL GGUF** model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.
|
||
|
||
```
|
||
Browser (car.html SPA)
|
||
│ HTTP / fetch
|
||
▼
|
||
FastAPI :8500 (edit_api.py)
|
||
├─── ComfyUI :8188 ← Qwen GGUF model
|
||
├─── SAM2 / rembg ← background removal
|
||
├─── insightface ← faceswap + face extraction
|
||
├─── GFPGAN ← face restoration
|
||
└─── PostgreSQL ← metadata (192.168.1.160:5433)
|
||
│
|
||
File system (/mnt/zim/tour-comfy/output/, wireframe/)
|
||
```
|
||
|
||
---
|
||
|
||
## Hardware
|
||
|
||
| Component | Spec |
|
||
|-----------|------|
|
||
| GPU | NVIDIA RTX A6000 (48 GB VRAM) |
|
||
| CUDA | 12.4 |
|
||
| PyTorch | 2.6.0+cu124 |
|
||
| ComfyUI | latest pinned (a6000-comfy/) |
|
||
|
||
---
|
||
|
||
## Services
|
||
|
||
### 1. ComfyUI Backend — `:8188`
|
||
|
||
- **Script**: `a6000-comfy/run_comfyui.sh`
|
||
- **Systemd**: `comfyui-backend`
|
||
- **Purpose**: Executes the Qwen GGUF model via `workflow_qwen_edit.json`
|
||
|
||
**Loaded models**
|
||
|
||
| Role | File |
|
||
|------|------|
|
||
| GGUF LLM | `Qwen-Rapid-NSFW-v23_Q8_0.gguf` |
|
||
| Text encoder | `qwen_2.5_vl_7b_fp8_scaled.safetensors` |
|
||
| VAE | `qwen_image_vae.safetensors` |
|
||
|
||
**Workflow nodes** (`workflow_qwen_edit.json`):
|
||
- `NODE_POSITIVE` — positive text prompt
|
||
- `NODE_NEGATIVE` — negative prompt
|
||
- `NODE_LATENT` — latent dimensions (width × height)
|
||
- `NODE_KSAMPLER` — sampler params (Euler/karras, seed, steps, CFG)
|
||
- `image1` — reference image (base64 uploaded to ComfyUI `/upload/image`)
|
||
- `image2` — optional wireframe pose guide (extracted from wireframe video)
|
||
|
||
---
|
||
|
||
### 2. FastAPI API — `:8500`
|
||
|
||
- **Script**: `tour-comfy/edit_api.py`
|
||
- **Systemd**: `comfyui-api`
|
||
- **DB**: PostgreSQL at `192.168.1.160:5433`, database `dv`, table `person`
|
||
|
||
#### Core Endpoints
|
||
|
||
| Method | Path | Purpose |
|
||
|--------|------|---------|
|
||
| `GET` | `/images` | List all images with DB metadata |
|
||
| `GET` | `/images/{filename}` | Serve image file |
|
||
| `POST` | `/upload` | Upload image; optionally add to existing group (`group_id` + `skip_poses`) |
|
||
| `POST` | `/batch` | Queue multi-prompt generation job (async) |
|
||
| `GET` | `/jobs/{id}` | Poll batch job status / per-item progress |
|
||
| `POST` | `/images/{filename}/set-preferred` | Move to group slot 0; triggers face extraction background task |
|
||
| `POST` | `/images/{filename}/extract-face` | Extract & save face crop via insightface (background task) |
|
||
| `POST` | `/images/{filename}/archive` | Soft-archive (hidden from default view) |
|
||
| `POST` | `/images/{filename}/unarchive` | Restore archived image |
|
||
| `POST` | `/images/{filename}/set-hidden` | Toggle studio visibility |
|
||
| `DELETE` | `/images/{filename}` | Delete image + DB record |
|
||
| `POST` | `/images/{filename}/crop` | Crop to pixel rect; `as_copy:true` crops a referenced copy instead of in-place |
|
||
| `POST` | `/images/{filename}/autocrop` | Auto-trim transparent border |
|
||
| `POST` | `/images/{filename}/rotate` | Rotate clockwise in 90° steps (lossless transpose, in-place) |
|
||
| `POST` | `/images/{filename}/invert-alpha` | Invert the alpha channel (fixes wrong-segment BG removal) |
|
||
| `POST` | `/images/{filename}/duplicate` | Copy into same group with `source_refs=[original]` |
|
||
| `POST` | `/faceswap` | insightface video faceswap |
|
||
| `POST` | `/remove-background/{filename}` | rembg BG removal, overwrites in-place; sets `has_background=false` + refreshes static data |
|
||
| `POST` | `/remove-background-sam/{filename}` | SAM2 BG removal → `.nobg.png` sidecar |
|
||
| `GET` | `/wireframe/frame/{video_name}` | Extract frame at `?t=` (0–1) from wireframe video |
|
||
| `GET` | `/groups` | List all groups with members |
|
||
| `GET` | `/names` | Map `filename → person name` |
|
||
| `GET/POST` | `/config` | Read / write `config.json` |
|
||
| `GET` | `/poses` | Load pose library from `poses.md` |
|
||
| `POST` | `/poses` | Create / update / rename a pose (`old_name` to rename) → rewrites `poses.md` |
|
||
| `DELETE` | `/poses/{name}` | Delete a pose from `poses.md` |
|
||
| `GET` | `/pose/check` | Report pose-estimator availability + backend |
|
||
| `POST` | `/images/{filename}/pose` | Estimate COCO-17 body keypoints (caches descriptor for similar-search) |
|
||
| `GET` | `/pose/similar/{filename}` | Rank library images by pose similarity to this image |
|
||
| `POST` | `/pose/similar` | Rank library images by similarity to a supplied (edited) skeleton |
|
||
| `POST` | `/pose/index` | Build pose descriptors for the whole library (daemon thread) |
|
||
| `GET` | `/pose/index/status` | Poll pose-index build progress |
|
||
|
||
---
|
||
|
||
## Generation Pipeline (`_run_pipeline`)
|
||
|
||
```
|
||
prompt
|
||
│
|
||
├─ "transparent background" keywords detected?
|
||
│ YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
|
||
│ set negative: "deformed anatomy, watermark, logo"
|
||
│
|
||
├─ Upload reference image(s) to ComfyUI /upload/image
|
||
│ image1 = source image
|
||
│ image2 = wireframe pose guide frame (optional)
|
||
│
|
||
├─ Patch workflow graph nodes (prompt, size, seed, sampler)
|
||
├─ POST to ComfyUI /prompt → poll /history until done
|
||
├─ Fetch PNG from ComfyUI /view
|
||
│
|
||
└─ is_transparent?
|
||
YES → _apply_transparency_sam2(png_bytes)
|
||
├─ SAM2 point-prompt segmentation
|
||
│ FG: 12 points (hair→shoes, lateral spread for ¾ poses)
|
||
│ BG: 7 points (corners + edge midpoints)
|
||
│ Coverage check: 5%–92% (else fallback)
|
||
│ Gaussian blur radius=1 for soft edges
|
||
└─ fallback: _apply_transparency (rembg U2Net)
|
||
```
|
||
|
||
---
|
||
|
||
## Batch Worker (`_batch_worker`)
|
||
|
||
Runs in a background thread pool. One thread per batch job item.
|
||
|
||
```
|
||
For each (filename, prompt) pair:
|
||
1. Load source image from output dir
|
||
2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
|
||
3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
|
||
4. Save PNG to output dir (timestamped filename)
|
||
5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
|
||
6. Update job progress dict (polled via /jobs/{id})
|
||
```
|
||
|
||
---
|
||
|
||
## Camera Angle Presets
|
||
|
||
Eight absolute + four relative angle prompts defined as `CAMERA_ANGLES` in car.html. Each triggers a `/batch` request with `pose=null` and the angle prompt injected.
|
||
|
||
| Name | Direction |
|
||
|------|-----------|
|
||
| Front | Straight-on frontal |
|
||
| ¾ Left / ¾ Right | 45° rotation |
|
||
| Side L / Side R | 90° profile |
|
||
| Back | Rear view |
|
||
| High / Low | Bird's-eye / worm's-eye |
|
||
| ↺ 45° / ↻ 45° | Relative +45° left / right |
|
||
| ↺ 90° / ↻ 90° | Relative +90° left / right |
|
||
|
||
---
|
||
|
||
## Background Removal
|
||
|
||
Two strategies are available:
|
||
|
||
### Configurable BG removal — `config.json` key `"bg_removal"`
|
||
|
||
Set to `"rembg"` (default) or `"sam2"`.
|
||
|
||
#### `"rembg"` mode
|
||
- Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
|
||
- `_apply_transparency` (rembg U2Net) post-processes the result
|
||
- Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)
|
||
|
||
#### `"sam2"` mode
|
||
- Replaces transparent keywords with `"black background"` in the Qwen prompt → Qwen renders the person against a pure black background
|
||
- Negative prompt discourages any real-background content
|
||
- `_apply_transparency_sam2` (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
|
||
- Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings
|
||
|
||
#### Explicit `"black background"` in prompt
|
||
- Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
|
||
- Prompt is passed to Qwen unchanged
|
||
|
||
#### SAM2 implementation (`_apply_transparency_sam2`)
|
||
- Model: `sam2.1_hiera_base_plus.pt` at `~/.sam/sam2.1_hiera_base_plus.pt`
|
||
- Bbox `[1%, 1%, 99%, 99%]`, `multimask_output=True`, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
|
||
- Also called by the explicit `/remove-background-sam/{filename}` endpoint
|
||
|
||
---
|
||
|
||
## Face Processing (insightface)
|
||
|
||
Model: `buffalo_l`
|
||
|
||
**Faceswap** (`/faceswap`):
|
||
- Reads source face from reference image
|
||
- Swaps into each frame of a target video via OpenCV
|
||
- Optionally runs GFPGAN face restoration post-swap
|
||
|
||
**Face extraction** (`_extract_face_bg` background task):
|
||
- Triggered on `set-preferred` or explicit `extract-face` endpoint
|
||
- Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
|
||
- Saves as `{group_id}_face.png` in output dir
|
||
- DB-registered for display in the studio Info tab face-book thumbnail
|
||
|
||
---
|
||
|
||
## Database Schema (`person` table)
|
||
|
||
| Column | Type | Purpose |
|
||
|--------|------|---------|
|
||
| filename | TEXT PK | Unique image filename |
|
||
| filepath | TEXT | Absolute file path |
|
||
| group_id | TEXT | Group membership |
|
||
| group_name | TEXT | Human-readable group label |
|
||
| sort_order | INTEGER | Position within group (0 = preferred) |
|
||
| name | TEXT | Person / character name |
|
||
| prompt | TEXT | Generation prompt used |
|
||
| pose | TEXT | Pose tag |
|
||
| hidden | BOOLEAN | Hidden from studio preview |
|
||
| archived | BOOLEAN | Soft-deleted |
|
||
| has_background | BOOLEAN | Whether BG was removed |
|
||
| has_clothing | BOOLEAN | Content classification flag |
|
||
| source_refs | TEXT (JSON) | Original filenames this was derived from |
|
||
| content_type | TEXT | `image` or `video` |
|
||
| faceswap_source_video | TEXT | Source video for faceswapped clips |
|
||
|
||
---
|
||
|
||
## Frontend (car.html)
|
||
|
||
Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.
|
||
|
||
**Modes**:
|
||
- **Gallery** — grid of all groups; click to open Studio
|
||
- **Studio** — filmstrip + large viewer + sidebar tabs
|
||
|
||
**Sidebar tabs**:
|
||
|
||
| Tab | Purpose |
|
||
|-----|---------|
|
||
| Generate | Camera angle picker, pose selector (with inline edit/add/delete → `poses.md`), wireframe guide, auto-growing prompt override with history autocomplete, fine-tune textareas, multi-ref thumbnails (click to deselect), batch submit |
|
||
| Info | Metadata, preferred toggle, face-book thumbnail, hide/archive/delete, manual crop (copy or in-place), rotate ±90°, No-BG / Invert-α, duplicate, Pose preview (draggable + similar-pose search) |
|
||
| Faceswap | Source face selector, target video, preview quality toggle |
|
||
| Segment | SAM2 and rembg BG removal buttons |
|
||
| Scenery | Scenery video reference for generation |
|
||
|
||
**Key state variables**:
|
||
- `lbCurrentGid` — active group ID
|
||
- `lbIdx` — current filmstrip position
|
||
- `_followLatestGid` — auto-follow newly generated image
|
||
- `_sbSelectedAngles` — selected camera angle names
|
||
- `_sbWireframeRef` / `_sbWireframeTime` — wireframe video + frame time
|
||
- `privacyMode` — privacy overlay toggle (key: `P`)
|
||
|
||
---
|
||
|
||
## Wireframe Pose Guide
|
||
|
||
Wireframe videos stored in `/mnt/zim/tour-comfy/wireframe/`.
|
||
The studio Generate tab shows a video picker + time scrubber. On generation:
|
||
1. Backend calls `GET /wireframe/frame/{name}?t=0.5` to extract the chosen frame
|
||
2. Frame passed as `image2` in the ComfyUI workflow (second visual reference)
|
||
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via `image2` slot
|
||
|
||
---
|
||
|
||
## Pose Tools (2D body pose)
|
||
|
||
A posenet-style toolset built on a feature-detected estimator. Preferred backend is
|
||
**rtmlib** (RTMPose, ONNX via the already-installed `onnxruntime`); **mediapipe** is a
|
||
fallback. If neither is installed the endpoints return `501` and the UI shows an install hint.
|
||
|
||
- **Install (A6000 venv):** `~/comfyui/venv/bin/pip install rtmlib`. Models (~150 MB) auto-
|
||
download to `~/.cache/rtmlib` on first use.
|
||
- **CUDA libs:** `onnxruntime-gpu` needs the venv's `nvidia-*-cu12` libs on the loader path.
|
||
`a6000-comfy/start_api.sh` adds them to `LD_LIBRARY_PATH` at launch (torch finds them via
|
||
RPATH, onnxruntime does not).
|
||
|
||
**Output format:** COCO-17 keypoints (`POSE_KEYPOINT_NAMES`) + `POSE_SKELETON` edge list,
|
||
returned in image pixels with per-joint score.
|
||
|
||
**Studio overlay (`car.html`):** the **Pose** button overlays the skeleton on the viewer
|
||
(letterbox-aware canvas). Joints are **draggable** to refine/explore a pose. Overlay toolbar:
|
||
*Reset* (re-detect) and *Similar pose* (rank the library by the current — possibly edited —
|
||
skeleton). Results render as a thumbnail strip; clicking one opens that image's group.
|
||
|
||
**Similarity model (`edit_api.py`):**
|
||
- `_pose_descriptor(keypoints)` → translation/scale-invariant vector: centered on hip midpoint
|
||
(fallback shoulders), scaled by torso length, 17×(x,y) + visibility mask. `None` if < 6 joints.
|
||
- `_pose_distance(a, b)` → weighted L2 over jointly-visible joints, taking the min of the direct
|
||
and the **left-right-mirrored** comparison so mirrored poses match.
|
||
- **Pose index:** descriptors are cached in `<output>/_data/poses_index.json`. It fills
|
||
incrementally whenever `/images/{f}/pose` runs, or in bulk via `/pose/index` (daemon thread,
|
||
batched writes every 50, progress via `/pose/index/status`).
|
||
|
||
## File Layout
|
||
|
||
```
|
||
qwen-image-edit-rapid-aio-nsfw-v23/
|
||
├── tour-comfy/
|
||
│ ├── car.html # frontend SPA
|
||
│ ├── edit_api.py # FastAPI backend
|
||
│ ├── database.py # PostgreSQL helpers
|
||
│ ├── watcher.py # output dir auto-processor
|
||
│ ├── config.json # output_dir, comfy_url, etc.
|
||
│ └── workflow_qwen_edit.json # ComfyUI graph template
|
||
├── a6000-comfy/ # ComfyUI install (symlinked to ~/comfyui)
|
||
├── AGENTS.md # this file
|
||
├── architecture2.svg # system architecture diagram
|
||
├── backlog.md # feature backlog
|
||
└── requirements.txt # Python deps
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration (`config.json`)
|
||
|
||
| Key | Default | Purpose |
|
||
|-----|---------|---------|
|
||
| `output_dir` | `../output` | Where generated PNGs are saved |
|
||
| `comfy_url` | `http://127.0.0.1:8188` | ComfyUI base URL |
|
||
| `host` | `0.0.0.0` | FastAPI bind host |
|
||
| `port` | `8500` | FastAPI bind port |
|
||
| `max_area` | `1048576` | Max pixel budget (1 MP) |
|
||
|
||
---
|
||
|
||
## Background Services
|
||
|
||
| Service | Script | Purpose |
|
||
|---------|--------|---------|
|
||
| `comfyui-backend` | `a6000-comfy/run_comfyui.sh` | Qwen GGUF inference |
|
||
| `comfyui-api` | `tour-comfy/start_api.sh` | FastAPI HTTP server |
|
||
| `watcher` | `tour-comfy/start_watcher.sh` | Monitors output dir, auto-processes dropped images |
|