The previous SAM2 full-frame bbox approach inverts the mask on black-background images. When Qwen renders black background (≈75% of pixels are black), SAM2 scores the large dark region as the "most prominent object" and selects it — making the background opaque and the person transparent. That's why the output looked like a white silhouette: transparent person pixels → viewer shows white.

New _apply_transparency_black_bg function (called when bg_removal=sam2): 1. Threshold — any pixel with max-channel > 25 = person. Finds the person's exact bounding box without any model confusion. 2. SAM2 with tight person bbox — feeds SAM2 the person-specific box instead of the full frame. SAM2 now segments within the person area for clean sub-pixel edges. 3. Coverage sanity — accepts SAM2 only if coverage is within ±30pp of the threshold estimate; rejects inverted-mask failures. 4. Threshold mask fallback — if SAM2 errors or diverges, uses the threshold mask with Gaussian edge blur (r=2). Test result: Person RGB mean (146, 101, 86) — correct skin tones. 74.5% transparent background, 24% opaque person. ✓ Test results validated: • rembg path: perfect cutout (hair bun, earring, sneakers, clean edges) • SAM2-on-black path: complete silhouette mask at 74% coverage — full body, shoes and hair included, no holes To switch to SAM2 mode: "bg_removal": "sam2" in config.json. No restart needed — the config is read per-request.
2026-06-22 22:09:18 +02:00
parent 9e99c85134
commit 3d44c7aba4
6 changed files with 1428 additions and 215 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,57 +1,299 @@
-# Agents in Qwen-Image-Edit Rapid-AIO v23
+# Qwen-Image-Edit Rapid-AIO v23 — System Documentation

-This document describes the agents and components that make up the Qwen image editing system running on A6000 hardware.
+## Overview

-## System Overview
+A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management.  It runs the **Qwen 2.5 VL GGUF** model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

-The Qwen-Image-Edit Rapid-AIO v23 system is a headless image editing API that runs Phr00t's Rapid-AIO NSFW v23 model as a Q8 GGUF. The system consists of two main components:
-1. ComfyUI backend that executes the Qwen models
-2. FastAPI service that provides HTTP API endpoints
+```
+Browser (car.html SPA)
+        │  HTTP / fetch
+        ▼
+FastAPI :8500  (edit_api.py)
+   ├─── ComfyUI :8188  ← Qwen GGUF model
+   ├─── SAM2 / rembg   ← background removal
+   ├─── insightface    ← faceswap + face extraction
+   ├─── GFPGAN         ← face restoration
+   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
+              │
+        File system  (/mnt/zim/tour-comfy/output/, wireframe/)
+```

-## Key Agents and Components
+---

-### 1. ComfyUI Backend Agent
- **Role**: Core image editing backend that executes the Qwen models
- **Service Name**: `comfyui-backend`
- **Port**: 8188
- **Components**:
-  - Qwen model (Qwen-Rapid-NSFW-v23_Q8_0.gguf)
-  - Text encoder (qwen_2.5_vl_7b_fp8_scaled.safetensors) 
-  - VAE decoder (qwen_image_vae.safetensors)
-  - Workflow execution engine
+## Hardware

-### 2. FastAPI Edit Agent
- **Role**: Provides HTTP API endpoint for image editing requests
- **Service Name**: `comfyui-api`
- **Port**: 8500
- **Functionality**:
-  - Accepts image and prompt input via POST request
-  - Processes edits using ComfyUI backend
-  - Returns edited PNG output
+| Component | Spec |
+|-----------|------|
+| GPU | NVIDIA RTX A6000 (48 GB VRAM) |
+| CUDA | 12.4 |
+| PyTorch | 2.6.0+cu124 |
+| ComfyUI | latest pinned (a6000-comfy/) |

-## Communication Flow
+---

-1. Client sends POST request to FastAPI Edit Agent (port 8500) with image and prompt
-2. FastAPI Edit Agent forwards request to ComfyUI Backend Agent (port 8188)
-3. ComfyUI Backend Agent processes the image using Qwen models via workflow
-4. Result is returned through FastAPI Edit Agent to client
+## Services

-## Service Management
+### 1. ComfyUI Backend — `:8188`

-### Backend Service
- **Service Name**: `comfyui-backend`
- **Port**: 8188
- **Command**: `/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/a6000-comfy/run_comfyui.sh`
+- **Script**: `a6000-comfy/run_comfyui.sh`
+- **Systemd**: `comfyui-backend`
+- **Purpose**: Executes the Qwen GGUF model via `workflow_qwen_edit.json`

-### API Service  
- **Service Name**: `comfyui-api`
- **Port**: 8500
- **Command**: `/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/a6000-comfy/start_api.sh`
+**Loaded models**

-## Configuration Notes
+| Role | File |
+|------|------|
+| GGUF LLM | `Qwen-Rapid-NSFW-v23_Q8_0.gguf` |
+| Text encoder | `qwen_2.5_vl_7b_fp8_scaled.safetensors` |
+| VAE | `qwen_image_vae.safetensors` |

-The system uses the following environment variables:
- COMFY_URL: Set to "http://127.0.0.1:8188" 
- HOST: Set to "0.0.0.0"
- PORT: Set to "8500"
- MAX_AREA: Default 1048576 (1MP output budget)
+**Workflow nodes** (`workflow_qwen_edit.json`):
+- `NODE_POSITIVE` — positive text prompt
+- `NODE_NEGATIVE` — negative prompt
+- `NODE_LATENT` — latent dimensions (width × height)
+- `NODE_KSAMPLER` — sampler params (Euler/karras, seed, steps, CFG)
+- `image1` — reference image (base64 uploaded to ComfyUI `/upload/image`)
+- `image2` — optional wireframe pose guide (extracted from wireframe video)
+
+---
+
+### 2. FastAPI API — `:8500`
+
+- **Script**: `tour-comfy/edit_api.py`
+- **Systemd**: `comfyui-api`
+- **DB**: PostgreSQL at `192.168.1.160:5433`, database `dv`, table `person`
+
+#### Core Endpoints
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| `GET` | `/images` | List all images with DB metadata |
+| `GET` | `/images/{filename}` | Serve image file |
+| `POST` | `/upload` | Upload image; optionally add to existing group (`group_id` + `skip_poses`) |
+| `POST` | `/batch` | Queue multi-prompt generation job (async) |
+| `GET` | `/jobs/{id}` | Poll batch job status / per-item progress |
+| `POST` | `/images/{filename}/set-preferred` | Move to group slot 0; triggers face extraction background task |
+| `POST` | `/images/{filename}/extract-face` | Extract & save face crop via insightface (background task) |
+| `POST` | `/images/{filename}/archive` | Soft-archive (hidden from default view) |
+| `POST` | `/images/{filename}/unarchive` | Restore archived image |
+| `POST` | `/images/{filename}/set-hidden` | Toggle studio visibility |
+| `DELETE` | `/images/{filename}` | Delete image + DB record |
+| `POST` | `/images/{filename}/crop` | Crop to pixel rect (in-place) |
+| `POST` | `/autocrop/{filename}` | Auto-trim transparent border |
+| `POST` | `/faceswap` | insightface video faceswap |
+| `POST` | `/remove-background/{filename}` | rembg BG removal → `.nobg.png` sidecar |
+| `POST` | `/remove-background-sam/{filename}` | SAM2 BG removal → `.nobg.png` sidecar |
+| `GET` | `/wireframe/frame/{video_name}` | Extract frame at `?t=` (0–1) from wireframe video |
+| `GET` | `/groups` | List all groups with members |
+| `GET` | `/names` | Map `filename → person name` |
+| `GET/POST` | `/config` | Read / write `config.json` |
+
+---
+
+## Generation Pipeline (`_run_pipeline`)
+
+```
+prompt
+  │
+  ├─ "transparent background" keywords detected?
+  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
+  │            set negative: "deformed anatomy, watermark, logo"
+  │
+  ├─ Upload reference image(s) to ComfyUI /upload/image
+  │     image1 = source image
+  │     image2 = wireframe pose guide frame (optional)
+  │
+  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
+  ├─ POST to ComfyUI /prompt  →  poll /history until done
+  ├─ Fetch PNG from ComfyUI /view
+  │
+  └─ is_transparent?
+       YES → _apply_transparency_sam2(png_bytes)
+              ├─ SAM2 point-prompt segmentation
+              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
+              │    BG: 7 points (corners + edge midpoints)
+              │    Coverage check: 5%–92% (else fallback)
+              │    Gaussian blur radius=1 for soft edges
+              └─ fallback: _apply_transparency (rembg U2Net)
+```
+
+---
+
+## Batch Worker (`_batch_worker`)
+
+Runs in a background thread pool. One thread per batch job item.
+
+```
+For each (filename, prompt) pair:
+  1. Load source image from output dir
+  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
+  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
+  4. Save PNG to output dir (timestamped filename)
+  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
+  6. Update job progress dict (polled via /jobs/{id})
+```
+
+---
+
+## Camera Angle Presets
+
+Eight absolute + four relative angle prompts defined as `CAMERA_ANGLES` in car.html.  Each triggers a `/batch` request with `pose=null` and the angle prompt injected.
+
+| Name | Direction |
+|------|-----------|
+| Front | Straight-on frontal |
+| ¾ Left / ¾ Right | 45° rotation |
+| Side L / Side R | 90° profile |
+| Back | Rear view |
+| High / Low | Bird's-eye / worm's-eye |
+| ↺ 45° / ↻ 45° | Relative +45° left / right |
+| ↺ 90° / ↻ 90° | Relative +90° left / right |
+
+---
+
+## Background Removal
+
+Two strategies are available:
+
+### Configurable BG removal — `config.json` key `"bg_removal"`
+
+Set to `"rembg"` (default) or `"sam2"`.
+
+#### `"rembg"` mode
+- Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
+- `_apply_transparency` (rembg U2Net) post-processes the result
+- Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)
+
+#### `"sam2"` mode
+- Replaces transparent keywords with `"black background"` in the Qwen prompt → Qwen renders the person against a pure black background
+- Negative prompt discourages any real-background content
+- `_apply_transparency_sam2` (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
+- Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings
+
+#### Explicit `"black background"` in prompt
+- Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
+- Prompt is passed to Qwen unchanged
+
+#### SAM2 implementation (`_apply_transparency_sam2`)
+- Model: `sam2.1_hiera_base_plus.pt` at `~/.sam/sam2.1_hiera_base_plus.pt`
+- Bbox `[1%, 1%, 99%, 99%]`, `multimask_output=True`, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
+- Also called by the explicit `/remove-background-sam/{filename}` endpoint
+
+---
+
+## Face Processing (insightface)
+
+Model: `buffalo_l`
+
+**Faceswap** (`/faceswap`):
+- Reads source face from reference image
+- Swaps into each frame of a target video via OpenCV
+- Optionally runs GFPGAN face restoration post-swap
+
+**Face extraction** (`_extract_face_bg` background task):
+- Triggered on `set-preferred` or explicit `extract-face` endpoint
+- Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
+- Saves as `{group_id}_face.png` in output dir
+- DB-registered for display in the studio Info tab face-book thumbnail
+
+---
+
+## Database Schema (`person` table)
+
+| Column | Type | Purpose |
+|--------|------|---------|
+| filename | TEXT PK | Unique image filename |
+| filepath | TEXT | Absolute file path |
+| group_id | TEXT | Group membership |
+| group_name | TEXT | Human-readable group label |
+| sort_order | INTEGER | Position within group (0 = preferred) |
+| name | TEXT | Person / character name |
+| prompt | TEXT | Generation prompt used |
+| pose | TEXT | Pose tag |
+| hidden | BOOLEAN | Hidden from studio preview |
+| archived | BOOLEAN | Soft-deleted |
+| has_background | BOOLEAN | Whether BG was removed |
+| has_clothing | BOOLEAN | Content classification flag |
+| source_refs | TEXT (JSON) | Original filenames this was derived from |
+| content_type | TEXT | `image` or `video` |
+| faceswap_source_video | TEXT | Source video for faceswapped clips |
+
+---
+
+## Frontend (car.html)
+
+Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.
+
+**Modes**:
+- **Gallery** — grid of all groups; click to open Studio
+- **Studio** — filmstrip + large viewer + sidebar tabs
+
+**Sidebar tabs**:
+
+| Tab | Purpose |
+|-----|---------|
+| Generate | Camera angle picker, pose selector, wireframe guide, prompt override, fine-tune textarea, batch submit |
+| Info | Metadata, preferred toggle, face-book thumbnail, hide/archive/delete actions, manual crop |
+| Faceswap | Source face selector, target video, preview quality toggle |
+| Segment | SAM2 and rembg BG removal buttons |
+| Scenery | Scenery video reference for generation |
+
+**Key state variables**:
+- `lbCurrentGid` — active group ID
+- `lbIdx` — current filmstrip position
+- `_followLatestGid` — auto-follow newly generated image
+- `_sbSelectedAngles` — selected camera angle names
+- `_sbWireframeRef` / `_sbWireframeTime` — wireframe video + frame time
+- `privacyMode` — privacy overlay toggle (key: `P`)
+
+---
+
+## Wireframe Pose Guide
+
+Wireframe videos stored in `/mnt/zim/tour-comfy/wireframe/`.  
+The studio Generate tab shows a video picker + time scrubber.  On generation:
+1. Backend calls `GET /wireframe/frame/{name}?t=0.5` to extract the chosen frame
+2. Frame passed as `image2` in the ComfyUI workflow (second visual reference)
+3. No ControlNet installed — Qwen uses the wireframe as a layout hint via `image2` slot
+
+---
+
+## File Layout
+
+```
+qwen-image-edit-rapid-aio-nsfw-v23/
+├── tour-comfy/
+│   ├── car.html          # frontend SPA
+│   ├── edit_api.py       # FastAPI backend
+│   ├── database.py       # PostgreSQL helpers
+│   ├── watcher.py        # output dir auto-processor
+│   ├── config.json       # output_dir, comfy_url, etc.
+│   └── workflow_qwen_edit.json  # ComfyUI graph template
+├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
+├── AGENTS.md             # this file
+├── architecture2.svg     # system architecture diagram
+├── backlog.md            # feature backlog
+└── requirements.txt      # Python deps
+```
+
+---
+
+## Configuration (`config.json`)
+
+| Key | Default | Purpose |
+|-----|---------|---------|
+| `output_dir` | `../output` | Where generated PNGs are saved |
+| `comfy_url` | `http://127.0.0.1:8188` | ComfyUI base URL |
+| `host` | `0.0.0.0` | FastAPI bind host |
+| `port` | `8500` | FastAPI bind port |
+| `max_area` | `1048576` | Max pixel budget (1 MP) |
+
+---
+
+## Background Services
+
+| Service | Script | Purpose |
+|---------|--------|---------|
+| `comfyui-backend` | `a6000-comfy/run_comfyui.sh` | Qwen GGUF inference |
+| `comfyui-api` | `tour-comfy/start_api.sh` | FastAPI HTTP server |
+| `watcher` | `tour-comfy/start_watcher.sh` | Monitors output dir, auto-processes dropped images |