diff --git a/AGENTS.md b/AGENTS.md index 707fa70..67eb793 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,57 +1,299 @@ -# Agents in Qwen-Image-Edit Rapid-AIO v23 +# Qwen-Image-Edit Rapid-AIO v23 — System Documentation -This document describes the agents and components that make up the Qwen image editing system running on A6000 hardware. +## Overview -## System Overview +A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the **Qwen 2.5 VL GGUF** model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service. -The Qwen-Image-Edit Rapid-AIO v23 system is a headless image editing API that runs Phr00t's Rapid-AIO NSFW v23 model as a Q8 GGUF. The system consists of two main components: -1. ComfyUI backend that executes the Qwen models -2. FastAPI service that provides HTTP API endpoints +``` +Browser (car.html SPA) + │ HTTP / fetch + ▼ +FastAPI :8500 (edit_api.py) + ├─── ComfyUI :8188 ← Qwen GGUF model + ├─── SAM2 / rembg ← background removal + ├─── insightface ← faceswap + face extraction + ├─── GFPGAN ← face restoration + └─── PostgreSQL ← metadata (192.168.1.160:5433) + │ + File system (/mnt/zim/tour-comfy/output/, wireframe/) +``` -## Key Agents and Components +--- -### 1. ComfyUI Backend Agent -- **Role**: Core image editing backend that executes the Qwen models -- **Service Name**: `comfyui-backend` -- **Port**: 8188 -- **Components**: - - Qwen model (Qwen-Rapid-NSFW-v23_Q8_0.gguf) - - Text encoder (qwen_2.5_vl_7b_fp8_scaled.safetensors) - - VAE decoder (qwen_image_vae.safetensors) - - Workflow execution engine +## Hardware -### 2. FastAPI Edit Agent -- **Role**: Provides HTTP API endpoint for image editing requests -- **Service Name**: `comfyui-api` -- **Port**: 8500 -- **Functionality**: - - Accepts image and prompt input via POST request - - Processes edits using ComfyUI backend - - Returns edited PNG output +| Component | Spec | +|-----------|------| +| GPU | NVIDIA RTX A6000 (48 GB VRAM) | +| CUDA | 12.4 | +| PyTorch | 2.6.0+cu124 | +| ComfyUI | latest pinned (a6000-comfy/) | -## Communication Flow +--- -1. Client sends POST request to FastAPI Edit Agent (port 8500) with image and prompt -2. FastAPI Edit Agent forwards request to ComfyUI Backend Agent (port 8188) -3. ComfyUI Backend Agent processes the image using Qwen models via workflow -4. Result is returned through FastAPI Edit Agent to client +## Services -## Service Management +### 1. ComfyUI Backend — `:8188` -### Backend Service -- **Service Name**: `comfyui-backend` -- **Port**: 8188 -- **Command**: `/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/a6000-comfy/run_comfyui.sh` +- **Script**: `a6000-comfy/run_comfyui.sh` +- **Systemd**: `comfyui-backend` +- **Purpose**: Executes the Qwen GGUF model via `workflow_qwen_edit.json` -### API Service -- **Service Name**: `comfyui-api` -- **Port**: 8500 -- **Command**: `/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/a6000-comfy/start_api.sh` +**Loaded models** -## Configuration Notes +| Role | File | +|------|------| +| GGUF LLM | `Qwen-Rapid-NSFW-v23_Q8_0.gguf` | +| Text encoder | `qwen_2.5_vl_7b_fp8_scaled.safetensors` | +| VAE | `qwen_image_vae.safetensors` | -The system uses the following environment variables: -- COMFY_URL: Set to "http://127.0.0.1:8188" -- HOST: Set to "0.0.0.0" -- PORT: Set to "8500" -- MAX_AREA: Default 1048576 (1MP output budget) \ No newline at end of file +**Workflow nodes** (`workflow_qwen_edit.json`): +- `NODE_POSITIVE` — positive text prompt +- `NODE_NEGATIVE` — negative prompt +- `NODE_LATENT` — latent dimensions (width × height) +- `NODE_KSAMPLER` — sampler params (Euler/karras, seed, steps, CFG) +- `image1` — reference image (base64 uploaded to ComfyUI `/upload/image`) +- `image2` — optional wireframe pose guide (extracted from wireframe video) + +--- + +### 2. FastAPI API — `:8500` + +- **Script**: `tour-comfy/edit_api.py` +- **Systemd**: `comfyui-api` +- **DB**: PostgreSQL at `192.168.1.160:5433`, database `dv`, table `person` + +#### Core Endpoints + +| Method | Path | Purpose | +|--------|------|---------| +| `GET` | `/images` | List all images with DB metadata | +| `GET` | `/images/{filename}` | Serve image file | +| `POST` | `/upload` | Upload image; optionally add to existing group (`group_id` + `skip_poses`) | +| `POST` | `/batch` | Queue multi-prompt generation job (async) | +| `GET` | `/jobs/{id}` | Poll batch job status / per-item progress | +| `POST` | `/images/{filename}/set-preferred` | Move to group slot 0; triggers face extraction background task | +| `POST` | `/images/{filename}/extract-face` | Extract & save face crop via insightface (background task) | +| `POST` | `/images/{filename}/archive` | Soft-archive (hidden from default view) | +| `POST` | `/images/{filename}/unarchive` | Restore archived image | +| `POST` | `/images/{filename}/set-hidden` | Toggle studio visibility | +| `DELETE` | `/images/{filename}` | Delete image + DB record | +| `POST` | `/images/{filename}/crop` | Crop to pixel rect (in-place) | +| `POST` | `/autocrop/{filename}` | Auto-trim transparent border | +| `POST` | `/faceswap` | insightface video faceswap | +| `POST` | `/remove-background/{filename}` | rembg BG removal → `.nobg.png` sidecar | +| `POST` | `/remove-background-sam/{filename}` | SAM2 BG removal → `.nobg.png` sidecar | +| `GET` | `/wireframe/frame/{video_name}` | Extract frame at `?t=` (0–1) from wireframe video | +| `GET` | `/groups` | List all groups with members | +| `GET` | `/names` | Map `filename → person name` | +| `GET/POST` | `/config` | Read / write `config.json` | + +--- + +## Generation Pipeline (`_run_pipeline`) + +``` +prompt + │ + ├─ "transparent background" keywords detected? + │ YES → strip keywords from Qwen prompt (keeps `is_transparent` flag) + │ set negative: "deformed anatomy, watermark, logo" + │ + ├─ Upload reference image(s) to ComfyUI /upload/image + │ image1 = source image + │ image2 = wireframe pose guide frame (optional) + │ + ├─ Patch workflow graph nodes (prompt, size, seed, sampler) + ├─ POST to ComfyUI /prompt → poll /history until done + ├─ Fetch PNG from ComfyUI /view + │ + └─ is_transparent? + YES → _apply_transparency_sam2(png_bytes) + ├─ SAM2 point-prompt segmentation + │ FG: 12 points (hair→shoes, lateral spread for ¾ poses) + │ BG: 7 points (corners + edge midpoints) + │ Coverage check: 5%–92% (else fallback) + │ Gaussian blur radius=1 for soft edges + └─ fallback: _apply_transparency (rembg U2Net) +``` + +--- + +## Batch Worker (`_batch_worker`) + +Runs in a background thread pool. One thread per batch job item. + +``` +For each (filename, prompt) pair: + 1. Load source image from output dir + 2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture + 3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame]) + 4. Save PNG to output dir (timestamped filename) + 5. upsert_person(new_filename, group_id, prompt, source_refs=[original]) + 6. Update job progress dict (polled via /jobs/{id}) +``` + +--- + +## Camera Angle Presets + +Eight absolute + four relative angle prompts defined as `CAMERA_ANGLES` in car.html. Each triggers a `/batch` request with `pose=null` and the angle prompt injected. + +| Name | Direction | +|------|-----------| +| Front | Straight-on frontal | +| ¾ Left / ¾ Right | 45° rotation | +| Side L / Side R | 90° profile | +| Back | Rear view | +| High / Low | Bird's-eye / worm's-eye | +| ↺ 45° / ↻ 45° | Relative +45° left / right | +| ↺ 90° / ↻ 90° | Relative +90° left / right | + +--- + +## Background Removal + +Two strategies are available: + +### Configurable BG removal — `config.json` key `"bg_removal"` + +Set to `"rembg"` (default) or `"sam2"`. + +#### `"rembg"` mode +- Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background +- `_apply_transparency` (rembg U2Net) post-processes the result +- Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.) + +#### `"sam2"` mode +- Replaces transparent keywords with `"black background"` in the Qwen prompt → Qwen renders the person against a pure black background +- Negative prompt discourages any real-background content +- `_apply_transparency_sam2` (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation +- Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings + +#### Explicit `"black background"` in prompt +- Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input +- Prompt is passed to Qwen unchanged + +#### SAM2 implementation (`_apply_transparency_sam2`) +- Model: `sam2.1_hiera_base_plus.pt` at `~/.sam/sam2.1_hiera_base_plus.pt` +- Bbox `[1%, 1%, 99%, 99%]`, `multimask_output=True`, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1 +- Also called by the explicit `/remove-background-sam/{filename}` endpoint + +--- + +## Face Processing (insightface) + +Model: `buffalo_l` + +**Faceswap** (`/faceswap`): +- Reads source face from reference image +- Swaps into each frame of a target video via OpenCV +- Optionally runs GFPGAN face restoration post-swap + +**Face extraction** (`_extract_face_bg` background task): +- Triggered on `set-preferred` or explicit `extract-face` endpoint +- Detects largest face via insightface, adds padding (50% sides, 200% top headroom) +- Saves as `{group_id}_face.png` in output dir +- DB-registered for display in the studio Info tab face-book thumbnail + +--- + +## Database Schema (`person` table) + +| Column | Type | Purpose | +|--------|------|---------| +| filename | TEXT PK | Unique image filename | +| filepath | TEXT | Absolute file path | +| group_id | TEXT | Group membership | +| group_name | TEXT | Human-readable group label | +| sort_order | INTEGER | Position within group (0 = preferred) | +| name | TEXT | Person / character name | +| prompt | TEXT | Generation prompt used | +| pose | TEXT | Pose tag | +| hidden | BOOLEAN | Hidden from studio preview | +| archived | BOOLEAN | Soft-deleted | +| has_background | BOOLEAN | Whether BG was removed | +| has_clothing | BOOLEAN | Content classification flag | +| source_refs | TEXT (JSON) | Original filenames this was derived from | +| content_type | TEXT | `image` or `video` | +| faceswap_source_video | TEXT | Source video for faceswapped clips | + +--- + +## Frontend (car.html) + +Single-page application (~5000 lines). No build step — pure HTML/CSS/JS. + +**Modes**: +- **Gallery** — grid of all groups; click to open Studio +- **Studio** — filmstrip + large viewer + sidebar tabs + +**Sidebar tabs**: + +| Tab | Purpose | +|-----|---------| +| Generate | Camera angle picker, pose selector, wireframe guide, prompt override, fine-tune textarea, batch submit | +| Info | Metadata, preferred toggle, face-book thumbnail, hide/archive/delete actions, manual crop | +| Faceswap | Source face selector, target video, preview quality toggle | +| Segment | SAM2 and rembg BG removal buttons | +| Scenery | Scenery video reference for generation | + +**Key state variables**: +- `lbCurrentGid` — active group ID +- `lbIdx` — current filmstrip position +- `_followLatestGid` — auto-follow newly generated image +- `_sbSelectedAngles` — selected camera angle names +- `_sbWireframeRef` / `_sbWireframeTime` — wireframe video + frame time +- `privacyMode` — privacy overlay toggle (key: `P`) + +--- + +## Wireframe Pose Guide + +Wireframe videos stored in `/mnt/zim/tour-comfy/wireframe/`. +The studio Generate tab shows a video picker + time scrubber. On generation: +1. Backend calls `GET /wireframe/frame/{name}?t=0.5` to extract the chosen frame +2. Frame passed as `image2` in the ComfyUI workflow (second visual reference) +3. No ControlNet installed — Qwen uses the wireframe as a layout hint via `image2` slot + +--- + +## File Layout + +``` +qwen-image-edit-rapid-aio-nsfw-v23/ +├── tour-comfy/ +│ ├── car.html # frontend SPA +│ ├── edit_api.py # FastAPI backend +│ ├── database.py # PostgreSQL helpers +│ ├── watcher.py # output dir auto-processor +│ ├── config.json # output_dir, comfy_url, etc. +│ └── workflow_qwen_edit.json # ComfyUI graph template +├── a6000-comfy/ # ComfyUI install (symlinked to ~/comfyui) +├── AGENTS.md # this file +├── architecture2.svg # system architecture diagram +├── backlog.md # feature backlog +└── requirements.txt # Python deps +``` + +--- + +## Configuration (`config.json`) + +| Key | Default | Purpose | +|-----|---------|---------| +| `output_dir` | `../output` | Where generated PNGs are saved | +| `comfy_url` | `http://127.0.0.1:8188` | ComfyUI base URL | +| `host` | `0.0.0.0` | FastAPI bind host | +| `port` | `8500` | FastAPI bind port | +| `max_area` | `1048576` | Max pixel budget (1 MP) | + +--- + +## Background Services + +| Service | Script | Purpose | +|---------|--------|---------| +| `comfyui-backend` | `a6000-comfy/run_comfyui.sh` | Qwen GGUF inference | +| `comfyui-api` | `tour-comfy/start_api.sh` | FastAPI HTTP server | +| `watcher` | `tour-comfy/start_watcher.sh` | Monitors output dir, auto-processes dropped images | diff --git a/architecture2.svg b/architecture2.svg index ae0e736..56ddfa2 100644 --- a/architecture2.svg +++ b/architecture2.svg @@ -1,93 +1,374 @@ - - - - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - FastAPI Edit Agent - Service: comfyui-api - Port: 8500 - HTTP API Endpoint - - - - ComfyUI Backend - Service: comfyui-backend - Port: 8188 - Qwen Model Execution - - - - Client - Image + Prompt Request - - - - - - - - - - Qwen Model - Q8 GGUF (Qwen-Rapid-NSFW-v23) - - - - Text Encoder - qwen_2.5_vl_7b_fp8_scaled.safetensors - - - - VAE Decoder - qwen_image_vae.safetensors - - - - Workflow - workflow_qwen_edit.json - - - - - - - - - - - - - \ No newline at end of file + + + + + + Qwen-Image-Edit Rapid-AIO v23 — System Architecture + RTX A6000 · CUDA 12.4 · FastAPI :8500 · ComfyUI :8188 · PostgreSQL :5433 + + + + BROWSER + + + + car.html + — Single-Page Application (no build step) + + + + Gallery View + group grid · open studio + + + Studio Viewer + filmstrip · angle bar · crop + + + Generate Tab + angles · poses · wireframe + + + Faceswap Tab + source face · video target + + + Segment Tab + SAM2 · rembg BG removal + + + Info Tab + preferred · face-book · archive + + + Privacy Overlay + disguised lock screen (P) + + + Paste / Upload Handler + add to group · skip_poses + + + + + HTTP fetch / EventSource + + + + FASTAPI :8500 — edit_api.py + + + + + Images & Groups + GET /images + GET /images/{filename} + DELETE /images/{filename} + POST /images/{f}/set-preferred + POST /images/{f}/archive|unarchive + POST /images/{f}/crop | /autocrop + GET /groups | /names | /config + + + + Upload & Batch Generation + POST /upload + group_id · skip_poses + POST /batch → async job + filenames · prompts · poses + wireframe_ref · wireframe_time + GET /jobs/{id} ← poll progress + GET /wireframe/frame/{name}?t= + + + + Face & Background + POST /faceswap + POST /remove-background/{f} + rembg U2Net + POST /remove-background-sam/{f} + SAM2 point-prompt + POST /images/{f}/extract-face + insightface crop → DB + + + + Generation Pipeline (_run_pipeline) + + + + Prompt Cleaning + strip transparent kws + + + + + ComfyUI Queue + workflow_qwen_edit.json + + + + + Qwen Inference + GGUF · VAE → PNG + + + + + SAM2 + BG removal + + + Batch worker: foreach (filename, prompt) → wireframe frame extract → pipeline → save → DB upsert + Job progress polled via GET /jobs/{id} · each item updates per-slot status dict + Camera angles: CAMERA_ANGLES[12] → prompt injected, pose=null, group_id tracked + + + + + + POST /prompt · GET /view + + + + + in-process call + + + + + psycopg2 · upsert / query + + + + COMFYUI :8188 — Qwen GGUF Inference + + + + workflow_qwen_edit.json — Node Graph + + + NODE_POSITIVE + + NODE_NEGATIVE + + NODE_LATENT + + NODE_KSAMPLER + + image1 + + image2 (wf) + + + + Qwen GGUF + Rapid-NSFW-v23_Q8_0.gguf + ~8 GB VRAM + Euler / karras sampler + + + Text Encoder + qwen_2.5_vl_7b_fp8 + _scaled.safetensors + FP8 quantised + + + VAE Decoder + qwen_image_vae + .safetensors + latent → RGB PNG + + + + Wireframe Pose Guide + /mnt/zim/tour-comfy/wireframe/*.mp4 → OpenCV frame extract → image2 slot + No ControlNet installed — Qwen uses image2 as visual layout reference + + + + ML POST-PROCESSING — in-process Python + + + + SAM2 + sam2.1_hiera_base_plus.pt + FG: 12 pts — hair → shoes + BG: 7 pts — corners + edges + Coverage check 5 %–92 % + Gaussian blur edge (r=1) + + + + rembg (U2Net) + Fallback BG removal + Used when SAM2 unavailable + or coverage out of range + Also: Segment tab button + + + + insightface buffalo_l + Faceswap: source → video frames + Face extract: bbox + 50 % pad + Saves {gid}_face.png → DB + Triggered on set-preferred + + + + GFPGAN + Face restoration + Applied post-faceswap + Optional quality upgrade + + + + POSTGRESQL 192.168.1.160:5433 db=dv + + + person table + + filename TEXT PRIMARY KEY + group_id TEXT + group_name TEXT + sort_order INTEGER (0 = preferred) + prompt TEXT — generation prompt + pose TEXT + hidden BOOLEAN + archived BOOLEAN + has_background BOOLEAN + source_refs TEXT (JSON array) + content_type TEXT (image | video) + + + + FILE SYSTEM + + + Output Directory + /mnt/zim/tour-comfy/output/ + PNGs · sidecars (.nobg.png) · face crops + timestamped — 20260622_HHMMSS_N_src.png + + + Wireframe Videos + /mnt/zim/tour-comfy/wireframe/ + *.mp4 pose reference videos + Frame extracted via OpenCV at t∈[0,1] + + + Configuration + tour-comfy/config.json + output_dir · comfy_url · host · port · max_area + R/W via GET /config · POST /config + + + + BACKGROUND SERVICES — systemd + + + watcher.py + monitors output dir + auto-processes drops + watcher.lock · watcher.log + + + Systemd Units + comfyui-backend + comfyui-api + start.sh · stop.sh + + + + + + + + LEGEND + + + HTTP (fetch) + + + ComfyUI API + + + In-process call + + + PostgreSQL (psycopg2) + + + File I/O + + TRANSPARENT BG FLOW: prompt → strip kw → Qwen (solid BG) → SAM2 point-prompt (12 FG + 7 BG pts) → coverage check → Gaussian edge + FACE FLOW: set-preferred → insightface detect → pad bbox → crop → {gid}_face.png → DB upsert → studio Info tab face-book thumbnail + diff --git a/tour-comfy/car.html b/tour-comfy/car.html index 2676000..afa3942 100644 --- a/tour-comfy/car.html +++ b/tour-comfy/car.html @@ -1428,6 +1428,33 @@ padding: 1px 3px; border-radius: 2px; margin-left: 3px; vertical-align: middle; } + .sb-camera-grid { display: grid; grid-template-columns: repeat(4,1fr); gap: 4px; margin-bottom: 8px; } + .sb-angle-btn { + padding: 5px 2px; border-radius: 5px; + border: 1px solid #2a2a2a; background: #18181b; + color: #777; font-size: 11px; cursor: pointer; user-select: none; + text-align: center; transition: all 0.12s; + } + .sb-angle-btn:hover { border-color: #444; color: #ccc; } + .sb-angle-btn.selected { border-color: #0e7490; background: #082f49; color: #7dd3fc; font-weight: 600; } + #studioAngleBar { + position: absolute; bottom: 8px; left: 50%; transform: translateX(-50%); + display: flex; gap: 4px; opacity: 0; transition: opacity 0.2s; + z-index: 10; pointer-events: none; + } + .studio-viewer:hover #studioAngleBar { opacity: 1; pointer-events: all; } + #studioAngleBar button { + font-size: 15px; background: rgba(0,0,0,0.65); border: 1px solid rgba(255,255,255,0.18); + border-radius: 4px; color: #fff; width: 34px; height: 34px; cursor: pointer; line-height: 1; + transition: background 0.12s; position: relative; + } + #studioAngleBar button:hover { background: rgba(255,255,255,0.18); } + #studioAngleBar button[title]:hover::after { + content: attr(title); position: absolute; bottom: 38px; left: 50%; transform: translateX(-50%); + background: rgba(0,0,0,0.8); color: #fff; font-size: 10px; padding: 2px 6px; + border-radius: 3px; white-space: nowrap; pointer-events: none; + } + /* template grid (faceswap / scenery) */ .sb-template-grid { display: grid; grid-template-columns: 1fr 1fr; @@ -1536,47 +1563,69 @@
You
-
What is the purpose of this system?
+
What is this system and what does it run on?
AI
- This system implements a headless image editing API using the Qwen-Rapid-NSFW-v23 model on an RTX A6000 GPU. It provides an HTTP service that accepts image and prompt inputs, processes edits using ComfyUI backend, and returns edited PNG outputs. + This is a self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs on a local RTX A6000 GPU (CUDA 12.4, 48 GB VRAM) using the Qwen 2.5 VL GGUF model (Rapid-NSFW-v23 Q8_0) via ComfyUI. The frontend is a single-page browser app (car.html) talking to a FastAPI backend on :8500.
You
-
How does it work?
+
Walk me through the image generation flow.
AI
- The system consists of two main components: a ComfyUI backend (running on port 8188) that executes the Qwen models and a FastAPI service (running on port 8500) that provides HTTP API endpoints. When you send an image and prompt to the API, it forwards the request to ComfyUI which processes the image using the Qwen-Rapid-NSFW-v23 model with specific workflow parameters. + When a generation is triggered the pipeline is:

+ 1. Prompt cleaning — if the prompt contains "transparent background", that keyword is stripped so Qwen renders a clean solid-BG image instead of producing washed-out halos.

+ 2. ComfyUI queue — the reference image is uploaded to ComfyUI (:8188) and injected into image1. An optional wireframe frame extracted from a pose-guide video goes into image2. The workflow graph (workflow_qwen_edit.json) is patched with the prompt, dimensions, seed, and sampler params, then submitted via POST /prompt.

+ 3. Qwen inference — the GGUF model (≈8 GB VRAM), text encoder (FP8), and VAE decode the latent into a PNG.

+ 4. SAM2 post-processing — if transparent was requested, SAM2 point-prompt segmentation removes the background using 12 foreground seed points (hair → shoes, shoulder / hip lateral spread) and 7 background points (corners + edge midpoints). A coverage sanity check (5–92 %) falls back to rembg if the mask is implausible.
You
-
What are the key components?
+
What are the main API endpoints?
AI
- Key components include:

- - Qwen model: Qwen-Rapid-NSFW-v23_Q8_0.gguf
- - Text encoder: qwen_2.5_vl_7b_fp8_scaled.safetensors
- - VAE decoder: qwen_image_vae.safetensors
- - Workflow execution engine using ComfyUI

- The system uses a specialized workflow (workflow_qwen_edit.json) that defines the processing pipeline with nodes for loading the model, text encoder, VAE, input image, positive/negative prompts, latent generation, sampling, and output saving. + The FastAPI service (:8500) exposes:

+ GET /images — full image list with PostgreSQL metadata (group, prompt, pose, sort order, archived flag).
+ POST /upload — add an image; with group_id + skip_poses it silently joins an existing group without triggering pose generation.
+ POST /batch — async multi-prompt generation job (filenames, prompts, poses, wireframe_ref). Poll progress via GET /jobs/{id}.
+ POST /faceswap — insightface video face swap with optional GFPGAN restoration.
+ POST /remove-background-sam/{f} — SAM2 BG removal to a .nobg.png sidecar.
+ POST /images/{f}/set-preferred — moves image to group slot 0 and queues face extraction in the background.
+ GET /wireframe/frame/{name}?t= — extract a frame at normalised time t∈[0,1] from a wireframe pose video. +
+
+
+
+
+
You
+
How does the face-book and multi-view workflow work?
+
+
+
+
+
AI
+
+ Face-book: marking an image as "preferred" (star button) moves it to sort_order=0 in the group and triggers a background task. insightface detects the largest face, adds 50 % padding on the sides and 200 % headroom above, and saves a {group_id}_face.png crop. This appears as a 72 px thumbnail in the studio Info tab for fast face-reference lookup.

+ Multi-view / camera angles: the Generate tab offers 12 camera angle presets — 8 absolute (Front, ¾ Left/Right, Side, Back, High, Low) and 4 relative (±45°, ±90°). Each fires a /batch request with the angle prompt injected and pose=null. The wireframe pose guide (a .mp4 video scrubbed to any frame) is passed as image2 in the Qwen workflow to constrain body layout without ControlNet — the controlnet model folder is empty by design.

+ Group management: all images belong to groups stored in PostgreSQL. Clipboard paste (Ctrl+V) while a group is open offers "add to this group — no poses" or "new group with pose generation". Archive hides images from the default view without deleting them.
@@ -1611,37 +1660,6 @@ titleElement.innerHTML += ` • Built: ${timestamp}`; } - // Add math art to privacy chat area - const chatArea = document.querySelector('.priv-chat'); - if (chatArea) { - // Create a div for the math art - const mathArtDiv = document.createElement('div'); - mathArtDiv.style.cssText = ` - margin-top: 20px; - padding: 15px; - background: rgba(30, 30, 30, 0.7); - border-radius: 8px; - font-family: monospace; - font-size: 12px; - line-height: 1.4; - overflow: hidden; - `; - mathArtDiv.innerHTML = ` -
Math Art Visualization:
- 
-                             .-~~-.    .-~~-.    .-~~-.    .-~~-.
-                            (  (  )   (  (  )   (  (  )   (  (  )
-                             '-~~-'    '-~~-'    '-~~-'    '-~~-'
-                            .-~~-.    .-~~-.    .-~~-.    .-~~-.
-                           (  (  )   (  (  )   (  (  )   (  (  )
-                            '-~~-'    '-~~-'    '-~~-'    '-~~-'
-
-
- π ≈ 3.14159265358979323846... -
- `; - chatArea.appendChild(mathArtDiv); - } } catch (e) { console.error('Error injecting build time or math art:', e); } @@ -1741,6 +1759,7 @@
Hidden from preview
+
@@ -1782,6 +1801,10 @@
Generation prompt
+
Order & visibility
@@ -2147,7 +2170,27 @@ let availableVideos = []; // populated from /videos let _fsActiveTab = 'swap'; let _fsSelectedPoses = new Set(); + let _sbSelectedAngles = new Set(); // selected camera angle names + let _sbWireframeRef = ''; // wireframe video name for pose guide + let _sbWireframeTime = 0.5; // normalized frame time 0–1 let _sbPoseEdits = {}; // poseName → edited text + + const CAMERA_ANGLES = [ + // Absolute camera positions + { name: 'Front', icon: '⊙', prompt: 'Head-on straight-on full-body portrait, frontal camera, realistic, transparent background', relative: false }, + { name: '¾ Left', icon: '↙', prompt: 'Three-quarter left, camera 45° to the left, full-body portrait, realistic, transparent background', relative: false }, + { name: '¾ Right', icon: '↗', prompt: 'Three-quarter right, camera 45° to the right, full-body portrait, realistic, transparent background', relative: false }, + { name: 'Side L', icon: '◁', prompt: 'Side profile, camera 90° to the left, full-body portrait, realistic, transparent background', relative: false }, + { name: 'Side R', icon: '▷', prompt: 'Side profile, camera 90° to the right, full-body portrait, realistic, transparent background', relative: false }, + { name: 'Back', icon: '⊕', prompt: 'Rear view, camera directly behind subject, full-body portrait, realistic, transparent background', relative: false }, + { name: 'High', icon: '△', prompt: "Bird's-eye view, camera above looking down, full-body, realistic, transparent background", relative: false }, + { name: 'Low', icon: '▽', prompt: "Worm's-eye view, camera below looking up, full-body portrait, realistic, transparent background", relative: false }, + // Relative rotations — applied on top of current view + { name: '↺ 45°', icon: '↺', prompt: 'Rotate 45° to the left from current view, full-body portrait, realistic, transparent background', relative: true }, + { name: '↻ 45°', icon: '↻', prompt: 'Rotate 45° to the right from current view, full-body portrait, realistic, transparent background', relative: true }, + { name: '↺ 90°', icon: '⟲', prompt: 'Rotate 90° to the left from current view, side profile, full-body portrait, realistic, transparent background', relative: true }, + { name: '↻ 90°', icon: '⟳', prompt: 'Rotate 90° to the right from current view, side profile, full-body portrait, realistic, transparent background', relative: true }, + ]; let _sbGenJobId = null; let _sbGenJobPollTimer= null; let _followLatestGid = null; // set to gid after generation to auto-jump to newest image @@ -2407,6 +2450,21 @@ }).join(''); const active = strip.querySelector('.lb-var-thumb.active'); if (active) active.scrollIntoView({ block: 'nearest', inline: 'center' }); + + // Camera angle overlay bar + updateSbAngleBar(); + + // Face-book thumbnail — show on slot 0 when a face crop exists for this group + const faceBook = document.getElementById('lbFaceBook'); + const faceThumb = document.getElementById('lbFaceThumb'); + if (faceBook && faceThumb && lbCurrentGid && lbIdx === 0 && !isVid) { + const faceFname = lbCurrentGid.replace(/\//g, '_') + '_face.png'; + faceThumb.src = IMAGE_FOLDER + faceFname + '?t=' + Date.now(); + faceThumb.onerror = () => { faceBook.style.display = 'none'; }; + faceThumb.onload = () => { faceBook.style.display = ''; }; + } else if (faceBook) { + faceBook.style.display = 'none'; + } } // Alias for old callsites @@ -2887,7 +2945,12 @@ } } if (files.length > 0) { - handleUpload(files); + // If studio is open, offer to add to current group without triggering poses + if (lbCurrentGid) { + showPasteGroupDialog(files); + } else { + handleUpload(files); + } } }); @@ -3816,7 +3879,47 @@ const donePoses = new Set(); lbNames.forEach(n => { if (filePoses[n]) donePoses.add(filePoses[n]); }); - let html = '
Poses
'; + // Camera angles section — absolute positions + const absAngles = CAMERA_ANGLES.filter(a => !a.relative); + const relAngles = CAMERA_ANGLES.filter(a => a.relative); + let html = '
Camera — absolute
'; + html += absAngles.map(a => { + const sel = _sbSelectedAngles.has(a.name) ? ' selected' : ''; + const nSafe = a.name.replace(/'/g, "\\'"); + return ``; + }).join(''); + html += '
'; + + // Relative rotations + html += '
Camera — relative
'; + html += relAngles.map(a => { + const sel = _sbSelectedAngles.has(a.name) ? ' selected' : ''; + const nSafe = a.name.replace(/'/g, "\\'"); + return ``; + }).join(''); + html += '
'; + + // Wireframe pose reference + const wfSel = _sbWireframeRef || ''; + const wfT = _sbWireframeTime; + html += `
+
Wireframe pose guide (optional)
+ + ${wfSel ? `
+ Frame: + + ${Math.round(wfT*100)}% + +
` : ''}`; + + html += '
Poses
'; if (!availablePoses || Object.keys(availablePoses).length === 0) { html += '
No poses loaded
'; } else { @@ -3874,13 +3977,89 @@ renderSidebarGenerate(); } + function toggleSbAngle(name) { + if (_sbSelectedAngles.has(name)) _sbSelectedAngles.delete(name); + else _sbSelectedAngles.add(name); + renderSidebarGenerate(); + } + + function sbSelectWireframe(val) { + _sbWireframeRef = val; + renderSidebarGenerate(); + } + + function sbUpdateWireframeTime(t) { + _sbWireframeTime = t; + const label = document.getElementById('sbWireframeTimeLabel'); + if (label) label.textContent = Math.round(t * 100) + '%'; + // Debounced thumb update + clearTimeout(sbUpdateWireframeTime._timer); + sbUpdateWireframeTime._timer = setTimeout(() => { + const thumb = document.getElementById('sbWireframeThumb'); + if (thumb && _sbWireframeRef) { + thumb.src = `${API}/wireframe/frame/${encodeURIComponent(_sbWireframeRef)}?t=${t}&_=${Date.now()}`; + } + }, 400); + } + + async function submitSingleAngle(prompt) { + if (!_fsModelFilename) return; + const bar = document.getElementById('studioAngleBar'); + if (bar) bar.style.pointerEvents = 'none'; + try { + const r = await fetch(`${API}/batch`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + filenames: [_fsModelFilename], prompt: [prompt], + poses: [null], seed: -1, max_area: 0, group_id: lbCurrentGid, + wireframe_ref: _sbWireframeRef || null, + wireframe_time: _sbWireframeTime, + }), + }); + if (r.ok) { + const { job_id } = await r.json(); + _followLatestGid = lbCurrentGid; + showToast('Generating angle…', 'info'); + // Light polling — refresh when done + const poll = () => setTimeout(async () => { + try { + const s = await fetch(`${API}/batch/${job_id}`).then(r => r.json()); + if (s.status === 'done') { refreshNow(); } + else if (s.status !== 'error' && s.status !== 'cancelled') poll(); + } catch(e) {} + }, 2000); + poll(); + } else { showToast('Angle generation failed', 'error'); } + } catch(e) { showToast('API error: ' + e, 'error'); } + if (bar) bar.style.pointerEvents = ''; + } + + function updateSbAngleBar() { + const bar = document.getElementById('studioAngleBar'); + if (!bar) return; + const isVid = lbNames[lbIdx] && (isVideo(lbNames[lbIdx]) || fileContentType[lbNames[lbIdx]] === 'video'); + const hasCrop = !!document.getElementById('cropCanvas'); + if (isVid || hasCrop || !lbNames[lbIdx]) { + bar.style.display = 'none'; + return; + } + bar.style.display = 'flex'; + // Show 6 most useful angles (skip High/Low for overlay — rare) + const overlay = CAMERA_ANGLES.slice(0, 6); + bar.innerHTML = overlay.map(a => { + const pSafe = a.prompt.replace(/'/g, "\\'"); + return ``; + }).join(''); + } + function updateSbGenBtn() { const btn = document.getElementById('sbGenBtn'); if (!btn) return; const hasPrompt = (document.getElementById('sbGenPromptInput')?.value || '').trim().length > 0; - btn.disabled = _fsSelectedPoses.size === 0 && !hasPrompt; - const n = _fsSelectedPoses.size + (hasPrompt ? 1 : 0); - btn.textContent = n > 1 ? `Generate (${n})` : 'Generate'; + const total = _fsSelectedPoses.size + _sbSelectedAngles.size + (hasPrompt ? 1 : 0); + btn.disabled = total === 0; + btn.textContent = total > 1 ? `Generate (${total})` : 'Generate'; } async function cancelSbGenerate() { @@ -3911,6 +4090,11 @@ const orig = availablePoses[name]?.text ?? availablePoses[name] ?? name; prompts.push((edited !== undefined && edited.trim()) ? edited.trim() : String(orig)); }); + // Camera angles — treated as prompts with no pose tag + _sbSelectedAngles.forEach(name => { + const ang = CAMERA_ANGLES.find(a => a.name === name); + if (ang) { prompts.push(ang.prompt); poses.push(null); } + }); if (promptVal) { prompts.push(promptVal); poses.push(null); savePromptHistory(promptVal); } if (prompts.length === 0) return; const btn = document.getElementById('sbGenBtn'); @@ -3927,7 +4111,9 @@ headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ filenames: [_fsModelFilename], prompt: prompts, - poses, seed: -1, max_area: 0, group_id: lbCurrentGid + poses, seed: -1, max_area: 0, group_id: lbCurrentGid, + wireframe_ref: _sbWireframeRef || null, + wireframe_time: _sbWireframeTime, }), }); if (!r.ok) { @@ -4656,7 +4842,7 @@ const fname = lbNames[lbIdx]; if (!fname) return; try { - const r = await fetch(`/images/${encodeURIComponent(fname)}/set-preferred`, {method:'POST'}); + const r = await fetch(`${API}/images/${encodeURIComponent(fname)}/set-preferred`, {method:'POST'}); if (!r.ok) return; fileSortOrders[fname] = 0; const url = lbUrls[lbIdx]; @@ -4666,8 +4852,48 @@ lbNames = [fname, ...otherNames]; lbIdx = 0; updateStudio(); + // Fire-and-forget face extraction for face-book + fetch(`${API}/images/${encodeURIComponent(fname)}/extract-face`, { method: 'POST' }) + .then(r => r.ok ? r.json() : null) + .then(d => { if (d?.status === 'queued') showToast('Extracting face reference…', 'info'); }) + .catch(() => {}); } catch(e) { console.error(e); } } + + function showPasteGroupDialog(files) { + document.getElementById('pasteDialog')?.remove(); + const gName = (lbCurrentGid && groupNames[lbCurrentGid]) || lbCurrentGid || 'current group'; + const gidSafe = escHtml(lbCurrentGid || ''); + const gNameSafe = escHtml(gName); + const d = document.createElement('div'); + d.id = 'pasteDialog'; + d.style.cssText = 'position:fixed;top:50%;left:50%;transform:translate(-50%,-50%);background:#1a1a1a;border:1px solid #333;border-radius:8px;padding:20px;z-index:600;min-width:280px;text-align:center;box-shadow:0 8px 32px rgba(0,0,0,0.7)'; + d.innerHTML = `
Add ${files.length} image(s)?
+
+ + + +
`; + window._pasteFiles = files; + document.body.appendChild(d); + setTimeout(() => document.getElementById('pasteDialog')?.remove(), 12000); + } + + async function handleUploadToGroup(files, groupId) { + showToast(`Adding ${files.length} image(s) to group…`); + for (const file of files) { + const fd = new FormData(); + fd.append('image', file); + fd.append('group_id', groupId); + fd.append('skip_poses', 'true'); + try { + const r = await fetch(`${API}/upload`, { method: 'POST', body: fd }); + if (!r.ok) showToast('Upload failed: ' + await r.text(), 'error'); + } catch (e) { showToast('Upload error: ' + e, 'error'); } + } + showToast('Added to group', 'success'); + refreshNow(); + } \ No newline at end of file diff --git a/tour-comfy/config.json b/tour-comfy/config.json index b01d2fe..c5d2bf9 100644 --- a/tour-comfy/config.json +++ b/tour-comfy/config.json @@ -25,5 +25,6 @@ "facefusion_dir": "~/facefusion", "facefusion_venv": "~/facefusion-venv", "sam2_checkpoint": "~/.sam/sam2.1_hiera_base_plus.pt", - "sam2_config": "configs/sam2.1/sam2.1_hiera_b+.yaml" + "sam2_config": "configs/sam2.1/sam2.1_hiera_b+.yaml", + "bg_removal": "sam2" } \ No newline at end of file diff --git a/tour-comfy/edit_api.py b/tour-comfy/edit_api.py index 3c00c42..d73dc8b 100644 --- a/tour-comfy/edit_api.py +++ b/tour-comfy/edit_api.py @@ -839,10 +839,63 @@ def _run_pipeline( } graph[NODE_POSITIVE]["inputs"][img_key] = [node_id, 0] - # Transparency detection - is_transparent = any(kw in prompt.lower() for kw in ["transparent", "no background", "remove background", "alpha channel"]) + # ── background-removal routing ──────────────────────────────────────────── + # Two configurable strategies (config.json key "bg_removal"): + # + # "rembg" (default) — strip transparent keyword → Qwen renders a natural + # scene → rembg (U2Net) separates person from any complex background. + # + # "sam2" — replace transparent keyword with "black background" → Qwen + # renders a solid black BG → SAM2 bbox segmentation on a black image + # works perfectly because the contrast is maximal. + # + # Either way, explicit "black background" in the prompt always routes to + # SAM2 (the user already set up the ideal SAM2 input). + # ───────────────────────────────────────────────────────────────────────── + _TRANSPARENT_KWS = ["transparent background", "no background", + "remove background", "alpha channel"] + _BLACK_BG_KWS = ["black background"] + + with open(CONFIG_PATH) as _cf: + _bg_conf = json.load(_cf) + bg_method = _bg_conf.get("bg_removal", "rembg") # "rembg" | "sam2" + + is_transparent = any(kw in prompt.lower() for kw in _TRANSPARENT_KWS) + is_black_bg = any(kw in prompt.lower() for kw in _BLACK_BG_KWS) + post_process = None # "rembg" | "sam2" + if is_transparent: - graph[NODE_NEGATIVE]["inputs"]["prompt"] = "checkerboard, grid, pattern, texture, background details, watermark, deformed anatomy" + if bg_method == "sam2": + # Swap "transparent background" → "black background" so Qwen renders + # a pure-black BG that SAM2 can segment with maximal contrast. + cleaned = prompt + for kw in _TRANSPARENT_KWS: + cleaned = re.sub(re.escape(kw), "black background", cleaned, flags=re.IGNORECASE) + # Collapse duplicates if multiple keywords matched + cleaned = re.sub(r"(?i)(black background[\s,]*){2,}", "black background, ", cleaned) + cleaned = re.sub(r",\s*,", ",", cleaned).strip(", ") + graph[NODE_POSITIVE]["inputs"]["prompt"] = cleaned + graph[NODE_NEGATIVE]["inputs"]["prompt"] = ( + "real background, outdoor scene, indoor scene, gradient, " + "colored background, watermark, deformed anatomy" + ) + post_process = "sam2" + else: + # Strip the keyword so Qwen renders a natural scene; rembg handles + # any background complexity reliably. + cleaned = prompt + for kw in _TRANSPARENT_KWS: + cleaned = re.sub(re.escape(kw), "", cleaned, flags=re.IGNORECASE) + cleaned = re.sub(r",\s*,", ",", cleaned) + cleaned = re.sub(r",\s*$", "", cleaned.strip()).strip(", ") + graph[NODE_POSITIVE]["inputs"]["prompt"] = cleaned + graph[NODE_NEGATIVE]["inputs"]["prompt"] = "deformed anatomy, watermark, logo" + post_process = "rembg" + + elif is_black_bg: + # Prompt already specifies a black background — ideal SAM2 input. + # Route to SAM2 regardless of the configured bg_removal method. + post_process = "sam2" graph[NODE_LATENT]["inputs"]["width"] = w graph[NODE_LATENT]["inputs"]["height"] = h @@ -853,7 +906,13 @@ def _run_pipeline( outputs = _comfy_wait(prompt_id, time.time() + GEN_TIMEOUT) png_bytes = _comfy_fetch_image(outputs) - if is_transparent: + if post_process == "sam2": + # Input has a black background (Qwen was told "black background"). + # Use threshold-derived bbox so SAM2 gets a person-specific hint + # rather than the full frame — full-frame bbox inverts the mask on + # black-bg images because the large dark region scores higher. + png_bytes = _apply_transparency_black_bg(png_bytes) + elif post_process == "rembg": png_bytes = _apply_transparency(png_bytes) return png_bytes @@ -891,7 +950,8 @@ def _move_to_trash(filepath: str): def _batch_worker(job_id: str, filenames: list, prompts: list[str], poses: list, - seed: int, max_area: int, group_id: str | None = None): + seed: int, max_area: int, group_id: str | None = None, + wireframe_ref: str | None = None, wireframe_time: float = 0.5): output_dir = _load_output_dir() for fname in filenames: actual_gid = None @@ -915,6 +975,24 @@ def _batch_worker(job_id: str, filenames: list, prompts: list[str], poses: list, try: base_pil = Image.open(fpath).convert("RGB") + + # Extract wireframe pose reference frame once per filename + pose_guide_pil = None + if wireframe_ref: + try: + wf_path = os.path.join(_load_wireframe_dir(), wireframe_ref) + cap = cv2.VideoCapture(wf_path) + total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + target_frame = max(0, min(total_frames - 1, int(total_frames * wireframe_time))) + cap.set(cv2.CAP_PROP_POS_FRAMES, target_frame) + ret, frame = cap.read() + cap.release() + if ret: + pose_guide_pil = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)) + print(f"[batch] using wireframe {wireframe_ref} frame {target_frame}/{total_frames}") + except Exception as wf_err: + print(f"[batch] wireframe extract error: {wf_err}") + for prompt, pose in zip(prompts, poses): if jobs[job_id].get("cancelled"): return @@ -924,7 +1002,8 @@ def _batch_worker(job_id: str, filenames: list, prompts: list[str], poses: list, if pose and pose.lower().strip() in ROTATE_180_POSES: pil = pil.rotate(180) - png = _run_pipeline(pil, prompt, seed, max_area) + extra_imgs = [pose_guide_pil] if pose_guide_pil else None + png = _run_pipeline(pil, prompt, seed, max_area, extra_images=extra_imgs) ts = time.strftime("%Y%m%d_%H%M%S") clean_fname = naming.get_base_name(fname) out_name = f"{ts}_{clean_fname}" @@ -1069,6 +1148,8 @@ class BatchRequest(BaseModel): max_area: int = 0 group_id: str | None = None poses: list[str | None] | None = None # pose name per prompt (same index), or None; None entries = no pose + wireframe_ref: str | None = None # wireframe video name to use as pose guide (image2 slot) + wireframe_time: float = 0.5 # normalized time (0–1) to extract the pose frame from @app.post("/batch") @@ -1085,6 +1166,7 @@ def start_batch(req: BatchRequest): t = threading.Thread( target=_batch_worker, args=(job_id, req.filenames, prompts, poses, req.seed, req.max_area, req.group_id), + kwargs={"wireframe_ref": req.wireframe_ref, "wireframe_time": req.wireframe_time}, daemon=True, ) t.start() @@ -1207,6 +1289,35 @@ def list_videos(): return {"videos": videos, "wireframe_dir": wireframe_dir} +@app.get("/wireframe/frame/{video_name}") +def wireframe_frame(video_name: str, t: float = 0.5): + """Extract a single frame at normalized time t (0–1) from a wireframe video. Returns PNG.""" + wireframe_dir = _load_wireframe_dir() + video_path = os.path.join(wireframe_dir, video_name) + if not os.path.exists(video_path): + raise HTTPException(404, f"Video not found: {video_name}") + try: + cap = cv2.VideoCapture(video_path) + total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + target = max(0, min(total - 1, int(total * max(0.0, min(1.0, t))))) + cap.set(cv2.CAP_PROP_POS_FRAMES, target) + ret, frame = cap.read() + cap.release() + if not ret: + raise HTTPException(500, "Could not read frame") + rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) + pil = Image.fromarray(rgb) + buf = io.BytesIO() + pil.save(buf, format="PNG") + buf.seek(0) + from fastapi.responses import Response + return Response(content=buf.getvalue(), media_type="image/png") + except HTTPException: + raise + except Exception as e: + raise HTTPException(500, f"Frame extraction error: {e}") + + @app.get("/wireframe/duration/{video_name}") def wireframe_duration(video_name: str): """Return duration (seconds) of a wireframe video via ffprobe.""" @@ -1573,6 +1684,42 @@ def _crop_to_bbox(pil_img: Image.Image, margin: int = 20, top_margin: int = 20, return cropped +def _extract_face_bg(filename: str, fpath: str): + """Background task: detect largest face, crop with padding, save as {group_id}_face.png.""" + try: + app_fa, _ = _load_faceswapper() + bgr = cv2.imread(fpath) + if bgr is None: + print(f"[extract-face] cannot read {fpath}") + return + faces = app_fa.get(bgr) + if not faces: + print(f"[extract-face] no face detected in {filename}") + return + face = max(faces, key=lambda f: (f.bbox[2] - f.bbox[0]) * (f.bbox[3] - f.bbox[1])) + x1, y1, x2, y2 = [int(v) for v in face.bbox] + h, w = bgr.shape[:2] + pad = int((y2 - y1) * 0.5) + x1 = max(0, x1 - pad) + y1 = max(0, y1 - pad * 2) # extra headroom above face + x2 = min(w, x2 + pad) + y2 = min(h, y2 + int(pad * 0.3)) + pil = Image.open(fpath).convert("RGBA") + cropped = pil.crop((x1, y1, x2, y2)) + person = database.get_person(filename) + group_id = person[1] if person else None + gid_tag = (group_id or "face").replace("/", "_") + face_fname = f"{gid_tag}_face.png" + face_path = os.path.join(os.path.dirname(fpath), face_fname) + cropped.save(face_path) + database.upsert_person(face_fname, filepath=face_path, group_id=group_id, + name=person[0] if person else None, + source_refs=json.dumps([filename])) + print(f"[extract-face] saved {face_fname}") + except Exception as e: + print(f"[extract-face] error for {filename}: {e}") + + def _process_upload(file_path: str, filename: str, prompts: list[str], name: str | None = None, group_id: str | None = None): output_dir = _load_output_dir() try: @@ -1631,40 +1778,49 @@ def upload_image( image: UploadFile = File(...), prompts: str = Form(""), name: str = Form(None), + group_id: str = Form(None), # optional: add to existing group + skip_poses: bool = Form(False), # optional: skip base_prompts generation ): # Load config to get output_dir (we use output_dir for UI uploads to avoid watcher conflict) with open(CONFIG_PATH, "r") as f: conf = json.load(f) output_dir = _load_output_dir() os.makedirs(output_dir, exist_ok=True) - + ts = time.strftime("%Y%m%d_%H%M%S") - safe_filename = re.sub(r'[^a-zA-Z0-9_.-]', '_', image.filename) + safe_filename = re.sub(r'[^a-zA-Z0-9_.-]', '_', image.filename or "paste") # Ensure extension if not safe_filename.lower().endswith(('.png', '.jpg', '.jpeg', '.webp')): safe_filename += ".png" - + filename = f"{ts}_{safe_filename}" file_path = os.path.join(output_dir, filename) - + with open(file_path, "wb") as f: shutil.copyfileobj(image.file, f) - + + # Fast path: add to existing group without pose generation + if group_id and skip_poses: + sort_order = database.get_next_sort_order(group_id) + database.upsert_person(filename, filepath=file_path, group_id=group_id, + sort_order=sort_order) + return {"status": "added", "filename": filename, "group_id": group_id} + prompt_list = [p.strip() for p in prompts.split(",") if p.strip()] - + # Add base-set prompts if defined in config base_prompts = conf.get("base_prompts", []) if isinstance(base_prompts, list): prompt_list.extend(base_prompts) - + if not prompt_list: # Use default prompt from config prompt_list = [conf.get("prompt", "high quality, masterpiece")] - - group_id = f"up_{uuid.uuid4().hex[:8]}" # unique per upload; avoids collisions when pasting generic filenames - background_tasks.add_task(_process_upload, file_path, filename, prompt_list, name, group_id) - - return {"status": "processing", "filename": filename, "group_id": group_id, "prompts": prompt_list} + + effective_gid = group_id or f"up_{uuid.uuid4().hex[:8]}" + background_tasks.add_task(_process_upload, file_path, filename, prompt_list, name, effective_gid) + + return {"status": "processing", "filename": filename, "group_id": effective_gid, "prompts": prompt_list} @app.post("/edit") @@ -1717,7 +1873,7 @@ def unarchive_image(filename: str): @app.post("/images/{filename}/set-preferred") -def set_image_preferred(filename: str): +def set_image_preferred(filename: str, background_tasks: BackgroundTasks): """Make this image sort_order=0 within its group, shifting others to 1,2,...""" person = database.get_person(filename) if not person: @@ -1728,9 +1884,23 @@ def set_image_preferred(filename: str): rows = database.get_group_order(group_id) others = [r[0] for r in rows if r[0] != filename] database.set_group_order(group_id, [filename] + others) + fpath = os.path.join(_load_output_dir(), filename) + if os.path.exists(fpath): + background_tasks.add_task(_extract_face_bg, filename, fpath) return {"filename": filename, "group_id": group_id} +@app.post("/images/{filename}/extract-face") +def extract_face_endpoint(filename: str, background_tasks: BackgroundTasks): + """Detect and crop the largest face from image; saves as {group_id}_face.png.""" + output_dir = _load_output_dir() + fpath = os.path.join(output_dir, filename) + if not os.path.exists(fpath): + raise HTTPException(404, "not found") + background_tasks.add_task(_extract_face_bg, filename, fpath) + return {"status": "queued", "filename": filename} + + @app.post("/images/{filename}/undress") def undress_image(filename: str, background_tasks: BackgroundTasks): """Queue a generation using the undress prompt on the given image.""" @@ -1967,14 +2137,14 @@ def _load_sam2(): def _apply_transparency_sam2(png_bytes: bytes) -> bytes: - """Remove background with SAM2 bbox-based segmentation; fallback to rembg. + """Remove background with SAM2 bbox segmentation; fallback to rembg. - Mimics the reference approach (bbox → SAM2) but without an external - detector: we pass a generous bbox covering the central subject area. - For portraits/full-body shots the subject fills most of the frame, so a - 5 % margin bbox reliably captures hair, glasses, and sandals without the - point-prompt clipping issues. multimask_output=True lets SAM2 propose - three masks; we pick the highest-scoring one. + Uses a near-full-frame bbox so SAM2 finds the largest foreground object + (the person) regardless of rotation or pose. This works well because + "transparent background" is stripped from the Qwen prompt upstream, so the + model renders a solid real background — giving SAM2 clear contrast to work + with. Point prompts were tried but produced holes in ¾-rotated poses + because the spine-column seeds land on background when the body is offset. """ predictor = _load_sam2() if predictor is False: @@ -1985,31 +2155,133 @@ def _apply_transparency_sam2(png_bytes: bytes) -> bytes: img = Image.open(io.BytesIO(png_bytes)).convert("RGB") arr = np.array(img) h, w = arr.shape[:2] - # Generous bbox — 5 % margin H, 2 % margin V — covers the whole subject - box = np.array([[int(w * 0.05), int(h * 0.02), - int(w * 0.95), int(h * 0.98)]], dtype=np.float32) + + # Near-full-frame bbox — 1 % margin so hair / shoes are inside the hint. + # SAM2 treats this as "find the prominent object within this region". + box = np.array([[int(w * 0.01), int(h * 0.01), + int(w * 0.99), int(h * 0.99)]], dtype=np.float32) + with torch.inference_mode(): predictor.set_image(arr) masks, scores, _ = predictor.predict( box=box, multimask_output=True, ) + if masks is None or len(masks) == 0: + print("[sam2] no masks returned, falling back to rembg") return _apply_transparency(png_bytes) + best = masks[int(np.argmax(scores))] + + # Sanity check: a person should cover 5 %–92 % of the frame + coverage = float(best.sum()) / (h * w) + if coverage < 0.05 or coverage > 0.92: + print(f"[sam2] mask coverage {coverage:.1%} out of range, falling back to rembg") + return _apply_transparency(png_bytes) + mask_np = best.astype(np.uint8) * 255 + + # Soft anti-aliased edge (radius 1 keeps accessory detail) + try: + from PIL import ImageFilter + alpha_img = Image.fromarray(mask_np, mode="L") + alpha_img = alpha_img.filter(ImageFilter.GaussianBlur(radius=1)) + except Exception: + alpha_img = Image.fromarray(mask_np, mode="L") + rgba = img.convert("RGBA") r, g, b, _ = rgba.split() - alpha = Image.fromarray(mask_np, mode="L") - out = Image.merge("RGBA", (r, g, b, alpha)) + out = Image.merge("RGBA", (r, g, b, alpha_img)) buf = io.BytesIO() out.save(buf, format="PNG") + print(f"[sam2] mask OK ({coverage:.1%} coverage)") return buf.getvalue() except Exception as e: print(f"[sam2] inference error, falling back to rembg: {e}") return _apply_transparency(png_bytes) +def _apply_transparency_black_bg(png_bytes: bytes) -> bytes: + """Background removal for black-background Qwen output (bg_removal=sam2 mode). + + Strategy: + 1. Threshold: any pixel with max-channel > 25 is person (non-black). + This correctly identifies the subject regardless of pose or rotation. + 2. Derive a tight person bounding-box from the threshold mask. + 3. Run SAM2 with that box for sub-pixel edge refinement. + Accept SAM2 result only when its coverage is close (±30 pp) to the + threshold estimate — this rejects the inverted-mask failure mode where + SAM2 picks the large dark region as the "object". + 4. Fall back to the threshold mask (Gaussian-blurred edges) if SAM2 + is unavailable, errors, or diverges. + + Do NOT use the full-frame bbox here: on black-background images the large + dark region scores higher than the person, causing SAM2 to invert the mask. + """ + import numpy as np + import torch + from PIL import ImageFilter + + img = Image.open(io.BytesIO(png_bytes)).convert("RGB") + arr = np.array(img) + h, w = arr.shape[:2] + + # Step 1 — threshold: non-black pixels are the person + is_person = np.max(arr, axis=2) > 25 + thresh_cov = float(is_person.sum()) / (h * w) + print(f"[bg-black] threshold coverage: {thresh_cov:.1%}") + + if not is_person.any(): + print("[bg-black] all-black image — falling back to rembg") + return _apply_transparency(png_bytes) + + # Step 2 — tight bounding box from threshold + rows = np.any(is_person, axis=1) + cols = np.any(is_person, axis=0) + rmin = int(np.where(rows)[0][0]); rmax = int(np.where(rows)[0][-1]) + cmin = int(np.where(cols)[0][0]); cmax = int(np.where(cols)[0][-1]) + margin = int(min(h, w) * 0.02) + x1 = max(0, cmin - margin); y1 = max(0, rmin - margin) + x2 = min(w, cmax + margin); y2 = min(h, rmax + margin) + + # Step 3 — SAM2 with the person-specific bbox + predictor = _load_sam2() + if predictor is not False: + box = np.array([[x1, y1, x2, y2]], dtype=np.float32) + try: + with torch.inference_mode(): + predictor.set_image(arr) + masks, scores, _ = predictor.predict(box=box, multimask_output=True) + + if masks is not None and len(masks) > 0: + best = masks[int(np.argmax(scores))] + sam_cov = float(best.sum()) / (h * w) + print(f"[bg-black] SAM2 coverage: {sam_cov:.1%}") + + if 0.03 < sam_cov < 0.95 and abs(sam_cov - thresh_cov) < 0.30: + mask_np = best.astype(np.uint8) * 255 + alpha_img = Image.fromarray(mask_np, "L").filter(ImageFilter.GaussianBlur(radius=1)) + rgba = img.convert("RGBA"); r, g, b, _ = rgba.split() + out = Image.merge("RGBA", (r, g, b, alpha_img)) + buf = io.BytesIO(); out.save(buf, "PNG") + print(f"[bg-black] SAM2 accepted ✓") + return buf.getvalue() + else: + print(f"[bg-black] SAM2 diverged ({sam_cov:.1%} vs {thresh_cov:.1%}) — threshold fallback") + except Exception as e: + print(f"[bg-black] SAM2 error: {e} — threshold fallback") + + # Step 4 — fallback: threshold mask with soft edge blur + print("[bg-black] using threshold mask") + mask_np = is_person.astype(np.uint8) * 255 + alpha_img = Image.fromarray(mask_np, "L").filter(ImageFilter.GaussianBlur(radius=2)) + rgba = img.convert("RGBA"); r, g, b, _ = rgba.split() + out = Image.merge("RGBA", (r, g, b, alpha_img)) + buf = io.BytesIO(); out.save(buf, "PNG") + return buf.getvalue() + + @app.post("/remove-background-sam/{filename}") def remove_background_sam(filename: str): """SAM2-based background removal. diff --git a/tour-comfy/test_transparency.py b/tour-comfy/test_transparency.py new file mode 100644 index 0000000..543687b --- /dev/null +++ b/tour-comfy/test_transparency.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +""" +Validate background removal strategies. + +Usage: + python test_transparency.py [image.png ...] + +Writes comparison files next to each input: + *_rembg.png — pure rembg (bg_removal=rembg path) + *_blackbg.png — simulated black-bg composite (what Qwen renders in sam2 mode) + *_thresh.png — threshold mask only (non-black pixels → person) + *_thresh_sam2.png — threshold bbox → SAM2 edge refinement (new sam2 mode path) +""" +import io, sys, os +import numpy as np +from PIL import Image, ImageFilter + +OUTPUT_DIR = "/mnt/zim/tour-comfy/output" +VENV_SITE = "/home/mike/comfyui/venv/lib/python3.13/site-packages" +SAM2_CKPT = os.path.expanduser("~/.sam/sam2.1_hiera_base_plus.pt") +SAM2_CFG = "configs/sam2.1/sam2.1_hiera_b+.yaml" + + +# ── rembg ────────────────────────────────────────────────────────────────────── +def apply_rembg(png_bytes: bytes) -> bytes: + from rembg import remove + return remove(png_bytes) + + +# ── SAM2 loader ──────────────────────────────────────────────────────────────── +_predictor = None +def load_sam2(): + global _predictor + if _predictor is not None: + return _predictor + try: + import torch + from sam2.build_sam import build_sam2 + from sam2.sam2_image_predictor import SAM2ImagePredictor + model = build_sam2(SAM2_CFG, SAM2_CKPT, device="cuda") + _predictor = SAM2ImagePredictor(model) + print("[sam2] loaded") + except Exception as e: + print(f"[sam2] FAILED: {e}") + _predictor = False + return _predictor + + +# ── Simulate black-bg Qwen output ───────────────────────────────────────────── +def make_black_bg(png_bytes: bytes) -> bytes: + """Composite a rembg cutout onto pure black — simulates Qwen 'black background' output.""" + rgba = Image.open(io.BytesIO(apply_rembg(png_bytes))).convert("RGBA") + bg = Image.new("RGBA", rgba.size, (0, 0, 0, 255)) + bg.paste(rgba, mask=rgba.split()[3]) + out = bg.convert("RGB") + buf = io.BytesIO(); out.save(buf, "PNG"); return buf.getvalue() + + +# ── Threshold-only mask ──────────────────────────────────────────────────────── +def apply_threshold_mask(png_bytes: bytes, threshold: int = 25) -> bytes: + """Find non-black pixels → person mask. No SAM2 needed.""" + img = Image.open(io.BytesIO(png_bytes)).convert("RGB") + arr = np.array(img) + h, w = arr.shape[:2] + + is_person = np.max(arr, axis=2) > threshold + coverage = is_person.sum() / (h * w) + print(f" [threshold] person coverage: {coverage:.1%}") + + if not is_person.any(): + print(" [threshold] all-black image — no person found") + return png_bytes + + mask_np = is_person.astype(np.uint8) * 255 + alpha_img = Image.fromarray(mask_np, "L").filter(ImageFilter.GaussianBlur(radius=2)) + rgba = img.convert("RGBA"); r, g, b, _ = rgba.split() + out = Image.merge("RGBA", (r, g, b, alpha_img)) + buf = io.BytesIO(); out.save(buf, "PNG"); return buf.getvalue() + + +# ── NEW: Threshold bbox → SAM2 refinement (sam2 mode path) ──────────────────── +def apply_thresh_sam2(png_bytes: bytes, threshold: int = 25) -> bytes: + """ + For black-background Qwen output: + 1. Threshold to find person bbox (non-black pixels) + 2. Run SAM2 with that tight bbox for clean edge refinement + 3. Fallback to threshold mask if SAM2 unavailable or mask looks wrong + """ + import torch + + img = Image.open(io.BytesIO(png_bytes)).convert("RGB") + arr = np.array(img) + h, w = arr.shape[:2] + + # Step 1 — threshold + is_person = np.max(arr, axis=2) > threshold + thresh_cov = is_person.sum() / (h * w) + print(f" [thresh_sam2] threshold person coverage: {thresh_cov:.1%}") + + if not is_person.any(): + print(" [thresh_sam2] all-black — fallback to rembg") + return apply_rembg(png_bytes) + + rows = np.any(is_person, axis=1) + cols = np.any(is_person, axis=0) + rmin = int(np.where(rows)[0][0]); rmax = int(np.where(rows)[0][-1]) + cmin = int(np.where(cols)[0][0]); cmax = int(np.where(cols)[0][-1]) + + margin = int(min(h, w) * 0.02) + y1 = max(0, rmin - margin); y2 = min(h, rmax + margin) + x1 = max(0, cmin - margin); x2 = min(w, cmax + margin) + print(f" [thresh_sam2] person bbox (+margin): ({x1},{y1})-({x2},{y2})") + + # Step 2 — SAM2 with person-specific bbox + predictor = load_sam2() + if predictor is not False: + box = np.array([[x1, y1, x2, y2]], dtype=np.float32) + try: + with torch.inference_mode(): + predictor.set_image(arr) + masks, scores, _ = predictor.predict(box=box, multimask_output=True) + + if masks is not None and len(masks) > 0: + best = masks[int(np.argmax(scores))] + sam_cov = float(best.sum()) / (h * w) + print(f" [thresh_sam2] SAM2 coverage: {sam_cov:.1%} (threshold was {thresh_cov:.1%})") + + # Accept SAM2 result if coverage is within reasonable range of threshold + if 0.03 < sam_cov < 0.95 and abs(sam_cov - thresh_cov) < 0.30: + mask_np = best.astype(np.uint8) * 255 + alpha_img = Image.fromarray(mask_np, "L").filter(ImageFilter.GaussianBlur(radius=1)) + rgba = img.convert("RGBA"); r, g, b, _ = rgba.split() + out = Image.merge("RGBA", (r, g, b, alpha_img)) + buf = io.BytesIO(); out.save(buf, "PNG") + print(" [thresh_sam2] SAM2 result accepted ✓") + return buf.getvalue() + else: + print(f" [thresh_sam2] SAM2 coverage diverged from threshold — using threshold mask") + except Exception as e: + print(f" [thresh_sam2] SAM2 error: {e} — using threshold mask") + else: + print(" [thresh_sam2] SAM2 not available — using threshold mask") + + # Step 3 — fallback: threshold mask with soft edges + mask_np = is_person.astype(np.uint8) * 255 + alpha_img = Image.fromarray(mask_np, "L").filter(ImageFilter.GaussianBlur(radius=2)) + rgba = img.convert("RGBA"); r, g, b, _ = rgba.split() + out = Image.merge("RGBA", (r, g, b, alpha_img)) + buf = io.BytesIO(); out.save(buf, "PNG") + print(" [thresh_sam2] threshold mask used as fallback") + return buf.getvalue() + + +# ── main ─────────────────────────────────────────────────────────────────────── +if __name__ == "__main__": + paths = sys.argv[1:] if len(sys.argv) > 1 else [ + os.path.join(OUTPUT_DIR, "20260622_181910_0_20260619_124038_image.png"), + ] + + for path in paths: + if not os.path.exists(path): + print(f"SKIP (not found): {path}"); continue + stem = os.path.splitext(path)[0] + print(f"\n══ {os.path.basename(path)} ══") + with open(path, "rb") as f: + raw = f.read() + + print("1. rembg (bg_removal=rembg path)...") + rb = apply_rembg(raw) + with open(stem + "_rembg.png", "wb") as f: f.write(rb) + print(f" → {os.path.basename(stem)}_rembg.png") + + print("2. Simulate black-bg Qwen output...") + bb = make_black_bg(raw) + with open(stem + "_blackbg.png", "wb") as f: f.write(bb) + print(f" → {os.path.basename(stem)}_blackbg.png") + + print("3. Threshold-only mask on black-bg image...") + tm = apply_threshold_mask(bb) + with open(stem + "_thresh.png", "wb") as f: f.write(tm) + print(f" → {os.path.basename(stem)}_thresh.png") + + print("4. Threshold bbox → SAM2 refinement on black-bg image (NEW sam2 mode path)...") + ts = apply_thresh_sam2(bb) + with open(stem + "_thresh_sam2.png", "wb") as f: f.write(ts) + print(f" → {os.path.basename(stem)}_thresh_sam2.png") + + print("\n── Done ──") + print(" *_rembg.png rembg on real background (bg_removal=rembg path)") + print(" *_thresh.png threshold-only on black bg") + print(" *_thresh_sam2.png threshold-bbox → SAM2 on black bg (NEW sam2 mode path)")