Files

mike 3d44c7aba4 The previous SAM2 full-frame bbox approach inverts the mask on black-background images. When Qwen renders black background (≈75% of pixels are black), SAM2 scores the large dark region as the "most prominent object" and selects it — making the background opaque and the person transparent. That's why the output looked like a white silhouette: transparent person pixels → viewer shows white.

New _apply_transparency_black_bg function (called when bg_removal=sam2):
1.
Threshold — any pixel with max-channel > 25 = person. Finds the person's exact bounding box without any model confusion.
2.
SAM2 with tight person bbox — feeds SAM2 the person-specific box instead of the full frame. SAM2 now segments within the person area for clean sub-pixel edges.
3.
Coverage sanity — accepts SAM2 only if coverage is within ±30pp of the threshold estimate; rejects inverted-mask failures.
4.
Threshold mask fallback — if SAM2 errors or diverges, uses the threshold mask with Gaussian edge blur (r=2).
Test result: Person RGB mean (146, 101, 86) — correct skin tones. 74.5% transparent background, 24% opaque person. ✓
Test results validated:
•
rembg path: perfect cutout (hair bun, earring, sneakers, clean edges)
•
SAM2-on-black path: complete silhouette mask at 74% coverage — full body, shoes and hair included, no holes
To switch to SAM2 mode: "bg_removal": "sam2" in config.json. No restart needed — the config is read per-request.

2026-06-22 22:09:18 +02:00

11 KiB

Raw Blame History

Qwen-Image-Edit Rapid-AIO v23 — System Documentation

Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the Qwen 2.5 VL GGUF model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)

Hardware

Component	Spec
GPU	NVIDIA RTX A6000 (48 GB VRAM)
CUDA	12.4
PyTorch	2.6.0+cu124
ComfyUI	latest pinned (a6000-comfy/)

Services

1. ComfyUI Backend — `:8188`

Script: a6000-comfy/run_comfyui.sh
Systemd: comfyui-backend
Purpose: Executes the Qwen GGUF model via workflow_qwen_edit.json

Loaded models

Role	File
GGUF LLM	`Qwen-Rapid-NSFW-v23_Q8_0.gguf`
Text encoder	`qwen_2.5_vl_7b_fp8_scaled.safetensors`
VAE	`qwen_image_vae.safetensors`

Workflow nodes (workflow_qwen_edit.json):

NODE_POSITIVE — positive text prompt
NODE_NEGATIVE — negative prompt
NODE_LATENT — latent dimensions (width × height)
NODE_KSAMPLER — sampler params (Euler/karras, seed, steps, CFG)
image1 — reference image (base64 uploaded to ComfyUI /upload/image)
image2 — optional wireframe pose guide (extracted from wireframe video)

2. FastAPI API — `:8500`

Script: tour-comfy/edit_api.py
Systemd: comfyui-api
DB: PostgreSQL at 192.168.1.160:5433, database dv, table person

Core Endpoints

Method	Path	Purpose
`GET`	`/images`	List all images with DB metadata
`GET`	`/images/{filename}`	Serve image file
`POST`	`/upload`	Upload image; optionally add to existing group (`group_id` + `skip_poses`)
`POST`	`/batch`	Queue multi-prompt generation job (async)
`GET`	`/jobs/{id}`	Poll batch job status / per-item progress
`POST`	`/images/{filename}/set-preferred`	Move to group slot 0; triggers face extraction background task
`POST`	`/images/{filename}/extract-face`	Extract & save face crop via insightface (background task)
`POST`	`/images/{filename}/archive`	Soft-archive (hidden from default view)
`POST`	`/images/{filename}/unarchive`	Restore archived image
`POST`	`/images/{filename}/set-hidden`	Toggle studio visibility
`DELETE`	`/images/{filename}`	Delete image + DB record
`POST`	`/images/{filename}/crop`	Crop to pixel rect (in-place)
`POST`	`/autocrop/{filename}`	Auto-trim transparent border
`POST`	`/faceswap`	insightface video faceswap
`POST`	`/remove-background/{filename}`	rembg BG removal → `.nobg.png` sidecar
`POST`	`/remove-background-sam/{filename}`	SAM2 BG removal → `.nobg.png` sidecar
`GET`	`/wireframe/frame/{video_name}`	Extract frame at `?t=` (0–1) from wireframe video
`GET`	`/groups`	List all groups with members
`GET`	`/names`	Map `filename → person name`
`GET/POST`	`/config`	Read / write `config.json`

Generation Pipeline (`_run_pipeline`)

prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)

Batch Worker (`_batch_worker`)

Runs in a background thread pool. One thread per batch job item.

For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})

Camera Angle Presets

Eight absolute + four relative angle prompts defined as CAMERA_ANGLES in car.html. Each triggers a /batch request with pose=null and the angle prompt injected.

Name	Direction
Front	Straight-on frontal
¾ Left / ¾ Right	45° rotation
Side L / Side R	90° profile
Back	Rear view
High / Low	Bird's-eye / worm's-eye
↺ 45° / ↻ 45°	Relative +45° left / right
↺ 90° / ↻ 90°	Relative +90° left / right

Background Removal

Two strategies are available:

Configurable BG removal — `config.json` key `"bg_removal"`

Set to "rembg" (default) or "sam2".

`"rembg"` mode

Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
_apply_transparency (rembg U2Net) post-processes the result
Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

`"sam2"` mode

Replaces transparent keywords with "black background" in the Qwen prompt → Qwen renders the person against a pure black background
Negative prompt discourages any real-background content
_apply_transparency_sam2 (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

Explicit `"black background"` in prompt

Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
Prompt is passed to Qwen unchanged

SAM2 implementation (`_apply_transparency_sam2`)

Model: sam2.1_hiera_base_plus.pt at ~/.sam/sam2.1_hiera_base_plus.pt
Bbox [1%, 1%, 99%, 99%], multimask_output=True, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
Also called by the explicit /remove-background-sam/{filename} endpoint

Face Processing (insightface)

Model: buffalo_l

Faceswap (/faceswap):

Reads source face from reference image
Swaps into each frame of a target video via OpenCV
Optionally runs GFPGAN face restoration post-swap

Face extraction (_extract_face_bg background task):

Triggered on set-preferred or explicit extract-face endpoint
Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
Saves as {group_id}_face.png in output dir
DB-registered for display in the studio Info tab face-book thumbnail

Database Schema (`person` table)

Column	Type	Purpose
filename	TEXT PK	Unique image filename
filepath	TEXT	Absolute file path
group_id	TEXT	Group membership
group_name	TEXT	Human-readable group label
sort_order	INTEGER	Position within group (0 = preferred)
name	TEXT	Person / character name
prompt	TEXT	Generation prompt used
pose	TEXT	Pose tag
hidden	BOOLEAN	Hidden from studio preview
archived	BOOLEAN	Soft-deleted
has_background	BOOLEAN	Whether BG was removed
has_clothing	BOOLEAN	Content classification flag
source_refs	TEXT (JSON)	Original filenames this was derived from
content_type	TEXT	`image` or `video`
faceswap_source_video	TEXT	Source video for faceswapped clips

Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

Modes:

Gallery — grid of all groups; click to open Studio
Studio — filmstrip + large viewer + sidebar tabs

Sidebar tabs:

Tab	Purpose
Generate	Camera angle picker, pose selector, wireframe guide, prompt override, fine-tune textarea, batch submit
Info	Metadata, preferred toggle, face-book thumbnail, hide/archive/delete actions, manual crop
Faceswap	Source face selector, target video, preview quality toggle
Segment	SAM2 and rembg BG removal buttons
Scenery	Scenery video reference for generation

Key state variables:

lbCurrentGid — active group ID
lbIdx — current filmstrip position
_followLatestGid — auto-follow newly generated image
_sbSelectedAngles — selected camera angle names
_sbWireframeRef / _sbWireframeTime — wireframe video + frame time
privacyMode — privacy overlay toggle (key: P)

Wireframe Pose Guide

Wireframe videos stored in /mnt/zim/tour-comfy/wireframe/.
The studio Generate tab shows a video picker + time scrubber. On generation:

Backend calls GET /wireframe/frame/{name}?t=0.5 to extract the chosen frame
Frame passed as image2 in the ComfyUI workflow (second visual reference)
No ControlNet installed — Qwen uses the wireframe as a layout hint via image2 slot

File Layout

qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file
├── architecture2.svg     # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps

Configuration (`config.json`)

Key	Default	Purpose
`output_dir`	`../output`	Where generated PNGs are saved
`comfy_url`	`http://127.0.0.1:8188`	ComfyUI base URL
`host`	`0.0.0.0`	FastAPI bind host
`port`	`8500`	FastAPI bind port
`max_area`	`1048576`	Max pixel budget (1 MP)

Background Services

Service	Script	Purpose
`comfyui-backend`	`a6000-comfy/run_comfyui.sh`	Qwen GGUF inference
`comfyui-api`	`tour-comfy/start_api.sh`	FastAPI HTTP server
`watcher`	`tour-comfy/start_watcher.sh`	Monitors output dir, auto-processes dropped images

11 KiB Raw Blame History Unescape Escape

Qwen-Image-Edit Rapid-AIO v23 — System Documentation

Overview

Hardware

Services

1. ComfyUI Backend — :8188

2. FastAPI API — :8500

Core Endpoints

Generation Pipeline (_run_pipeline)

Batch Worker (_batch_worker)