qwen-image/AGENTS.md

Qwen-Image-Edit Rapid-AIO v23 — System Documentation

Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the Qwen 2.5 VL GGUF model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)

---

Hardware

Component	Spec
GPU	NVIDIA RTX A6000 (48 GB VRAM)
CUDA	12.4
PyTorch	2.6.0+cu124
ComfyUI	latest pinned (a6000-comfy/)

---

Services

1. ComfyUI Backend — :8188

• Script: a6000-comfy/run_comfyui.sh
• Systemd: comfyui-backend
• Purpose: Executes the Qwen GGUF model via workflow_qwen_edit.json

Loaded models

Role	File
GGUF LLM	Qwen-Rapid-NSFW-v23_Q8_0.gguf
Text encoder	qwen_2.5_vl_7b_fp8_scaled.safetensors
VAE	qwen_image_vae.safetensors

Workflow nodes (workflow_qwen_edit.json):

• NODE_POSITIVE — positive text prompt
• NODE_NEGATIVE — negative prompt
• NODE_LATENT — latent dimensions (width × height)
• NODE_KSAMPLER — sampler params (Euler/karras, seed, steps, CFG)
• image1 — reference image (base64 uploaded to ComfyUI /upload/image)
• image2 — optional wireframe pose guide (extracted from wireframe video)

---

2. FastAPI API — :8500

• Script: tour-comfy/edit_api.py
• Systemd: comfyui-api
• DB: PostgreSQL at 192.168.1.160:5433, database dv, table person

Core Endpoints

Method	Path	Purpose
GET	/images	List all images with DB metadata
GET	/images/{filename}	Serve image file
POST	/upload	Upload image; optionally add to existing group (group_id + skip_poses)
POST	/batch	Queue multi-prompt generation job (async)
GET	/jobs/{id}	Poll batch job status / per-item progress
POST	/images/{filename}/set-preferred	Move to group slot 0; triggers face extraction background task
POST	/images/{filename}/extract-face	Extract & save face crop via insightface (background task)
POST	/images/{filename}/archive	Soft-archive (hidden from default view)
POST	/images/{filename}/unarchive	Restore archived image
POST	/images/{filename}/set-hidden	Toggle studio visibility
DELETE	/images/{filename}	Delete image + DB record
POST	/images/{filename}/crop	Crop to pixel rect; as_copy:true crops a referenced copy instead of in-place
POST	/images/{filename}/autocrop	Auto-trim transparent border
POST	/images/{filename}/rotate	Rotate clockwise in 90° steps (lossless transpose, in-place)
POST	/images/{filename}/invert-alpha	Invert the alpha channel (fixes wrong-segment BG removal)
POST	/images/{filename}/duplicate	Copy into same group with source_refs=[original]
POST	/faceswap	insightface video faceswap
POST	/remove-background/{filename}	rembg BG removal, overwrites in-place; sets has_background=false + refreshes static data
POST	/remove-background-sam/{filename}	SAM2 BG removal → .nobg.png sidecar
GET	/wireframe/frame/{video_name}	Extract frame at ?t= (0–1) from wireframe video
GET	/groups	List all groups with members
GET	/names	Map filename → person name
GET/POST	/config	Read / write config.json
GET	/poses	Load pose library from poses.md
POST	/poses	Create / update / rename a pose (old_name to rename) → rewrites poses.md
DELETE	/poses/{name}	Delete a pose from poses.md
GET	/pose/check	Report pose-estimator availability + backend
POST	/images/{filename}/pose	Estimate COCO-17 body keypoints (caches descriptor for similar-search)
GET	/pose/similar/{filename}	Rank library images by pose similarity to this image
POST	/pose/similar	Rank library images by similarity to a supplied (edited) skeleton
POST	/pose/index	Build pose descriptors for the whole library (daemon thread)
GET	/pose/index/status	Poll pose-index build progress

---

Generation Pipeline (_run_pipeline)

prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)

---

Batch Worker (_batch_worker)

Runs in a background thread pool. One thread per batch job item.

For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})

---

Camera Angle Presets

Eight absolute + four relative angle prompts defined as CAMERA_ANGLES in car.html. Each triggers a /batch request with pose=null and the angle prompt injected.

Name	Direction
Front	Straight-on frontal
¾ Left / ¾ Right	45° rotation
Side L / Side R	90° profile
Back	Rear view
High / Low	Bird's-eye / worm's-eye
↺ 45° / ↻ 45°	Relative +45° left / right
↺ 90° / ↻ 90°	Relative +90° left / right

---

Background Removal

Two strategies are available:

Configurable BG removal — config.json key "bg_removal"

Set to "rembg" (default) or "sam2".

"rembg" mode

• Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
• _apply_transparency (rembg U2Net) post-processes the result
• Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

"sam2" mode

• Replaces transparent keywords with "black background" in the Qwen prompt → Qwen renders the person against a pure black background
• Negative prompt discourages any real-background content
• _apply_transparency_sam2 (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
• Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

Explicit "black background" in prompt

• Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
• Prompt is passed to Qwen unchanged

SAM2 implementation (_apply_transparency_sam2)

• Model: sam2.1_hiera_base_plus.pt at ~/.sam/sam2.1_hiera_base_plus.pt
• Bbox [1%, 1%, 99%, 99%], multimask_output=True, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
• Also called by the explicit /remove-background-sam/{filename} endpoint

---

Face Processing (insightface)

Model: buffalo_l

Faceswap (/faceswap):

• Reads source face from reference image
• Swaps into each frame of a target video via OpenCV
• Optionally runs GFPGAN face restoration post-swap

Face extraction (_extract_face_bg background task):

• Triggered on set-preferred or explicit extract-face endpoint
• Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
• Saves as {group_id}_face.png in output dir
• DB-registered for display in the studio Info tab face-book thumbnail

---

Database Schema (person table)

Column	Type	Purpose
filename	TEXT PK	Unique image filename
filepath	TEXT	Absolute file path
group_id	TEXT	Group membership
group_name	TEXT	Human-readable group label
sort_order	INTEGER	Position within group (0 = preferred)
name	TEXT	Person / character name
prompt	TEXT	Generation prompt used
pose	TEXT	Pose tag
hidden	BOOLEAN	Hidden from studio preview
archived	BOOLEAN	Soft-deleted
has_background	BOOLEAN	Whether BG was removed
has_clothing	BOOLEAN	Content classification flag
source_refs	TEXT (JSON)	Original filenames this was derived from
content_type	TEXT	image or video
faceswap_source_video	TEXT	Source video for faceswapped clips

---

Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

Modes:

• Gallery — grid of all groups; click to open Studio
• Studio — filmstrip + large viewer + sidebar tabs

Sidebar tabs:

Tab	Purpose
Generate	Camera angle picker, pose selector (with inline edit/add/delete → poses.md), wireframe guide, auto-growing prompt override with history autocomplete, fine-tune textareas, multi-ref thumbnails (click to deselect), batch submit
Info	Metadata, preferred toggle, face-book thumbnail, hide/archive/delete, manual crop (copy or in-place), rotate ±90°, No-BG / Invert-α, duplicate, Pose preview (draggable + similar-pose search)
Faceswap	Source face selector, target video, preview quality toggle
Segment	SAM2 and rembg BG removal buttons
Scenery	Scenery video reference for generation

Key state variables:

• lbCurrentGid — active group ID
• lbIdx — current filmstrip position
• _followLatestGid — auto-follow newly generated image
• _sbSelectedAngles — selected camera angle names
• _sbWireframeRef / _sbWireframeTime — wireframe video + frame time
• privacyMode — privacy overlay toggle (key: P)

---

Wireframe Pose Guide

Wireframe videos stored in /mnt/zim/tour-comfy/wireframe/.
The studio Generate tab shows a video picker + time scrubber. On generation:

1. Backend calls GET /wireframe/frame/{name}?t=0.5 to extract the chosen frame
2. Frame passed as image2 in the ComfyUI workflow (second visual reference)
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via image2 slot

---

Pose Tools (2D body pose)

A posenet-style toolset built on a feature-detected estimator. Preferred backend is rtmlib (RTMPose, ONNX via the already-installed onnxruntime); mediapipe is a fallback. If neither is installed the endpoints return 501 and the UI shows an install hint.

• Install (A6000 venv): ~/comfyui/venv/bin/pip install rtmlib. Models (~150 MB) auto- download to ~/.cache/rtmlib on first use.
• CUDA libs: onnxruntime-gpu needs the venv's nvidia-*-cu12 libs on the loader path. a6000-comfy/start_api.sh adds them to LD_LIBRARY_PATH at launch (torch finds them via RPATH, onnxruntime does not).

Output format: COCO-17 keypoints (POSE_KEYPOINT_NAMES) + POSE_SKELETON edge list, returned in image pixels with per-joint score.

Studio overlay (car.html): the Pose button overlays the skeleton on the viewer (letterbox-aware canvas). Joints are draggable to refine/explore a pose. Overlay toolbar: Reset (re-detect) and Similar pose (rank the library by the current — possibly edited — skeleton). Results render as a thumbnail strip; clicking one opens that image's group.

Similarity model (edit_api.py):

• _pose_descriptor(keypoints) → translation/scale-invariant vector: centered on hip midpoint (fallback shoulders), scaled by torso length, 17×(x,y) + visibility mask. None if < 6 joints.
• _pose_distance(a, b) → weighted L2 over jointly-visible joints, taking the min of the direct and the left-right-mirrored comparison so mirrored poses match.
• Pose index: descriptors are cached in <output>/_data/poses_index.json. It fills incrementally whenever /images/{f}/pose runs, or in bulk via /pose/index (daemon thread, batched writes every 50, progress via /pose/index/status).

File Layout

qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file
├── architecture2.svg     # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps

---

Configuration (config.json)

Key	Default	Purpose
output_dir	../output	Where generated PNGs are saved
comfy_url	http://127.0.0.1:8188	ComfyUI base URL
host	0.0.0.0	FastAPI bind host
port	8500	FastAPI bind port
max_area	1048576	Max pixel budget (1 MP)

---

Background Services

Service	Script	Purpose
comfyui-backend	a6000-comfy/run_comfyui.sh	Qwen GGUF inference
comfyui-api	tour-comfy/start_api.sh	FastAPI HTTP server
watcher	tour-comfy/start_watcher.sh	Monitors output dir, auto-processes dropped images