qwen-image/AGENTS.md

Qwen-Image-Edit Rapid-AIO v23 — System Documentation

Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the Qwen 2.5 VL GGUF model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)

---

Hardware

Component	Spec
GPU	NVIDIA RTX A6000 (48 GB VRAM)
CUDA	12.4
PyTorch	2.6.0+cu124
ComfyUI	latest pinned (a6000-comfy/)

---

Services

1. ComfyUI Backend — :8188

• Script: a6000-comfy/run_comfyui.sh
• Systemd: comfyui-backend
• Purpose: Executes the Qwen GGUF model via workflow_qwen_edit.json

Loaded models

Role	File
GGUF LLM	Qwen-Rapid-NSFW-v23_Q8_0.gguf
Text encoder	qwen_2.5_vl_7b_fp8_scaled.safetensors
VAE	qwen_image_vae.safetensors

Workflow nodes (workflow_qwen_edit.json):

• NODE_POSITIVE — positive text prompt
• NODE_NEGATIVE — negative prompt
• NODE_LATENT — latent dimensions (width × height)
• NODE_KSAMPLER — sampler params (Euler/karras, seed, steps, CFG)
• image1 — reference image (base64 uploaded to ComfyUI /upload/image)
• image2 — optional wireframe pose guide (extracted from wireframe video)

---

2. FastAPI API — :8500

• Script: tour-comfy/edit_api.py
• Systemd: comfyui-api
• DB: PostgreSQL at 192.168.1.160:5433, database dv, table person

Endpoints Library

Core Generation & Upload

Method	Path	Purpose
POST	/upload	Upload image; optionally add to existing group (group_id + skip_poses)
POST	/batch	Queue multi-prompt generation job (async)
GET	/batch/{job_id}	Poll batch job status / per-item progress
DELETE	/batch/{job_id}	Cancel active batch job
POST	/multi-ref	Queue a generation job with multiple reference images
POST	/refine-prompt	Refine a generation prompt using the external uncensored chat LLM
POST	/edit	Edit uploaded image
POST	/generate-scenery	Generate scenery background based on video frame + reference image
GET	/scenery/library	Return all scenery images grouped by source video reference

Image Editing & Canvas Expansion (Padding & Crop)

Method	Path	Purpose
POST	/images/{filename}/crop	Crop to pixel rect; as_copy:true crops a referenced copy instead of modifying in-place
POST	/images/{filename}/autocrop	Auto-trim transparent borders in-place
POST	/images/{filename}/pad	Expand image canvas with custom padding on each side, optional background fills (transparent, black, white), and optional Qwen-based Outpainting (outpaint:true)
POST	/images/{filename}/rotate	Rotate clockwise in 90° steps (lossless transpose, in-place)
POST	/images/{filename}/invert-alpha	Invert the alpha channel (recovers background removal errors)
POST	/images/{filename}/duplicate	Copy image into the same group with timestamp-based filename and source_refs=[original]
POST	/restore-background/{filename}	Flatten RGBA to opaque RGB via white composite background

Background Removal

Method	Path	Purpose
POST	/remove-background/{filename}	rembg BG removal, overwrites in-place; sets has_background=false + refreshes static data
POST	/remove-background-sam/{filename}	SAM2 BG removal → .nobg.png sidecar
POST	/remove-background/group/{group_id}	Queue background removal for all group members as an async background task
GET	/sam2/check	Return whether SAM2 background remover is available

Metadata, Custom Labels & Group Management

Method	Path	Purpose
GET	/images	List all images with DB metadata
GET	/images/{filename}	Serve image file
GET	/groups	List all groups with members
GET	/group-names	Return map of custom group names
POST	/group-names/{group_id}	Save custom group name
POST	/groups/merge	Merge several groups into one group
POST	/groups/extract	Extract a file into its own solo group
GET	/groups/{group_id}/order	Return order of members in a group
POST	/groups/{group_id}/order	Set custom sort order of group members
GET	/similar/{filename}	Find visually similar images using database embeddings
GET	/names	Map filename → person name
POST	/names/{filename}	Map image file to custom character name

Visibility, Archiving & Deletion

Method	Path	Purpose
POST	/images/{filename}/hidden	Toggle studio visibility (soft-hide)
POST	/images/{filename}/archive	Soft-archive image (hidden from default view)
POST	/images/{filename}/unarchive	Restore soft-archived image
DELETE	/images/{filename}	Delete image file + DB record
POST	/groups/{group_id}/archive	Soft-archive an entire group
POST	/groups/{group_id}/unarchive	Restore an archived group
DELETE	/groups/{group_id}	Delete entire group files and DB records

2D Body Pose Tools (rtmlib / RTMPose)

Method	Path	Purpose
GET	/pose/check	Report pose-estimator availability + backend (rtmlib or mediapipe)
POST	/images/{filename}/pose	Estimate COCO-17 body keypoints (caches descriptor for similar-search)
GET	/pose/similar/{filename}	Rank library images by pose similarity to this image
POST	/pose/similar	Rank library images by similarity to a supplied (edited) skeleton
POST	/pose/index	Build pose descriptors for the whole library (daemon thread)
GET	/pose/index/status	Poll pose-index build progress
POST	/pose/from-wireframe	Extract frame from a wireframe video and estimate body pose keypoints
POST	/pose/render	Render keypoints as OpenPose skeleton image (base64)
POST	/generate-with-pose	Generate image from a specific pose skeleton (avoids wireframe bleed-through)
GET	/poses	Load pose library from poses.md
POST	/poses	Create / update / rename a pose (old_name to rename) → rewrites poses.md
DELETE	/poses/{name}	Delete a pose from poses.md

3D Turntable & Orbit Previews

Method	Path	Purpose
POST	/orbit	Queue an orbit preview generation. Engine depth (fast 2.5D card parallax) or qwen (turntable)
GET	/turntable/status	Return background turntable generation state/progress summary
GET	/turntable/status/{group_id}	Return turntable state for a specific group
POST	/turntable/pause	Pause background turntable generation
POST	/turntable/resume	Resume background turntable generation
DELETE	/turntable/{group_id}	Reset turntable cache & records for group to force re-generation

Face Processing (insightface)

Method	Path	Purpose
POST	/faceswap	insightface video faceswap
GET	/faceswap/check	Report available face enhancement backends
POST	/images/{filename}/extract-face	Extract & save face crop via insightface (background task)
POST	/faces/similar	Find groups with visually similar faces using embeddings
POST	/faces/index	Build face embedding index for all preferred faces in library
GET	/faces/index/status	Poll face-indexing progress status
GET	/faces/{group_id}	Check if face crop exists for group

Videos & Wireframe Guides

Method	Path	Purpose
GET	/videos	List all available wireframe guide videos
GET	/wireframe/frame/{video_name}	Extract frame at ?t= (0–1) from wireframe video
GET	/wireframe/duration/{video_name}	Get video duration (seconds) via ffprobe
POST	/wireframe/trim	Trim a wireframe video to given start/end offsets
POST	/wireframe/frame	Extract frame at timestamp, returns base64 PNG
POST	/generate-video	Stitch list of output images into a short looping MP4 video

Database, Config & System Diagnostics

Method	Path	Purpose
GET/POST	/config	Read / write config.json
GET	/db/inconsistencies	Retrieve consistency report or run a check with ?run_now=true
POST	/db/repair	Run repair actions (restore, import file, assign group, permanently delete)
POST	/db/cleanup	Delete DB records of files that are missing on disk
GET	/health	API health check status

---

Generation Pipeline (_run_pipeline)

prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Canvas Expansion / Outpainting requested?
  │     YES → if outpaint=True, append/use natural outpainting instructions:
  │           "Naturally outpaint and extend the borders of the image to complete the scene."
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)

---

Batch Worker (_batch_worker)

Runs in a background thread pool. One thread per batch job item.

For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})

---

Camera Angle Presets

Eight absolute + four relative angle prompts defined as CAMERA_ANGLES in car.html. Each triggers a /batch request with pose=null and the angle prompt injected.

Name	Direction
Front	Straight-on frontal
¾ Left / ¾ Right	45° rotation
Side L / Side R	90° profile
Back	Rear view
High / Low	Bird's-eye / worm's-eye
↺ 45° / ↻ 45°	Relative +45° left / right
↺ 90° / ↻ 90°	Relative +90° left / right

---

Background Removal

Two strategies are available:

Configurable BG removal — config.json key "bg_removal"

Set to "rembg" (default) or "sam2".

"rembg" mode

• Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
• _apply_transparency (rembg U2Net) post-processes the result
• Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

"sam2" mode

• Replaces transparent keywords with "black background" in the Qwen prompt → Qwen renders the person against a pure black background
• Negative prompt discourages any real-background content
• _apply_transparency_sam2 (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
• Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

Explicit "black background" in prompt

• Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
• Prompt is passed to Qwen unchanged

SAM2 implementation (_apply_transparency_sam2)

• Model: sam2.1_hiera_base_plus.pt at ~/.sam/sam2.1_hiera_base_plus.pt
• Bbox [1%, 1%, 99%, 99%], multimask_output=True, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
• Also called by the explicit /remove-background-sam/{filename} endpoint

---

Face Processing (insightface)

Model: buffalo_l

Faceswap (/faceswap):

• Reads source face from reference image
• Swaps into each frame of a target video via OpenCV
• Optionally runs GFPGAN face restoration post-swap

Face extraction (_extract_face_bg background task):

• Triggered on set-preferred or explicit extract-face endpoint
• Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
• Saves as {group_id}_face.png in output dir
• DB-registered for display in the studio Info tab face-book thumbnail

---

Database Schema (person table)

Column	Type	Purpose
filename	TEXT PK	Unique image filename
filepath	TEXT	Absolute file path
group_id	TEXT	Group membership
group_name	TEXT	Human-readable group label
sort_order	INTEGER	Position within group (0 = preferred)
name	TEXT	Person / character name
prompt	TEXT	Generation prompt used
pose	TEXT	Pose tag
hidden	BOOLEAN	Hidden from studio preview
archived	BOOLEAN	Soft-deleted
has_background	BOOLEAN	Whether BG was removed
has_clothing	BOOLEAN	Content classification flag
source_refs	TEXT (JSON)	Original filenames this was derived from
content_type	TEXT	image or video
faceswap_source_video	TEXT	Source video for faceswapped clips
face_embedding	vector(512)	Face recognition embedding for visually matching character faces
is_source	BOOLEAN	Whether the file is a primary reference/source asset

---

Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

Modes:

• Gallery — grid of all groups; click to open Studio
• Studio — filmstrip + large viewer + sidebar tabs
• Database Consistency (trash.html SPA) — dedicated panel for identifying legacy/orphaned records, untracked files, and manual restore/delete of archived assets

Sidebar tabs:

Tab	Purpose
Generate	Camera angle picker, pose selector (with inline edit/add/delete → poses.md), wireframe guide, auto-growing prompt override with history autocomplete, fine-tune textareas, multi-ref thumbnails (click to deselect), batch submit
Info	Metadata, preferred toggle, face-book thumbnail, hide/archive/delete, manual crop/expand padding (copy or in-place), rotate ±90°, No-BG / Invert-α, duplicate, Pose preview (draggable + similar-pose search)
Faceswap	Source face selector, target video, preview quality toggle
Segment	SAM2 and rembg BG removal buttons
Scenery	Scenery video reference for generation

Key Studio Features

Canvas Expansion & Manual Padding (Interactive Tooling)

The Info sidebar and dedicated toolbar provide professional, reactive manual padding controls:

• Percent/Pixel Harmony: Seamlessly accepts both absolute integer pixel values (e.g., 100px) and percentage-based inputs (e.g., 10% or 0.1), ensuring perfect formatting and suffix preservation.
• Quick Adjustment Presets: Rapidly adjust margins with single-click actions for +10%, +20%, +50%, +100px, or Reset to clear all.
• Uniform Locking: Enabling the Uniform option synchronizes changes on any single padding field to all four sides simultaneously.
• Intelligent Previews: Dynamically renders real-time color-coded canvas border previews based on the selected fill type (Amber dashed for transparent, translucent black for black, and translucent white for white).
• Shortcuts: Confirm cropping or padding instantly with Enter, or dismiss/cancel the operation using Escape.

Reactive Layout Optimization

To prevent desyncs during image transitions, crops, canvas expansion, or 2.2x zooming:

• A ResizeObserver is registered on the primary studio viewer image (lbImg).
• Automatically triggers a layout and geometry recalculation, cleanly and immediately adjusting the background transparency checkerboard grid and padding visual overlays to match the actual visual boundaries of the contained image.

Key state variables:

• lbCurrentGid — active group ID
• lbIdx — current filmstrip position
• _followLatestGid — auto-follow newly generated image
• _sbSelectedAngles — selected camera angle names
• _sbWireframeRef / _sbWireframeTime — wireframe video + frame time
• privacyMode — privacy overlay toggle (key: P)

---

Wireframe Pose Guide

Wireframe videos stored in /mnt/zim/tour-comfy/wireframe/.
The studio Generate tab shows a video picker + time scrubber. On generation:

1. Backend calls GET /wireframe/frame/{name}?t=0.5 to extract the chosen frame
2. Frame passed as image2 in the ComfyUI workflow (second visual reference)
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via image2 slot

---

Pose Tools (2D body pose)

A posenet-style toolset built on a feature-detected estimator. Preferred backend is rtmlib (RTMPose, ONNX via the already-installed onnxruntime); mediapipe is a fallback. If neither is installed the endpoints return 501 and the UI shows an install hint.

• Install (A6000 venv): ~/comfyui/venv/bin/pip install rtmlib. Models (~150 MB) auto- download to ~/.cache/rtmlib on first use.
• CUDA libs: onnxruntime-gpu needs the venv's nvidia-*-cu12 libs on the loader path. a6000-comfy/start_api.sh adds them to LD_LIBRARY_PATH at launch (torch finds them via RPATH, onnxruntime does not).

Output format: COCO-17 keypoints (POSE_KEYPOINT_NAMES) + POSE_SKELETON edge list, returned in image pixels with per-joint score.

Studio overlay (car.html): the Pose button overlays the skeleton on the viewer (letterbox-aware canvas). Joints are draggable to refine/explore a pose. Overlay toolbar: Reset (re-detect) and Similar pose (rank the library by the current — possibly edited — skeleton). Results render as a thumbnail strip; clicking one opens that image's group.

Similarity model (edit_api.py):

• _pose_descriptor(keypoints) → translation/scale-invariant vector: centered on hip midpoint (fallback shoulders), scaled by torso length, 17×(x,y) + visibility mask. None if < 6 joints.
• _pose_distance(a, b) → weighted L2 over jointly-visible joints, taking the min of the direct and the left-right-mirrored comparison so mirrored poses match.
• Pose index: descriptors are cached in <output>/_data/poses_index.json. It fills incrementally whenever /images/{f}/pose runs, or in bulk via /pose/index (daemon thread, batched writes every 50, progress via /pose/index/status).

File Layout

qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA (Studio Interface)
│   ├── trash.html        # frontend Database Consistency Panel SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file (System Documentation)
├── ARCHETECTURE.svg      # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps

---

Configuration (config.json)

Key	Default	Purpose
output_dir	../output	Where generated PNGs are saved
comfy_url	http://127.0.0.1:8188	ComfyUI base URL
host	0.0.0.0	FastAPI bind host
port	8500	FastAPI bind port
max_area	1048576	Max pixel budget (1 MP)

---

Background Services

Service	Script	Purpose
comfyui-backend	a6000-comfy/run_comfyui.sh	Qwen GGUF inference
comfyui-api	tour-comfy/start_api.sh	FastAPI HTTP server
watcher	tour-comfy/start_watcher.sh	Monitors output dir, auto-processes dropped images