qwen-image/AGENTS.md

Qwen-Image-Edit Rapid-AIO v23 — System Documentation

Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management. It runs the Qwen 2.5 VL GGUF model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)

---

Hardware

Component	Spec
GPU	NVIDIA RTX A6000 (48 GB VRAM)
CUDA	12.4
PyTorch	2.6.0+cu124
ComfyUI	latest pinned (a6000-comfy/)

---

Services

1. ComfyUI Backend — :8188

• Script: a6000-comfy/run_comfyui.sh
• Systemd: comfyui-backend
• Purpose: Executes the Qwen GGUF model via workflow_qwen_edit.json

Loaded models

Role	File
GGUF LLM	Qwen-Rapid-NSFW-v23_Q8_0.gguf
Text encoder	qwen_2.5_vl_7b_fp8_scaled.safetensors
VAE	qwen_image_vae.safetensors

Workflow nodes (workflow_qwen_edit.json):

• NODE_POSITIVE — positive text prompt
• NODE_NEGATIVE — negative prompt
• NODE_LATENT — latent dimensions (width × height)
• NODE_KSAMPLER — sampler params (Euler/karras, seed, steps, CFG)
• image1 — reference image (base64 uploaded to ComfyUI /upload/image)
• image2 — optional wireframe pose guide (extracted from wireframe video)

---

2. FastAPI API — :8500

• Script: tour-comfy/edit_api.py
• Systemd: comfyui-api
• DB: PostgreSQL at 192.168.1.160:5433, database dv, table person

In-Memory File Metadata Cache

To mitigate performance and network latency bottlenecks on remote/shared storage filesystems (e.g. /mnt/zim), the API server employs a light, thread-safe in-memory metadata cache (_file_meta_cache):

• Cached Items: File existence (exists) and modification timestamps (mtime) mapped by filename.
• Cache TTL: 5.0 seconds.
• Idempotency & Invalidation: Any mutation endpoints (such as crop, padding, duplication, background removal, or deletion) automatically update (_update_cached_file_meta) or explicitly delete (_clear_cached_file_meta) entries to keep cache and disk perfectly in sync.
• Preloading Optimization: Leverages cache queries in static indexing loops (_sync_preloaded_images, _write_all_static, list_images) and tracks changes to completely skip redundant static HTML rewrites if the preloaded image set is identical (_last_preloaded_images_set).

Endpoints Library

Core Generation & Upload

Method	Path	Purpose
POST	/upload	Upload image; optionally add to existing group (group_id + skip_poses)
POST	/batch	Queue multi-prompt generation job (async)
GET	/batch/{job_id}	Poll batch job status / per-item progress
DELETE	/batch/{job_id}	Cancel active batch job
POST	/multi-ref	Queue a generation job with multiple reference images
POST	/refine-prompt	Refine a generation prompt using the external uncensored chat LLM
POST	/edit	Edit uploaded image
POST	/generate-scenery	Generate scenery background based on video frame + reference image
GET	/scenery/library	Return all scenery images grouped by source video reference

Image Editing & Canvas Expansion (Padding & Crop)

Method	Path	Purpose
POST	/images/{filename}/crop	Crop to pixel rect; as_copy:true crops a referenced copy instead of modifying in-place
POST	/images/{filename}/autocrop	Auto-trim transparent borders in-place
POST	/images/{filename}/pad	Expand image canvas with custom padding on each side, optional background fills (transparent, black, white), and optional Qwen-based Outpainting (outpaint:true)
POST	/images/{filename}/rotate	Rotate clockwise in 90° steps (lossless transpose, in-place)
POST	/images/{filename}/invert-alpha	Invert the alpha channel (recovers background removal errors)
POST	/images/{filename}/duplicate	Copy image into the same group with timestamp-based filename and source_refs=[original]
POST	/restore-background/{filename}	Flatten RGBA to opaque RGB via white composite background

Background Removal

Method	Path	Purpose
POST	/remove-background/{filename}	rembg BG removal, overwrites in-place; sets has_background=false + refreshes static data
POST	/remove-background-sam/{filename}	SAM2 BG removal → .nobg.png sidecar
POST	/remove-background/group/{group_id}	Queue background removal for all group members as an async background task
GET	/sam2/check	Return whether SAM2 background remover is available

Metadata, Custom Labels & Group Management

Method	Path	Purpose
GET	/images	List all images with DB metadata
GET	/images/{filename}	Serve image file
GET	/groups	List all groups with members
GET	/group-names	Return map of custom group names
POST	/group-names/{group_id}	Save custom group name
POST	/groups/merge	Merge several groups into one group
POST	/groups/extract	Extract a file into its own solo group
GET	/groups/{group_id}/order	Return order of members in a group
POST	/groups/{group_id}/order	Set custom sort order of group members
GET	/similar/{filename}	Find visually similar images using database embeddings
GET	/names	Map filename → person name
POST	/names/{filename}	Map image file to custom character name
POST	/images/{filename}/tags	Set image tags list (supports tags like ARCHIVED, HIDDEN, SOURCE, VISIBLE, LIKE, DISLIKE, 21+)
POST	/images/{filename}/tag-action	Add, remove, or toggle a specific tag on an image (automatically handles mutual exclusivity of LIKE/DISLIKE)
POST	/images/{filename}/source	Explicitly toggle marking an image as a source asset (sets/clears is_source and toggles the SOURCE tag)
POST	/images/{filename}/estimate-21plus	Analyze image with the WD tagger to estimate adult/sensitive rating and apply 21+ tag
POST	/images/bulk-move	Move a list of filenames to a specific target filmstrip (VISIBLE, HIDDEN, ARCHIVED, SOURCE) in bulk, updating tags/columns
POST	/images/{filename}/undress	Queue an asynchronous batch job utilizing the UNDRESS_PROMPT on the given reference image

Visibility, Archiving & Deletion

Method	Path	Purpose
POST	/images/{filename}/hidden	Toggle studio visibility (soft-hide)
POST	/images/{filename}/archive	Soft-archive image (hidden from default view)
POST	/images/{filename}/unarchive	Restore soft-archived image
DELETE	/images/{filename}	Delete image file + DB record
POST	/groups/{group_id}/archive	Soft-archive an entire group
POST	/groups/{group_id}/unarchive	Restore an archived group
DELETE	/groups/{group_id}	Delete entire group files and DB records

2D Body Pose Tools (rtmlib / RTMPose)

Method	Path	Purpose
GET	/pose/check	Report pose-estimator availability + backend (rtmlib or mediapipe)
POST	/images/{filename}/pose	Estimate COCO-17 body keypoints (caches descriptor for similar-search)
GET	/pose/similar/{filename}	Rank library images by pose similarity to this image
POST	/pose/similar	Rank library images by similarity to a supplied (edited) skeleton
POST	/pose/index	Build pose descriptors for the whole library (daemon thread)
GET	/pose/index/status	Poll pose-index build progress
POST	/pose/from-wireframe	Extract frame from a wireframe video and estimate body pose keypoints
POST	/pose/render	Render keypoints as OpenPose skeleton image (base64)
POST	/generate-with-pose	Generate image from a specific pose skeleton (avoids wireframe bleed-through)
GET	/poses	Load pose library from poses.md
POST	/poses	Create / update / rename a pose (old_name to rename) → rewrites poses.md
DELETE	/poses/{name}	Delete a pose from poses.md

3D Turntable & Orbit Previews

Method	Path	Purpose
POST	/orbit	Queue an orbit preview generation. Engine depth (fast 2.5D card parallax) or qwen (turntable)
GET	/turntable/status	Return background turntable generation state/progress summary
GET	/turntable/status/{group_id}	Return turntable state for a specific group
POST	/turntable/pause	Pause background turntable generation
POST	/turntable/resume	Resume background turntable generation
DELETE	/turntable/{group_id}	Reset turntable cache & records for group to force re-generation

Face Processing (insightface)

Method	Path	Purpose
POST	/faceswap	insightface video faceswap
GET	/faceswap/check	Report available face enhancement backends
POST	/images/{filename}/extract-face	Extract & save face crop via insightface (background task)
POST	/faces/similar	Find groups with visually similar faces using embeddings
POST	/faces/index	Build face embedding index for all preferred faces in library
GET	/faces/index/status	Poll face-indexing progress status
GET	/faces/{group_id}	Check if face crop exists for group

Videos & Wireframe Guides

Method	Path	Purpose
GET	/videos	List all available wireframe guide videos
GET	/wireframe/frame/{video_name}	Extract frame at ?t= (0–1) from wireframe video
GET	/wireframe/duration/{video_name}	Get video duration (seconds) via ffprobe
POST	/wireframe/trim	Trim a wireframe video to given start/end offsets
POST	/wireframe/frame	Extract frame at timestamp, returns base64 PNG
POST	/generate-video	Stitch list of output images into a short looping MP4 video

Database, Config & System Diagnostics

Method	Path	Purpose
GET/POST	/config	Read / write config.json
GET	/db/inconsistencies	Retrieve consistency report or run a check with ?run_now=true
POST	/db/repair	Run repair actions (restore, import file, assign group, permanently delete)
POST	/db/cleanup	Delete DB records of files that are missing on disk
GET	/health	API health check status

---

Generation Pipeline (_run_pipeline)

prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Canvas Expansion / Outpainting requested?
  │     YES → if outpaint=True:
  │             ├─ If source is transparent (RGBA): composite onto flat black background for maximum contrast.
  │             ├─ Formulate dynamic prompt description: e.g. "replacing the black/white background areas"
  │             └─ Append outpainting prompt instructions: "Naturally outpaint and extend..."
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image (composited RGB if outpainting transparent images)
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)

---

Batch Worker (_batch_worker)

Runs in a background thread pool. One thread per batch job item.

For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})

---

Camera Angle Presets

Eight absolute + four relative angle prompts defined as CAMERA_ANGLES in car.html. Each triggers a /batch request with pose=null and the angle prompt injected.

Name	Direction
Front	Straight-on frontal
¾ Left / ¾ Right	45° rotation
Side L / Side R	90° profile
Back	Rear view
High / Low	Bird's-eye / worm's-eye
↺ 45° / ↻ 45°	Relative +45° left / right
↺ 90° / ↻ 90°	Relative +90° left / right

---

Background Removal

Two strategies are available:

Configurable BG removal — config.json key "bg_removal"

Set to "rembg" (default) or "sam2".

"rembg" mode

• Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
• _apply_transparency (rembg U2Net) post-processes the result
• Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

"sam2" mode

• Replaces transparent keywords with "black background" in the Qwen prompt → Qwen renders the person against a pure black background
• Negative prompt discourages any real-background content
• _apply_transparency_sam2 (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
• Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

Explicit "black background" in prompt

• Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
• Prompt is passed to Qwen unchanged

SAM2 implementation (_apply_transparency_sam2)

• Model: sam2.1_hiera_base_plus.pt at ~/.sam/sam2.1_hiera_base_plus.pt
• Bbox [1%, 1%, 99%, 99%], multimask_output=True, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
• Also called by the explicit /remove-background-sam/{filename} endpoint

---

Face Processing (insightface)

Model: buffalo_l

Faceswap (/faceswap):

• Reads source face from reference image
• Swaps into each frame of a target video via OpenCV
• Optionally runs GFPGAN face restoration post-swap

Face extraction (_extract_face_bg background task):

• Triggered on set-preferred or explicit extract-face endpoint
• Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
• Saves as {group_id}_face.png in output dir
• DB-registered for display in the studio Info tab face-book thumbnail

---

Database Schema (person table)

Column	Type	Purpose
filename	TEXT PK	Unique image filename
filepath	TEXT	Absolute file path
group_id	TEXT	Group membership
group_name	TEXT	Human-readable group label
sort_order	INTEGER	Position within group (0 = preferred)
name	TEXT	Person / character name
prompt	TEXT	Generation prompt used
pose	TEXT	Pose tag
hidden	BOOLEAN	Hidden from studio preview
archived	BOOLEAN	Soft-deleted
has_background	BOOLEAN	Whether BG was removed
has_clothing	BOOLEAN	Content classification flag
source_refs	TEXT (JSON)	Original filenames this was derived from
content_type	TEXT	image or video
faceswap_source_video	TEXT	Source video for faceswapped clips
face_embedding	vector(512)	Face recognition embedding for visually matching character faces
is_source	BOOLEAN	Whether the file is a primary reference/source asset
tags	TEXT (JSON)	Custom user-applied tags/labels (e.g., ['VISIBLE', 'LIKE', '21+'])
pose_description	TEXT	AI-generated descriptive text of the body pose
pose_skeleton	TEXT	Coordinates and scoring details of the 2D COCO-17 body pose skeleton

---

Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

Modes:

• Gallery — grid of all groups; click to open Studio
• Studio — filmstrip + large viewer + sidebar tabs
• Database Consistency (trash.html SPA) — dedicated panel for identifying legacy/orphaned records, untracked files, and manual restore/delete of archived assets

Sidebar tabs:

Tab	Purpose
Generate	Camera angle picker, pose selector (with inline edit/add/delete → poses.md), wireframe guide, auto-growing prompt override with history autocomplete, fine-tune textareas, multi-ref thumbnails (click to deselect), batch submit
Info	Metadata, preferred toggle, face-book thumbnail, hide/archive/delete, manual crop/expand padding (copy or in-place), rotate ±90°, No-BG / Invert-α, duplicate, Pose preview (draggable + similar-pose search)
Faceswap	Source face selector, target video, preview quality toggle
Segment	SAM2 and rembg BG removal buttons
Scenery	Scenery video reference for generation

Key Studio Features

Canvas Expansion & Manual Padding (Interactive Tooling)

The Info sidebar and dedicated toolbar provide professional, reactive manual padding controls:

• Percent/Pixel Harmony: Seamlessly accepts both absolute integer pixel values (e.g., 100px) and percentage-based inputs (e.g., 10% or 0.1), ensuring perfect formatting and suffix preservation.
• Quick Adjustment Presets: Rapidly adjust margins with single-click actions for +10%, +20%, +50%, +100px, or Reset to clear all.
• Uniform Locking: Enabling the Uniform option synchronizes changes on any single padding field to all four sides simultaneously.
• Intelligent Previews: Dynamically renders real-time color-coded canvas border previews based on the selected fill type (Amber dashed for transparent, translucent black for black, and translucent white for white).
• Shortcuts: Confirm cropping or padding instantly with Enter, or dismiss/cancel the operation using Escape.

Reactive Layout Optimization

To prevent desyncs during image transitions, crops, canvas expansion, or 2.2x zooming:

• A ResizeObserver is registered on the primary studio viewer image (lbImg).
• Automatically triggers a layout and geometry recalculation, cleanly and immediately adjusting the background transparency checkerboard grid and padding visual overlays to match the actual visual boundaries of the contained image.

Multiple Filmstrips System

The filmstrip bar has been updated with five tabs that filter the group's images dynamically:

• Group Active: The default filmstrip containing the main/preferred assets of the group.
• Group Variant (VISIBLE): Shows all standard visible variants within the group.
• Hidden (HIDDEN): Soft-hidden variants (excluded from main cycling).
• Source (SOURCE): Displays reference images explicitly marked as source assets.
• Archived (ARCHIVED): Displays soft-archived variants.
• Drag-and-Drop Reclassification: Users can drag any variant thumbnail from the filmstrip and drop it directly onto any of the tab buttons to instantly update its tag/status and reassign it to that list.

Bulk Multi-Select & Actions

Toggling the Multi-select button unlocks checkbox-based selection across the filmstrip:

• Provides a live selection counter.
• Enables bulk move/reclassification buttons to dispatch batch updates to /images/bulk-move (e.g., Move to Group, Move to Hidden, Move to Source, Move to Archived).

Seamless Non-Blinking State Updates

To prevent irritating browser reload blinks or blank screen latency during intensive image editing operations:

• Local Metadata Cloning: Duplication, manual cropping, padding, and background removal immediately clone the original's DB metadata values locally on the client and splice the new file record into the active filmstrip list.
• Optimistic Merge: During periodic background server syncs (refreshNow), an optimistic merge algorithm compares the local client-side list and the server JSON. It preserves any freshly created local assets that haven't yet been processed/synced on the backend, maintaining perfect continuity.

Studio Keyboard Shortcuts Reference

Key	Action	Description
P	Toggle Privacy Mode	Replaces the page with a discrete "AI Assist" study-chat and disguises taskbar thumbnails.
C	Toggle Checkerboard	Toggle transparent background checkerboard grid.
A	Expand/Pad Canvas	Instantly trigger the manual padding toolbar.
G / Alt+Enter	Trigger Generation	Clicks the main "Generate" button (when Generate tab is active).
ArrowLeft / ArrowRight	Nav Filmstrip	Cycle through the current list of image variants (debounced).
Home / End	Jump to Bounds	Instantly jump to the first or last variant in the filmstrip.
Space	Play/Pause Video	Pauses or plays a running wireframe/clip inside the viewer.
H	Toggle Hidden	Reclasses the image between standard variants and hidden list.
F	Set as Preferred	Reorders the group, placing the current image at index 0 (as preferred thumbnail).
S	Toggle Source	Mark/unmark current image as a primary reference/source asset.
E	Run 21+ Tagger	Analyze image via WD tagger and apply/remove the 21+ tag.
Delete	Soft-Archive	Move the current image into the Archive filmstrip.
ArrowUp / ArrowDown	Shift Order	Move/reorder the current image earlier or later in the group (debounced).
2	Move to Second	Instantly reorder current image to the 2nd position in the group.
9	Move to Last-1	Instantly reorder current image to the second to last position.
Escape	Cancel / Dismiss	Cancel cropping, padding, close variant picker, or exit selection mode.

Key state variables:

• lbCurrentGid — active group ID
• lbIdx — current filmstrip position
• _followLatestGid — auto-follow newly generated image
• _followLatestFilename — auto-follow specific new output file
• _sbSelectedAngles — selected camera angle names
• _sbWireframeRef / _sbWireframeTime — wireframe video + frame time
• privacyMode — privacy overlay toggle (key: P)

---

Wireframe Pose Guide

Wireframe videos stored in /mnt/zim/tour-comfy/wireframe/.
The studio Generate tab shows a video picker + time scrubber. On generation:

1. Backend calls GET /wireframe/frame/{name}?t=0.5 to extract the chosen frame
2. Frame passed as image2 in the ComfyUI workflow (second visual reference)
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via image2 slot

---

Pose Tools (2D body pose)

A posenet-style toolset built on a feature-detected estimator. Preferred backend is rtmlib (RTMPose, ONNX via the already-installed onnxruntime); mediapipe is a fallback. If neither is installed the endpoints return 501 and the UI shows an install hint.

• Install (A6000 venv): ~/comfyui/venv/bin/pip install rtmlib. Models (~150 MB) auto- download to ~/.cache/rtmlib on first use.
• CUDA libs: onnxruntime-gpu needs the venv's nvidia-*-cu12 libs on the loader path. a6000-comfy/start_api.sh adds them to LD_LIBRARY_PATH at launch (torch finds them via RPATH, onnxruntime does not).

Output format: COCO-17 keypoints (POSE_KEYPOINT_NAMES) + POSE_SKELETON edge list, returned in image pixels with per-joint score.

Studio overlay (car.html): the Pose button overlays the skeleton on the viewer (letterbox-aware canvas). Joints are draggable to refine/explore a pose. Overlay toolbar: Reset (re-detect) and Similar pose (rank the library by the current — possibly edited — skeleton). Results render as a thumbnail strip; clicking one opens that image's group.

Similarity model (edit_api.py):

• _pose_descriptor(keypoints) → translation/scale-invariant vector: centered on hip midpoint (fallback shoulders), scaled by torso length, 17×(x,y) + visibility mask. None if < 6 joints.
• _pose_distance(a, b) → weighted L2 over jointly-visible joints, taking the min of the direct and the left-right-mirrored comparison so mirrored poses match.
• Pose index: descriptors are cached in <output>/_data/poses_index.json. It fills incrementally whenever /images/{f}/pose runs, or in bulk via /pose/index (daemon thread, batched writes every 50, progress via /pose/index/status).

File Layout

qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA (Studio Interface)
│   ├── trash.html        # frontend Database Consistency Panel SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file (System Documentation)
├── ARCHETECTURE.svg      # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps

---

Configuration (config.json)

Key	Default	Purpose
output_dir	../output	Where generated PNGs are saved
comfy_url	http://127.0.0.1:8188	ComfyUI base URL
host	0.0.0.0	FastAPI bind host
port	8500	FastAPI bind port
max_area	1048576	Max pixel budget (1 MP)

---

Background Services

Service	Script	Purpose
comfyui-backend	a6000-comfy/run_comfyui.sh	Qwen GGUF inference
comfyui-api	tour-comfy/start_api.sh	FastAPI HTTP server
watcher	tour-comfy/start_watcher.sh	Monitors output dir, auto-processes dropped images