qwen-image/AGENTS.md

# Qwen-Image-Edit Rapid-AIO v23 — System Documentation

## Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management.  It runs the **Qwen 2.5 VL GGUF** model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

```
Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)
```

---

## Hardware

| Component | Spec |
|-----------|------|
| GPU | NVIDIA RTX A6000 (48 GB VRAM) |
| CUDA | 12.4 |
| PyTorch | 2.6.0+cu124 |
| ComfyUI | latest pinned (a6000-comfy/) |

---

## Services

### 1. ComfyUI Backend — `:8188`

- **Script**: `a6000-comfy/run_comfyui.sh`
- **Systemd**: `comfyui-backend`
- **Purpose**: Executes the Qwen GGUF model via `workflow_qwen_edit.json`

**Loaded models**

| Role | File |
|------|------|
| GGUF LLM | `Qwen-Rapid-NSFW-v23_Q8_0.gguf` |
| Text encoder | `qwen_2.5_vl_7b_fp8_scaled.safetensors` |
| VAE | `qwen_image_vae.safetensors` |

**Workflow nodes** (`workflow_qwen_edit.json`):
- `NODE_POSITIVE` — positive text prompt
- `NODE_NEGATIVE` — negative prompt
- `NODE_LATENT` — latent dimensions (width × height)
- `NODE_KSAMPLER` — sampler params (Euler/karras, seed, steps, CFG)
- `image1` — reference image (base64 uploaded to ComfyUI `/upload/image`)
- `image2` — optional wireframe pose guide (extracted from wireframe video)

---

### 2. FastAPI API — `:8500`

- **Script**: `tour-comfy/edit_api.py`
- **Systemd**: `comfyui-api`
- **DB**: PostgreSQL at `192.168.1.160:5433`, database `dv`, table `person`

#### Endpoints Library

##### Core Generation & Upload
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/upload` | Upload image; optionally add to existing group (`group_id` + `skip_poses`) |
| `POST` | `/batch` | Queue multi-prompt generation job (async) |
| `GET` | `/batch/{job_id}` | Poll batch job status / per-item progress |
| `DELETE` | `/batch/{job_id}` | Cancel active batch job |
| `POST` | `/multi-ref` | Queue a generation job with multiple reference images |
| `POST` | `/refine-prompt` | Refine a generation prompt using the external uncensored chat LLM |
| `POST` | `/edit` | Edit uploaded image |
| `POST` | `/generate-scenery` | Generate scenery background based on video frame + reference image |
| `GET` | `/scenery/library` | Return all scenery images grouped by source video reference |

##### Image Editing & Canvas Expansion (Padding & Crop)
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/images/{filename}/crop` | Crop to pixel rect; `as_copy:true` crops a referenced copy instead of modifying in-place |
| `POST` | `/images/{filename}/autocrop` | Auto-trim transparent borders in-place |
| `POST` | `/images/{filename}/pad` | Expand image canvas with custom padding on each side, optional background fills (`transparent`, `black`, `white`), and optional Qwen-based Outpainting (`outpaint:true`) |
| `POST` | `/images/{filename}/rotate` | Rotate clockwise in 90° steps (lossless transpose, in-place) |
| `POST` | `/images/{filename}/invert-alpha` | Invert the alpha channel (recovers background removal errors) |
| `POST` | `/images/{filename}/duplicate` | Copy image into the same group with timestamp-based filename and `source_refs=[original]` |
| `POST` | `/restore-background/{filename}` | Flatten RGBA to opaque RGB via white composite background |

##### Background Removal
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/remove-background/{filename}` | rembg BG removal, overwrites in-place; sets `has_background=false` + refreshes static data |
| `POST` | `/remove-background-sam/{filename}` | SAM2 BG removal → `.nobg.png` sidecar |
| `POST` | `/remove-background/group/{group_id}` | Queue background removal for all group members as an async background task |
| `GET` | `/sam2/check` | Return whether SAM2 background remover is available |

##### Metadata, Custom Labels & Group Management
| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/images` | List all images with DB metadata |
| `GET` | `/images/{filename}` | Serve image file |
| `GET` | `/groups` | List all groups with members |
| `GET` | `/group-names` | Return map of custom group names |
| `POST` | `/group-names/{group_id}` | Save custom group name |
| `POST` | `/groups/merge` | Merge several groups into one group |
| `POST` | `/groups/extract` | Extract a file into its own solo group |
| `GET` | `/groups/{group_id}/order` | Return order of members in a group |
| `POST` | `/groups/{group_id}/order` | Set custom sort order of group members |
| `GET` | `/similar/{filename}` | Find visually similar images using database embeddings |
| `GET` | `/names` | Map `filename → person name` |
| `POST` | `/names/{filename}` | Map image file to custom character name |

##### Visibility, Archiving & Deletion
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/images/{filename}/hidden` | Toggle studio visibility (soft-hide) |
| `POST` | `/images/{filename}/archive` | Soft-archive image (hidden from default view) |
| `POST` | `/images/{filename}/unarchive` | Restore soft-archived image |
| `DELETE` | `/images/{filename}` | Delete image file + DB record |
| `POST` | `/groups/{group_id}/archive` | Soft-archive an entire group |
| `POST` | `/groups/{group_id}/unarchive` | Restore an archived group |
| `DELETE` | `/groups/{group_id}` | Delete entire group files and DB records |

##### 2D Body Pose Tools (rtmlib / RTMPose)
| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/pose/check` | Report pose-estimator availability + backend (rtmlib or mediapipe) |
| `POST` | `/images/{filename}/pose` | Estimate COCO-17 body keypoints (caches descriptor for similar-search) |
| `GET` | `/pose/similar/{filename}` | Rank library images by pose similarity to this image |
| `POST` | `/pose/similar` | Rank library images by similarity to a supplied (edited) skeleton |
| `POST` | `/pose/index` | Build pose descriptors for the whole library (daemon thread) |
| `GET` | `/pose/index/status` | Poll pose-index build progress |
| `POST` | `/pose/from-wireframe` | Extract frame from a wireframe video and estimate body pose keypoints |
| `POST` | `/pose/render` | Render keypoints as OpenPose skeleton image (base64) |
| `POST` | `/generate-with-pose` | Generate image from a specific pose skeleton (avoids wireframe bleed-through) |
| `GET` | `/poses` | Load pose library from `poses.md` |
| `POST` | `/poses` | Create / update / rename a pose (`old_name` to rename) → rewrites `poses.md` |
| `DELETE` | `/poses/{name}` | Delete a pose from `poses.md` |

##### 3D Turntable & Orbit Previews
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/orbit` | Queue an orbit preview generation. Engine `depth` (fast 2.5D card parallax) or `qwen` (turntable) |
| `GET` | `/turntable/status` | Return background turntable generation state/progress summary |
| `GET` | `/turntable/status/{group_id}` | Return turntable state for a specific group |
| `POST` | `/turntable/pause` | Pause background turntable generation |
| `POST` | `/turntable/resume` | Resume background turntable generation |
| `DELETE` | `/turntable/{group_id}` | Reset turntable cache & records for group to force re-generation |

##### Face Processing (insightface)
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/faceswap` | insightface video faceswap |
| `GET` | `/faceswap/check` | Report available face enhancement backends |
| `POST` | `/images/{filename}/extract-face` | Extract & save face crop via insightface (background task) |
| `POST` | `/faces/similar` | Find groups with visually similar faces using embeddings |
| `POST` | `/faces/index` | Build face embedding index for all preferred faces in library |
| `GET` | `/faces/index/status` | Poll face-indexing progress status |
| `GET` | `/faces/{group_id}` | Check if face crop exists for group |

##### Videos & Wireframe Guides
| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/videos` | List all available wireframe guide videos |
| `GET` | `/wireframe/frame/{video_name}` | Extract frame at `?t=` (0–1) from wireframe video |
| `GET` | `/wireframe/duration/{video_name}` | Get video duration (seconds) via ffprobe |
| `POST` | `/wireframe/trim` | Trim a wireframe video to given start/end offsets |
| `POST` | `/wireframe/frame` | Extract frame at timestamp, returns base64 PNG |
| `POST` | `/generate-video` | Stitch list of output images into a short looping MP4 video |

##### Database, Config & System Diagnostics
| Method | Path | Purpose |
|--------|------|---------|
| `GET/POST` | `/config` | Read / write `config.json` |
| `GET` | `/db/inconsistencies` | Retrieve consistency report or run a check with `?run_now=true` |
| `POST` | `/db/repair` | Run repair actions (restore, import file, assign group, permanently delete) |
| `POST` | `/db/cleanup` | Delete DB records of files that are missing on disk |
| `GET` | `/health` | API health check status |

---

## Generation Pipeline (`_run_pipeline`)

```
prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Canvas Expansion / Outpainting requested?
  │     YES → if outpaint=True, append/use natural outpainting instructions:
  │           "Naturally outpaint and extend the borders of the image to complete the scene."
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)
```

---

## Batch Worker (`_batch_worker`)

Runs in a background thread pool. One thread per batch job item.

```
For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})
```

---

## Camera Angle Presets

Eight absolute + four relative angle prompts defined as `CAMERA_ANGLES` in car.html.  Each triggers a `/batch` request with `pose=null` and the angle prompt injected.

| Name | Direction |
|------|-----------|
| Front | Straight-on frontal |
| ¾ Left / ¾ Right | 45° rotation |
| Side L / Side R | 90° profile |
| Back | Rear view |
| High / Low | Bird's-eye / worm's-eye |
| ↺ 45° / ↻ 45° | Relative +45° left / right |
| ↺ 90° / ↻ 90° | Relative +90° left / right |

---

## Background Removal

Two strategies are available:

### Configurable BG removal — `config.json` key `"bg_removal"`

Set to `"rembg"` (default) or `"sam2"`.

#### `"rembg"` mode
- Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
- `_apply_transparency` (rembg U2Net) post-processes the result
- Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

#### `"sam2"` mode
- Replaces transparent keywords with `"black background"` in the Qwen prompt → Qwen renders the person against a pure black background
- Negative prompt discourages any real-background content
- `_apply_transparency_sam2` (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
- Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

#### Explicit `"black background"` in prompt
- Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
- Prompt is passed to Qwen unchanged

#### SAM2 implementation (`_apply_transparency_sam2`)
- Model: `sam2.1_hiera_base_plus.pt` at `~/.sam/sam2.1_hiera_base_plus.pt`
- Bbox `[1%, 1%, 99%, 99%]`, `multimask_output=True`, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
- Also called by the explicit `/remove-background-sam/{filename}` endpoint

---

## Face Processing (insightface)

Model: `buffalo_l`

**Faceswap** (`/faceswap`):
- Reads source face from reference image
- Swaps into each frame of a target video via OpenCV
- Optionally runs GFPGAN face restoration post-swap

**Face extraction** (`_extract_face_bg` background task):
- Triggered on `set-preferred` or explicit `extract-face` endpoint
- Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
- Saves as `{group_id}_face.png` in output dir
- DB-registered for display in the studio Info tab face-book thumbnail

---

## Database Schema (`person` table)

| Column | Type | Purpose |
|--------|------|---------|
| filename | TEXT PK | Unique image filename |
| filepath | TEXT | Absolute file path |
| group_id | TEXT | Group membership |
| group_name | TEXT | Human-readable group label |
| sort_order | INTEGER | Position within group (0 = preferred) |
| name | TEXT | Person / character name |
| prompt | TEXT | Generation prompt used |
| pose | TEXT | Pose tag |
| hidden | BOOLEAN | Hidden from studio preview |
| archived | BOOLEAN | Soft-deleted |
| has_background | BOOLEAN | Whether BG was removed |
| has_clothing | BOOLEAN | Content classification flag |
| source_refs | TEXT (JSON) | Original filenames this was derived from |
| content_type | TEXT | `image` or `video` |
| faceswap_source_video | TEXT | Source video for faceswapped clips |
| face_embedding | `vector(512)` | Face recognition embedding for visually matching character faces |
| is_source | `BOOLEAN` | Whether the file is a primary reference/source asset |

---

## Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

**Modes**:
- **Gallery** — grid of all groups; click to open Studio
- **Studio** — filmstrip + large viewer + sidebar tabs
- **Database Consistency** (`trash.html` SPA) — dedicated panel for identifying legacy/orphaned records, untracked files, and manual restore/delete of archived assets

**Sidebar tabs**:

| Tab | Purpose |
|-----|---------|
| Generate | Camera angle picker, pose selector (with inline edit/add/delete → `poses.md`), wireframe guide, auto-growing prompt override with history autocomplete, fine-tune textareas, multi-ref thumbnails (click to deselect), batch submit |
| Info | Metadata, preferred toggle, face-book thumbnail, hide/archive/delete, manual crop/expand padding (copy or in-place), rotate ±90°, No-BG / Invert-α, duplicate, Pose preview (draggable + similar-pose search) |
| Faceswap | Source face selector, target video, preview quality toggle |
| Segment | SAM2 and rembg BG removal buttons |
| Scenery | Scenery video reference for generation |

### Key Studio Features

#### Canvas Expansion & Manual Padding (Interactive Tooling)
The **Info** sidebar and dedicated toolbar provide professional, reactive manual padding controls:
- **Percent/Pixel Harmony**: Seamlessly accepts both absolute integer pixel values (e.g., `100px`) and percentage-based inputs (e.g., `10%` or `0.1`), ensuring perfect formatting and suffix preservation.
- **Quick Adjustment Presets**: Rapidly adjust margins with single-click actions for `+10%`, `+20%`, `+50%`, `+100px`, or `Reset` to clear all.
- **Uniform Locking**: Enabling the **Uniform** option synchronizes changes on any single padding field to all four sides simultaneously.
- **Intelligent Previews**: Dynamically renders real-time color-coded canvas border previews based on the selected fill type (Amber dashed for `transparent`, translucent black for `black`, and translucent white for `white`).
- **Shortcuts**: Confirm cropping or padding instantly with `Enter`, or dismiss/cancel the operation using `Escape`.

#### Reactive Layout Optimization
To prevent desyncs during image transitions, crops, canvas expansion, or 2.2x zooming:
- A `ResizeObserver` is registered on the primary studio viewer image (`lbImg`).
- Automatically triggers a layout and geometry recalculation, cleanly and immediately adjusting the background transparency checkerboard grid and padding visual overlays to match the actual visual boundaries of the contained image.

**Key state variables**:
- `lbCurrentGid` — active group ID
- `lbIdx` — current filmstrip position
- `_followLatestGid` — auto-follow newly generated image
- `_sbSelectedAngles` — selected camera angle names
- `_sbWireframeRef` / `_sbWireframeTime` — wireframe video + frame time
- `privacyMode` — privacy overlay toggle (key: `P`)

---

## Wireframe Pose Guide

Wireframe videos stored in `/mnt/zim/tour-comfy/wireframe/`.  
The studio Generate tab shows a video picker + time scrubber.  On generation:
1. Backend calls `GET /wireframe/frame/{name}?t=0.5` to extract the chosen frame
2. Frame passed as `image2` in the ComfyUI workflow (second visual reference)
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via `image2` slot

---

## Pose Tools (2D body pose)

A posenet-style toolset built on a feature-detected estimator. Preferred backend is
**rtmlib** (RTMPose, ONNX via the already-installed `onnxruntime`); **mediapipe** is a
fallback. If neither is installed the endpoints return `501` and the UI shows an install hint.

- **Install (A6000 venv):** `~/comfyui/venv/bin/pip install rtmlib`. Models (~150 MB) auto-
  download to `~/.cache/rtmlib` on first use.
- **CUDA libs:** `onnxruntime-gpu` needs the venv's `nvidia-*-cu12` libs on the loader path.
  `a6000-comfy/start_api.sh` adds them to `LD_LIBRARY_PATH` at launch (torch finds them via
  RPATH, onnxruntime does not).

**Output format:** COCO-17 keypoints (`POSE_KEYPOINT_NAMES`) + `POSE_SKELETON` edge list,
returned in image pixels with per-joint score.

**Studio overlay (`car.html`):** the **Pose** button overlays the skeleton on the viewer
(letterbox-aware canvas). Joints are **draggable** to refine/explore a pose. Overlay toolbar:
*Reset* (re-detect) and *Similar pose* (rank the library by the current — possibly edited —
skeleton). Results render as a thumbnail strip; clicking one opens that image's group.

**Similarity model (`edit_api.py`):**
- `_pose_descriptor(keypoints)` → translation/scale-invariant vector: centered on hip midpoint
  (fallback shoulders), scaled by torso length, 17×(x,y) + visibility mask. `None` if < 6 joints.
- `_pose_distance(a, b)` → weighted L2 over jointly-visible joints, taking the min of the direct
  and the **left-right-mirrored** comparison so mirrored poses match.
- **Pose index:** descriptors are cached in `<output>/_data/poses_index.json`. It fills
  incrementally whenever `/images/{f}/pose` runs, or in bulk via `/pose/index` (daemon thread,
  batched writes every 50, progress via `/pose/index/status`).

## File Layout

```
qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA (Studio Interface)
│   ├── trash.html        # frontend Database Consistency Panel SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file (System Documentation)
├── ARCHETECTURE.svg      # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps
```

---

## Configuration (`config.json`)

| Key | Default | Purpose |
|-----|---------|---------|
| `output_dir` | `../output` | Where generated PNGs are saved |
| `comfy_url` | `http://127.0.0.1:8188` | ComfyUI base URL |
| `host` | `0.0.0.0` | FastAPI bind host |
| `port` | `8500` | FastAPI bind port |
| `max_area` | `1048576` | Max pixel budget (1 MP) |

---

## Background Services

| Service | Script | Purpose |
|---------|--------|---------|
| `comfyui-backend` | `a6000-comfy/run_comfyui.sh` | Qwen GGUF inference |
| `comfyui-api` | `tour-comfy/start_api.sh` | FastAPI HTTP server |
| `watcher` | `tour-comfy/start_watcher.sh` | Monitors output dir, auto-processes dropped images |