qwen-image/AGENTS.md

# Qwen-Image-Edit Rapid-AIO v23 — System Documentation

## Overview

A self-hosted AI image studio for generative editing, multi-view character rendering, background removal, face swapping, and group-based asset management.  It runs the **Qwen 2.5 VL GGUF** model through ComfyUI and exposes a feature-rich browser frontend backed by a FastAPI service.

```
Browser (car.html SPA)
        │  HTTP / fetch
        ▼
FastAPI :8500  (edit_api.py)
   ├─── ComfyUI :8188  ← Qwen GGUF model
   ├─── SAM2 / rembg   ← background removal
   ├─── insightface    ← faceswap + face extraction
   ├─── GFPGAN         ← face restoration
   └─── PostgreSQL     ← metadata (192.168.1.160:5433)
              │
        File system  (/mnt/zim/tour-comfy/output/, wireframe/)
```

---

## Hardware

| Component | Spec |
|-----------|------|
| GPU | NVIDIA RTX A6000 (48 GB VRAM) |
| CUDA | 12.4 |
| PyTorch | 2.6.0+cu124 |
| ComfyUI | latest pinned (a6000-comfy/) |

---

## Services

### 1. ComfyUI Backend — `:8188`

- **Script**: `a6000-comfy/run_comfyui.sh`
- **Systemd**: `comfyui-backend`
- **Purpose**: Executes the Qwen GGUF model via `workflow_qwen_edit.json`

**Loaded models**

| Role | File |
|------|------|
| GGUF LLM | `Qwen-Rapid-NSFW-v23_Q8_0.gguf` |
| Text encoder | `qwen_2.5_vl_7b_fp8_scaled.safetensors` |
| VAE | `qwen_image_vae.safetensors` |

**Workflow nodes** (`workflow_qwen_edit.json`):
- `NODE_POSITIVE` — positive text prompt
- `NODE_NEGATIVE` — negative prompt
- `NODE_LATENT` — latent dimensions (width × height)
- `NODE_KSAMPLER` — sampler params (Euler/karras, seed, steps, CFG)
- `image1` — reference image (base64 uploaded to ComfyUI `/upload/image`)
- `image2` — optional wireframe pose guide (extracted from wireframe video)

---

### 2. FastAPI API — `:8500`

- **Script**: `tour-comfy/edit_api.py`
- **Systemd**: `comfyui-api`
- **DB**: PostgreSQL at `192.168.1.160:5433`, database `dv`, table `person`

#### Core Endpoints

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/images` | List all images with DB metadata |
| `GET` | `/images/{filename}` | Serve image file |
| `POST` | `/upload` | Upload image; optionally add to existing group (`group_id` + `skip_poses`) |
| `POST` | `/batch` | Queue multi-prompt generation job (async) |
| `GET` | `/jobs/{id}` | Poll batch job status / per-item progress |
| `POST` | `/images/{filename}/set-preferred` | Move to group slot 0; triggers face extraction background task |
| `POST` | `/images/{filename}/extract-face` | Extract & save face crop via insightface (background task) |
| `POST` | `/images/{filename}/archive` | Soft-archive (hidden from default view) |
| `POST` | `/images/{filename}/unarchive` | Restore archived image |
| `POST` | `/images/{filename}/set-hidden` | Toggle studio visibility |
| `DELETE` | `/images/{filename}` | Delete image + DB record |
| `POST` | `/images/{filename}/crop` | Crop to pixel rect (in-place) |
| `POST` | `/autocrop/{filename}` | Auto-trim transparent border |
| `POST` | `/faceswap` | insightface video faceswap |
| `POST` | `/remove-background/{filename}` | rembg BG removal → `.nobg.png` sidecar |
| `POST` | `/remove-background-sam/{filename}` | SAM2 BG removal → `.nobg.png` sidecar |
| `GET` | `/wireframe/frame/{video_name}` | Extract frame at `?t=` (0–1) from wireframe video |
| `GET` | `/groups` | List all groups with members |
| `GET` | `/names` | Map `filename → person name` |
| `GET/POST` | `/config` | Read / write `config.json` |

---

## Generation Pipeline (`_run_pipeline`)

```
prompt
  │
  ├─ "transparent background" keywords detected?
  │     YES → strip keywords from Qwen prompt (keeps `is_transparent` flag)
  │            set negative: "deformed anatomy, watermark, logo"
  │
  ├─ Upload reference image(s) to ComfyUI /upload/image
  │     image1 = source image
  │     image2 = wireframe pose guide frame (optional)
  │
  ├─ Patch workflow graph nodes (prompt, size, seed, sampler)
  ├─ POST to ComfyUI /prompt  →  poll /history until done
  ├─ Fetch PNG from ComfyUI /view
  │
  └─ is_transparent?
       YES → _apply_transparency_sam2(png_bytes)
              ├─ SAM2 point-prompt segmentation
              │    FG: 12 points (hair→shoes, lateral spread for ¾ poses)
              │    BG: 7 points (corners + edge midpoints)
              │    Coverage check: 5%–92% (else fallback)
              │    Gaussian blur radius=1 for soft edges
              └─ fallback: _apply_transparency (rembg U2Net)
```

---

## Batch Worker (`_batch_worker`)

Runs in a background thread pool. One thread per batch job item.

```
For each (filename, prompt) pair:
  1. Load source image from output dir
  2. If wireframe_ref: extract frame at wireframe_time via OpenCV VideoCapture
  3. Call _run_pipeline(pil, prompt, seed, max_area, extra_images=[wireframe_frame])
  4. Save PNG to output dir (timestamped filename)
  5. upsert_person(new_filename, group_id, prompt, source_refs=[original])
  6. Update job progress dict (polled via /jobs/{id})
```

---

## Camera Angle Presets

Eight absolute + four relative angle prompts defined as `CAMERA_ANGLES` in car.html.  Each triggers a `/batch` request with `pose=null` and the angle prompt injected.

| Name | Direction |
|------|-----------|
| Front | Straight-on frontal |
| ¾ Left / ¾ Right | 45° rotation |
| Side L / Side R | 90° profile |
| Back | Rear view |
| High / Low | Bird's-eye / worm's-eye |
| ↺ 45° / ↻ 45° | Relative +45° left / right |
| ↺ 90° / ↻ 90° | Relative +90° left / right |

---

## Background Removal

Two strategies are available:

### Configurable BG removal — `config.json` key `"bg_removal"`

Set to `"rembg"` (default) or `"sam2"`.

#### `"rembg"` mode
- Strips transparent-related keywords from the Qwen prompt entirely → Qwen renders with a natural real background
- `_apply_transparency` (rembg U2Net) post-processes the result
- Most reliable on complex Qwen-generated backgrounds (streets, shadows, cobblestones, etc.)

#### `"sam2"` mode
- Replaces transparent keywords with `"black background"` in the Qwen prompt → Qwen renders the person against a pure black background
- Negative prompt discourages any real-background content
- `_apply_transparency_sam2` (SAM2 bbox) post-processes: pure black gives maximal contrast → reliable segmentation
- Tested: produces clean full-body mask at ~74% coverage including hair, shoes, and earrings

#### Explicit `"black background"` in prompt
- Regardless of the configured method, always routes to SAM2 — the user has already set up the ideal SAM2 input
- Prompt is passed to Qwen unchanged

#### SAM2 implementation (`_apply_transparency_sam2`)
- Model: `sam2.1_hiera_base_plus.pt` at `~/.sam/sam2.1_hiera_base_plus.pt`
- Bbox `[1%, 1%, 99%, 99%]`, `multimask_output=True`, best-score mask, coverage check 5%–92%, Gaussian edge blur radius=1
- Also called by the explicit `/remove-background-sam/{filename}` endpoint

---

## Face Processing (insightface)

Model: `buffalo_l`

**Faceswap** (`/faceswap`):
- Reads source face from reference image
- Swaps into each frame of a target video via OpenCV
- Optionally runs GFPGAN face restoration post-swap

**Face extraction** (`_extract_face_bg` background task):
- Triggered on `set-preferred` or explicit `extract-face` endpoint
- Detects largest face via insightface, adds padding (50% sides, 200% top headroom)
- Saves as `{group_id}_face.png` in output dir
- DB-registered for display in the studio Info tab face-book thumbnail

---

## Database Schema (`person` table)

| Column | Type | Purpose |
|--------|------|---------|
| filename | TEXT PK | Unique image filename |
| filepath | TEXT | Absolute file path |
| group_id | TEXT | Group membership |
| group_name | TEXT | Human-readable group label |
| sort_order | INTEGER | Position within group (0 = preferred) |
| name | TEXT | Person / character name |
| prompt | TEXT | Generation prompt used |
| pose | TEXT | Pose tag |
| hidden | BOOLEAN | Hidden from studio preview |
| archived | BOOLEAN | Soft-deleted |
| has_background | BOOLEAN | Whether BG was removed |
| has_clothing | BOOLEAN | Content classification flag |
| source_refs | TEXT (JSON) | Original filenames this was derived from |
| content_type | TEXT | `image` or `video` |
| faceswap_source_video | TEXT | Source video for faceswapped clips |

---

## Frontend (car.html)

Single-page application (~5000 lines). No build step — pure HTML/CSS/JS.

**Modes**:
- **Gallery** — grid of all groups; click to open Studio
- **Studio** — filmstrip + large viewer + sidebar tabs

**Sidebar tabs**:

| Tab | Purpose |
|-----|---------|
| Generate | Camera angle picker, pose selector, wireframe guide, prompt override, fine-tune textarea, batch submit |
| Info | Metadata, preferred toggle, face-book thumbnail, hide/archive/delete actions, manual crop |
| Faceswap | Source face selector, target video, preview quality toggle |
| Segment | SAM2 and rembg BG removal buttons |
| Scenery | Scenery video reference for generation |

**Key state variables**:
- `lbCurrentGid` — active group ID
- `lbIdx` — current filmstrip position
- `_followLatestGid` — auto-follow newly generated image
- `_sbSelectedAngles` — selected camera angle names
- `_sbWireframeRef` / `_sbWireframeTime` — wireframe video + frame time
- `privacyMode` — privacy overlay toggle (key: `P`)

---

## Wireframe Pose Guide

Wireframe videos stored in `/mnt/zim/tour-comfy/wireframe/`.  
The studio Generate tab shows a video picker + time scrubber.  On generation:
1. Backend calls `GET /wireframe/frame/{name}?t=0.5` to extract the chosen frame
2. Frame passed as `image2` in the ComfyUI workflow (second visual reference)
3. No ControlNet installed — Qwen uses the wireframe as a layout hint via `image2` slot

---

## File Layout

```
qwen-image-edit-rapid-aio-nsfw-v23/
├── tour-comfy/
│   ├── car.html          # frontend SPA
│   ├── edit_api.py       # FastAPI backend
│   ├── database.py       # PostgreSQL helpers
│   ├── watcher.py        # output dir auto-processor
│   ├── config.json       # output_dir, comfy_url, etc.
│   └── workflow_qwen_edit.json  # ComfyUI graph template
├── a6000-comfy/          # ComfyUI install (symlinked to ~/comfyui)
├── AGENTS.md             # this file
├── architecture2.svg     # system architecture diagram
├── backlog.md            # feature backlog
└── requirements.txt      # Python deps
```

---

## Configuration (`config.json`)

| Key | Default | Purpose |
|-----|---------|---------|
| `output_dir` | `../output` | Where generated PNGs are saved |
| `comfy_url` | `http://127.0.0.1:8188` | ComfyUI base URL |
| `host` | `0.0.0.0` | FastAPI bind host |
| `port` | `8500` | FastAPI bind port |
| `max_area` | `1048576` | Max pixel budget (1 MP) |

---

## Background Services

| Service | Script | Purpose |
|---------|--------|---------|
| `comfyui-backend` | `a6000-comfy/run_comfyui.sh` | Qwen GGUF inference |
| `comfyui-api` | `tour-comfy/start_api.sh` | FastAPI HTTP server |
| `watcher` | `tour-comfy/start_watcher.sh` | Monitors output dir, auto-processes dropped images |