## Add completion API for IDE LLM integration

_2026-06-26 17:08 · 20 messages_

**User:**

<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>

**User:** w

<command-name>/model</command-name>
            <command-message>model</command-message>
            <command-args>sonnet</command-args>

**User:**

<local-command-stdout>Set model to sonnet (claude-sonnet-4-6)</local-command-stdout>

**User:**

we have a network llm, I want to use it to refine the poses.md within IDE. 
The IDE prove Thrid party AI provider. 
we run the api at  http://192.168.1.160:8001 i think we need to add some more api feauters for completion or something in  `pose_llm_api.py` and redeploy

[@pose_llm_api.py](file:///home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py)

**Claude:**

The user wants to use their network LLM (running at http://192.168.1.160:8001) as a third-party AI provider in their IDE. They want to add API features to `pose_llm_api.py` to make it compatible with IDE integration (likely OpenAI-compatible endpoints for completion).

Let me read the current API file to understand what's there.

**Claude:**

Let me read the current API file first.

**Claude:**

[Tool: Read]
{"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py"}

**Claude:**

The user wants to use this LLM API as a third-party AI provider in their IDE (likely JetBrains PyCharm, based on the `.cache/JetBrains/PyCharm2026.1` path). 

Most IDEs that support third-party AI providers look for OpenAI-compatible APIs. The current API already has:
- `POST /v1/chat/completions` - OpenAI-compatible chat
- `GET /v1/models` - list models

What might be missing for IDE integration:
1. **Streaming support** - Many IDEs expect streaming responses (SSE - Server-Sent Events) for `/v1/chat/completions`. The current implementation doesn't support `stream: true`.
2. **`/v1/completions`** - Some IDEs use the legacy text completion endpoint
3. **CORS headers** - Needed for browser-based IDE integrations

The path `.cache/JetBrains/PyCharm2026.1/aia/agents` confirms they're using JetBrains AI Assistant, which almost certainly requires streaming support. I should add SSE streaming for the chat endpoint, consider the legacy completions endpoint, and ensure CORS is properly configured.

The main challenge is that my current implementation generates responses all at once, but I need to stream tokens back using the OpenAI format with proper SSE events. I'll need to refactor the chat logic to use a token iterator that yields chunks as they're generated.

**Claude:**

The API already has `chat/completions` and `models`, but it's missing **streaming** — which IDEs almost universally require for the typeahead/inline completion experience. I'll add:

1. `stream: true` support to `/v1/chat/completions` (SSE via `TextIteratorStreamer`)
2. `/v1/completions` (legacy text completion — some IDE plugins call this endpoint)
3. CORS headers (needed when IDE makes cross-origin requests)

**Claude:**

[Tool: Edit]
{"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"#!/usr/bin/env python3\n\"\"\"\nUncensored chat LLM API — loads an instruct model once, serves an\nOpenAI-compatible /v1/chat/completions endpoint. Runs on the AMD MI50 (gfx906)\nvia ROCm 5.7 + torch 2.3.1, mirroring the joycaption service pattern.\n\nModel fits in fp16 inside the 32GB VRAM (≈12B ceiling). Override with env MODEL_ID.\n\nEndpoints:\n  POST /v1/chat/completions  — OpenAI-compatible chat (used by gen_poses.py)\n  GET  /v1/models            — list the loaded model\n  GET  /health               — health check + GPU info\n\nEnv:\n  MODEL_ID   HuggingFace repo id (default below)\n  PORT       listen port (default 8001)\n  HSA_OVERRIDE_GFX_VERSION=9.0.6  set by start script for gfx906\n\"\"\"\n\nimport asyncio\nimport os\nimport subprocess\nimport time\nimport uuid\nfrom concurrent.futures import ThreadPoolExecutor\nfrom contextlib import asynccontextmanager\n\nimport torch\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer","new_string":"#!/usr/bin/env python3\n\"\"\"\nUncensored chat LLM API — loads an instruct model once, serves an\nOpenAI-compatible /v1/chat/completions endpoint. Runs on the AMD MI50 (gfx906)\nvia ROCm 5.7 + torch 2.3.1, mirroring the joycaption service pattern.\n\nModel fits in fp16 inside the 32GB VRAM (≈12B ceiling). Override with env MODEL_ID.\n\nEndpoints:\n  POST /v1/chat/completions  — OpenAI-compatible chat, supports stream=true (SSE)\n  POST /v1/completions       — legacy text completion, supports stream=true (SSE)\n  GET  /v1/models            — list the loaded model\n  GET  /health               — health check + GPU info\n\nEnv:\n  MODEL_ID   HuggingFace repo id (default below)\n  PORT       listen port (default 8001)\n  HSA_OVERRIDE_GFX_VERSION=9.0.6  set by start script for gfx906\n\"\"\"\n\nimport asyncio\nimport json\nimport os\nimport subprocess\nimport threading\nimport time\nimport uuid\nfrom concurrent.futures import ThreadPoolExecutor\nfrom contextlib import asynccontextmanager\n\nimport torch\nfrom fastapi import FastAPI, HTTPException\nfrom fastapi.middleware.cors import CORSMiddleware\nfrom fastapi.responses import StreamingResponse\nfrom pydantic import BaseModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer"}

**Claude:**

[Tool: Edit]
{"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"app = FastAPI(title=\"Uncensored Chat LLM API\", lifespan=lifespan)","new_string":"app = FastAPI(title=\"Uncensored Chat LLM API\", lifespan=lifespan)\n\napp.add_middleware(\n    CORSMiddleware,\n    allow_origins=[\"*\"],\n    allow_methods=[\"*\"],\n    allow_headers=[\"*\"],\n)"}

**Claude:**

Now add the streaming helper and update the models + endpoint:

**Claude:**

[Tool: Edit]
{"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"async def run_chat(*args):\n    return await asyncio.get_running_loop().run_in_executor(gpu_executor, _run_chat, *args)\n\n\nclass ChatMessage(BaseModel):\n    role: str\n    content: str\n\n\nclass ChatRequest(BaseModel):\n    model: str | None = None\n    messages: list[ChatMessage]\n    max_tokens: int = 512\n    temperature: float = 0.8\n    top_p: float = 0.95\n    stop: list[str] | str | None = None\n\n\n@app.post(\"/v1/chat/completions\")\nasync def chat_completions(req: ChatRequest):\n    if not req.messages:\n        raise HTTPException(400, \"messages must not be empty\")\n    msgs = [{\"role\": m.role, \"content\": m.content} for m in req.messages]\n    r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n    return {\n        \"id\": f\"chatcmpl-{uuid.uuid4().hex[:24]}\",\n        \"object\": \"chat.completion\",\n        \"created\": int(time.time()),\n        \"model\": MODEL_ID,\n        \"choices\": [\n            {\n                \"index\": 0,\n                \"message\": {\"role\": \"assistant\", \"content\": r[\"text\"]},\n                \"finish_reason\": r[\"finish_reason\"],\n            }\n        ],\n        \"usage\": {\n            \"prompt_tokens\": r[\"prompt_tokens\"],\n            \"completion_tokens\": r[\"completion_tokens\"],\n            \"total_tokens\": r[\"prompt_tokens\"] + r[\"completion_tokens\"],\n        },\n        \"timing\": {\"generate_s\": r[\"generate_s\"]},\n    }\n\n\n@app.get(\"/v1/models\")\nasync def list_models():\n    return {\"object\": \"list\", \"data\": [{\"id\": MODEL_ID, \"object\": \"model\"}]}","new_string":"async def run_chat(*args):\n    return await asyncio.get_running_loop().run_in_executor(gpu_executor, _run_chat, *args)\n\n\ndef _iter_stream(messages, max_tokens, temperature, top_p, stop):\n    \"\"\"Generator that yields SSE lines for a streaming chat completion.\"\"\"\n    tokenizer = state[\"tokenizer\"]\n    model = state[\"model\"]\n    cid = f\"chatcmpl-{uuid.uuid4().hex[:24]}\"\n    created = int(time.time())\n\n    input_ids = tokenizer.apply_chat_template(\n        messages, add_generation_prompt=True, return_tensors=\"pt\"\n    ).to(model.device)\n\n    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n    do_sample = temperature is not None and temperature > 0\n    gen_kwargs = dict(\n        input_ids=input_ids,\n        max_new_tokens=max_tokens,\n        do_sample=do_sample,\n        use_cache=True,\n        pad_token_id=tokenizer.eos_token_id,\n        streamer=streamer,\n    )\n    if do_sample:\n        gen_kwargs[\"temperature\"] = temperature\n        gen_kwargs[\"top_p\"] = top_p\n\n    stops = []\n    if stop:\n        stops = [stop] if isinstance(stop, str) else list(stop)\n\n    def _generate():\n        with torch.inference_mode():\n            model.generate(**gen_kwargs)\n\n    t = threading.Thread(target=_generate, daemon=True)\n    t.start()\n\n    buffer = \"\"\n    for token in streamer:\n        buffer += token\n        # Check stop sequences across the accumulated buffer\n        if stops:\n            cut = min((buffer.find(s) for s in stops if s and s in buffer), default=-1)\n            if cut != -1:\n                token = buffer[len(buffer) - len(token):cut] if cut < len(buffer) else \"\"\n                chunk = {\n                    \"id\": cid, \"object\": \"chat.completion.chunk\", \"created\": created,\n                    \"model\": MODEL_ID,\n                    \"choices\": [{\"index\": 0, \"delta\": {\"content\": token}, \"finish_reason\": None}],\n                }\n                yield f\"data: {json.dumps(chunk)}\\n\\n\"\n                break\n        chunk = {\n            \"id\": cid, \"object\": \"chat.completion.chunk\", \"created\": created,\n            \"model\": MODEL_ID,\n            \"choices\": [{\"index\": 0, \"delta\": {\"content\": token}, \"finish_reason\": None}],\n        }\n        yield f\"data: {json.dumps(chunk)}\\n\\n\"\n\n    t.join()\n    done = {\n        \"id\": cid, \"object\": \"chat.completion.chunk\", \"created\": created,\n        \"model\": MODEL_ID,\n        \"choices\": [{\"index\": 0, \"delta\": {}, \"finish_reason\": \"stop\"}],\n    }\n    yield f\"data: {json.dumps(done)}\\n\\n\"\n    yield \"data: [DONE]\\n\\n\"\n    del input_ids\n\n\nclass ChatMessage(BaseModel):\n    role: str\n    content: str\n\n\nclass ChatRequest(BaseModel):\n    model: str | None = None\n    messages: list[ChatMessage]\n    max_tokens: int = 512\n    temperature: float = 0.8\n    top_p: float = 0.95\n    stop: list[str] | str | None = None\n    stream: bool = False\n\n\nclass CompletionRequest(BaseModel):\n    model: str | None = None\n    prompt: str\n    max_tokens: int = 512\n    temperature: float = 0.8\n    top_p: float = 0.95\n    stop: list[str] | str | None = None\n    stream: bool = False\n\n\n@app.post(\"/v1/chat/completions\")\nasync def chat_completions(req: ChatRequest):\n    if not req.messages:\n        raise HTTPException(400, \"messages must not be empty\")\n    msgs = [{\"role\": m.role, \"content\": m.content} for m in req.messages]\n\n    if req.stream:\n        gen = _iter_stream(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n        return StreamingResponse(gen, media_type=\"text/event-stream\")\n\n    r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n    return {\n        \"id\": f\"chatcmpl-{uuid.uuid4().hex[:24]}\",\n        \"object\": \"chat.completion\",\n        \"created\": int(time.time()),\n        \"model\": MODEL_ID,\n        \"choices\": [\n            {\n                \"index\": 0,\n                \"message\": {\"role\": \"assistant\", \"content\": r[\"text\"]},\n                \"finish_reason\": r[\"finish_reason\"],\n            }\n        ],\n        \"usage\": {\n            \"prompt_tokens\": r[\"prompt_tokens\"],\n            \"completion_tokens\": r[\"completion_tokens\"],\n            \"total_tokens\": r[\"prompt_tokens\"] + r[\"completion_tokens\"],\n        },\n        \"timing\": {\"generate_s\": r[\"generate_s\"]},\n    }\n\n\n@app.post(\"/v1/completions\")\nasync def completions(req: CompletionRequest):\n    \"\"\"Legacy text completion endpoint — wraps prompt as a user message.\"\"\"\n    msgs = [{\"role\": \"user\", \"content\": req.prompt}]\n\n    if req.stream:\n        cid = f\"cmpl-{uuid.uuid4().hex[:24]}\"\n        created = int(time.time())\n\n        def _legacy_stream():\n            for sse in _iter_stream(msgs, req.max_tokens, req.temperature, req.top_p, req.stop):\n                # Re-wrap chat chunk format as legacy completion chunk\n                if sse.startswith(\"data: [DONE]\"):\n                    yield sse\n                    return\n                if sse.startswith(\"data: \"):\n                    chat_chunk = json.loads(sse[6:])\n                    delta = chat_chunk[\"choices\"][0][\"delta\"].get(\"content\", \"\")\n                    finish = chat_chunk[\"choices\"][0][\"finish_reason\"]\n                    chunk = {\n                        \"id\": cid, \"object\": \"text_completion\", \"created\": created,\n                        \"model\": MODEL_ID,\n                        \"choices\": [{\"index\": 0, \"text\": delta, \"finish_reason\": finish}],\n                    }\n                    yield f\"data: {json.dumps(chunk)}\\n\\n\"\n\n        return StreamingResponse(_legacy_stream(), media_type=\"text/event-stream\")\n\n    r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n    return {\n        \"id\": f\"cmpl-{uuid.uuid4().hex[:24]}\",\n        \"object\": \"text_completion\",\n        \"created\": int(time.time()),\n        \"model\": MODEL_ID,\n        \"choices\": [\n            {\n                \"index\": 0,\n                \"text\": r[\"text\"],\n                \"finish_reason\": r[\"finish_reason\"],\n            }\n        ],\n        \"usage\": {\n            \"prompt_tokens\": r[\"prompt_tokens\"],\n            \"completion_tokens\": r[\"completion_tokens\"],\n            \"total_tokens\": r[\"prompt_tokens\"] + r[\"completion_tokens\"],\n        },\n    }\n\n\n@app.get(\"/v1/models\")\nasync def list_models():\n    return {\n        \"object\": \"list\",\n        \"data\": [{\"id\": MODEL_ID, \"object\": \"model\", \"owned_by\": \"local\", \"permission\": []}],\n    }"}

**Claude:**

Let me verify the final file looks clean:

**Claude:**

[Tool: Read]
{"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py"}

**Claude:**

Good. Now let me check the deploy script to see if there's anything to update there:

**Claude:**

[Tool: Bash]
{"command":"ls /home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/ && cat /home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/tour-comfy/deploy_pose_llm.sh 2>/dev/null || echo \"not found\"","description":"List pose_llm dir and check deploy script"}

**Claude:**

[Tool: Read]
{"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/deploy_pose_llm.sh"}

**Claude:**

I see - the deploy script is using `llama-server` (llama.cpp), not the Python FastAPI `pose_llm_api.py`. The llama.cpp server already supports streaming natively because it implements the OpenAI-compatible API. 

So there are actually two separate things:
1. `pose_llm_api.py` - the Python FastAPI implementation
2. `deploy_pose_llm.sh` - deploys `llama-server` (llama.cpp) which is the actual server running

The deploy script uses llama.cpp's `llama-server` binary directly, which already supports:
- `/v1/chat/completions` with streaming
- `/v1/completions` with streaming  
- `/v1/models`
- `/health`

So the llama.cpp server should already be compatible with IDE providers! The Python `pose_llm_api.py` seems to be an alternative/development version. For the IDE to connect, I just need the base URL `http://192.168.1.160:8001` since it already has the OpenAI-compatible endpoints.

Though the user specifically asked about `pose_llm_api.py` and adding completion features, so they might want to use the Python API instead. I've completed the edits to that file, but the deploy script would need updating since it currently runs llama.cpp with GGUF models, while the Python version uses HuggingFace transformers—these are different backends entirely.

**Claude:**

Done. The file is ready. One thing to note: the `deploy_pose_llm.sh` deploys `llama-server` (llama.cpp with the GGUF model), not the Python `pose_llm_api.py`. llama.cpp already speaks OpenAI-compatible streaming natively, so the live server at `:8001` may already work.

**What I added to `pose_llm_api.py`:**

| Addition | Why |
|---|---|
| `stream: bool = False` on both request models | IDEs send `"stream": true` by default |
| `_iter_stream()` using `TextIteratorStreamer` | Runs `model.generate` in a thread, yields SSE chunks token-by-token |
| `StreamingResponse` on both `chat/completions` and `completions` | FastAPI returns `text/event-stream` |
| `POST /v1/completions` | Legacy endpoint — JetBrains AI and some other IDEs call this for inline completion |
| `CORSMiddleware` with `allow_origins=["*"]` | IDE browser-side requests need CORS headers |
| `owned_by`/`permission` on `/v1/models` | Some clients validate the full model object shape |

**To configure your IDE (e.g. JetBrains AI Assistant):**
- Provider type: **OpenAI-compatible**
- Base URL: `http://192.168.1.160:8001`  
- API key: anything (it's ignored, but the field usually can't be blank — use `local`)
- Model: `dphn/Dolphin3.0-Mistral-24B`

If the current deployment is still the llama.cpp `llama-server`, try the IDE connection first — it may already work. If you want to switch to the Python API, run `./deploy_pose_llm.sh stop` and start `pose_llm_api.py` instead.