qwen-image/Add completion API for IDE LLM integration.md

Add completion API for IDE LLM integration

2026-06-26 17:08 · 20 messages

User:

Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.

User: w

/model model sonnet

User:

Set model to sonnet (claude-sonnet-4-6)

User:

we have a network llm, I want to use it to refine the poses.md within IDE. The IDE prove Thrid party AI provider. we run the api at http://192.168.1.160:8001 i think we need to add some more api feauters for completion or something in pose_llm_api.py and redeploy

@pose_llm_api.py

Claude:

The user wants to use their network LLM (running at http://192.168.1.160:8001) as a third-party AI provider in their IDE. They want to add API features to pose_llm_api.py to make it compatible with IDE integration (likely OpenAI-compatible endpoints for completion).

Let me read the current API file to understand what's there.

Claude:

Let me read the current API file first.

Claude:

[Tool: Read] {"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py"}

Claude:

The user wants to use this LLM API as a third-party AI provider in their IDE (likely JetBrains PyCharm, based on the .cache/JetBrains/PyCharm2026.1 path).

Most IDEs that support third-party AI providers look for OpenAI-compatible APIs. The current API already has:

• POST /v1/chat/completions - OpenAI-compatible chat
• GET /v1/models - list models

What might be missing for IDE integration:

1. Streaming support - Many IDEs expect streaming responses (SSE - Server-Sent Events) for /v1/chat/completions. The current implementation doesn't support stream: true.
2. /v1/completions - Some IDEs use the legacy text completion endpoint
3. CORS headers - Needed for browser-based IDE integrations

The path .cache/JetBrains/PyCharm2026.1/aia/agents confirms they're using JetBrains AI Assistant, which almost certainly requires streaming support. I should add SSE streaming for the chat endpoint, consider the legacy completions endpoint, and ensure CORS is properly configured.

The main challenge is that my current implementation generates responses all at once, but I need to stream tokens back using the OpenAI format with proper SSE events. I'll need to refactor the chat logic to use a token iterator that yields chunks as they're generated.

Claude:

The API already has chat/completions and models, but it's missing streaming — which IDEs almost universally require for the typeahead/inline completion experience. I'll add:

1. stream: true support to /v1/chat/completions (SSE via TextIteratorStreamer)
2. /v1/completions (legacy text completion — some IDE plugins call this endpoint)
3. CORS headers (needed when IDE makes cross-origin requests)

Claude:

[Tool: Edit] {"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"#!/usr/bin/env python3\n"""\nUncensored chat LLM API — loads an instruct model once, serves an\nOpenAI-compatible /v1/chat/completions endpoint. Runs on the AMD MI50 (gfx906)\nvia ROCm 5.7 + torch 2.3.1, mirroring the joycaption service pattern.\n\nModel fits in fp16 inside the 32GB VRAM (≈12B ceiling). Override with env MODEL_ID.\n\nEndpoints:\n POST /v1/chat/completions — OpenAI-compatible chat (used by gen_poses.py)\n GET /v1/models — list the loaded model\n GET /health — health check + GPU info\n\nEnv:\n MODEL_ID HuggingFace repo id (default below)\n PORT listen port (default 8001)\n HSA_OVERRIDE_GFX_VERSION=9.0.6 set by start script for gfx906\n"""\n\nimport asyncio\nimport os\nimport subprocess\nimport time\nimport uuid\nfrom concurrent.futures import ThreadPoolExecutor\nfrom contextlib import asynccontextmanager\n\nimport torch\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer","new_string":"#!/usr/bin/env python3\n"""\nUncensored chat LLM API — loads an instruct model once, serves an\nOpenAI-compatible /v1/chat/completions endpoint. Runs on the AMD MI50 (gfx906)\nvia ROCm 5.7 + torch 2.3.1, mirroring the joycaption service pattern.\n\nModel fits in fp16 inside the 32GB VRAM (≈12B ceiling). Override with env MODEL_ID.\n\nEndpoints:\n POST /v1/chat/completions — OpenAI-compatible chat, supports stream=true (SSE)\n POST /v1/completions — legacy text completion, supports stream=true (SSE)\n GET /v1/models — list the loaded model\n GET /health — health check + GPU info\n\nEnv:\n MODEL_ID HuggingFace repo id (default below)\n PORT listen port (default 8001)\n HSA_OVERRIDE_GFX_VERSION=9.0.6 set by start script for gfx906\n"""\n\nimport asyncio\nimport json\nimport os\nimport subprocess\nimport threading\nimport time\nimport uuid\nfrom concurrent.futures import ThreadPoolExecutor\nfrom contextlib import asynccontextmanager\n\nimport torch\nfrom fastapi import FastAPI, HTTPException\nfrom fastapi.middleware.cors import CORSMiddleware\nfrom fastapi.responses import StreamingResponse\nfrom pydantic import BaseModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer"}

Claude:

[Tool: Edit] {"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"app = FastAPI(title="Uncensored Chat LLM API", lifespan=lifespan)","new_string":"app = FastAPI(title="Uncensored Chat LLM API", lifespan=lifespan)\n\napp.add_middleware(\n CORSMiddleware,\n allow_origins=[""],\n allow_methods=[""],\n allow_headers=["*"],\n)"}

Claude:

Now add the streaming helper and update the models + endpoint:

Claude:

[Tool: Edit] {"replace_all":false,"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py","old_string":"async def run_chat(*args):\n return await asyncio.get_running_loop().run_in_executor(gpu_executor, _run_chat, *args)\n\n\nclass ChatMessage(BaseModel):\n role: str\n content: str\n\n\nclass ChatRequest(BaseModel):\n model: str | None = None\n messages: list[ChatMessage]\n max_tokens: int = 512\n temperature: float = 0.8\n top_p: float = 0.95\n stop: list[str] | str | None = None\n\n\n@app.post("/v1/chat/completions")\nasync def chat_completions(req: ChatRequest):\n if not req.messages:\n raise HTTPException(400, "messages must not be empty")\n msgs = [{"role": m.role, "content": m.content} for m in req.messages]\n r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n return {\n "id": f"chatcmpl-{uuid.uuid4().hex[:24]}",\n "object": "chat.completion",\n "created": int(time.time()),\n "model": MODEL_ID,\n "choices": [\n {\n "index": 0,\n "message": {"role": "assistant", "content": r["text"]},\n "finish_reason": r["finish_reason"],\n }\n ],\n "usage": {\n "prompt_tokens": r["prompt_tokens"],\n "completion_tokens": r["completion_tokens"],\n "total_tokens": r["prompt_tokens"] + r["completion_tokens"],\n },\n "timing": {"generate_s": r["generate_s"]},\n }\n\n\n@app.get("/v1/models")\nasync def list_models():\n return {"object": "list", "data": [{"id": MODEL_ID, "object": "model"}]}","new_string":"async def run_chat(*args):\n return await asyncio.get_running_loop().run_in_executor(gpu_executor, _run_chat, *args)\n\n\ndef _iter_stream(messages, max_tokens, temperature, top_p, stop):\n """Generator that yields SSE lines for a streaming chat completion."""\n tokenizer = state["tokenizer"]\n model = state["model"]\n cid = f"chatcmpl-{uuid.uuid4().hex[:24]}"\n created = int(time.time())\n\n input_ids = tokenizer.apply_chat_template(\n messages, add_generation_prompt=True, return_tensors="pt"\n ).to(model.device)\n\n streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n do_sample = temperature is not None and temperature > 0\n gen_kwargs = dict(\n input_ids=input_ids,\n max_new_tokens=max_tokens,\n do_sample=do_sample,\n use_cache=True,\n pad_token_id=tokenizer.eos_token_id,\n streamer=streamer,\n )\n if do_sample:\n gen_kwargs["temperature"] = temperature\n gen_kwargs["top_p"] = top_p\n\n stops = []\n if stop:\n stops = [stop] if isinstance(stop, str) else list(stop)\n\n def _generate():\n with torch.inference_mode():\n model.generate(**gen_kwargs)\n\n t = threading.Thread(target=_generate, daemon=True)\n t.start()\n\n buffer = ""\n for token in streamer:\n buffer += token\n # Check stop sequences across the accumulated buffer\n if stops:\n cut = min((buffer.find(s) for s in stops if s and s in buffer), default=-1)\n if cut != -1:\n token = buffer[len(buffer) - len(token):cut] if cut < len(buffer) else ""\n chunk = {\n "id": cid, "object": "chat.completion.chunk", "created": created,\n "model": MODEL_ID,\n "choices": [{"index": 0, "delta": {"content": token}, "finish_reason": None}],\n }\n yield f"data: {json.dumps(chunk)}\n\n"\n break\n chunk = {\n "id": cid, "object": "chat.completion.chunk", "created": created,\n "model": MODEL_ID,\n "choices": [{"index": 0, "delta": {"content": token}, "finish_reason": None}],\n }\n yield f"data: {json.dumps(chunk)}\n\n"\n\n t.join()\n done = {\n "id": cid, "object": "chat.completion.chunk", "created": created,\n "model": MODEL_ID,\n "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],\n }\n yield f"data: {json.dumps(done)}\n\n"\n yield "data: [DONE]\n\n"\n del input_ids\n\n\nclass ChatMessage(BaseModel):\n role: str\n content: str\n\n\nclass ChatRequest(BaseModel):\n model: str | None = None\n messages: list[ChatMessage]\n max_tokens: int = 512\n temperature: float = 0.8\n top_p: float = 0.95\n stop: list[str] | str | None = None\n stream: bool = False\n\n\nclass CompletionRequest(BaseModel):\n model: str | None = None\n prompt: str\n max_tokens: int = 512\n temperature: float = 0.8\n top_p: float = 0.95\n stop: list[str] | str | None = None\n stream: bool = False\n\n\n@app.post("/v1/chat/completions")\nasync def chat_completions(req: ChatRequest):\n if not req.messages:\n raise HTTPException(400, "messages must not be empty")\n msgs = [{"role": m.role, "content": m.content} for m in req.messages]\n\n if req.stream:\n gen = _iter_stream(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n return StreamingResponse(gen, media_type="text/event-stream")\n\n r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n return {\n "id": f"chatcmpl-{uuid.uuid4().hex[:24]}",\n "object": "chat.completion",\n "created": int(time.time()),\n "model": MODEL_ID,\n "choices": [\n {\n "index": 0,\n "message": {"role": "assistant", "content": r["text"]},\n "finish_reason": r["finish_reason"],\n }\n ],\n "usage": {\n "prompt_tokens": r["prompt_tokens"],\n "completion_tokens": r["completion_tokens"],\n "total_tokens": r["prompt_tokens"] + r["completion_tokens"],\n },\n "timing": {"generate_s": r["generate_s"]},\n }\n\n\n@app.post("/v1/completions")\nasync def completions(req: CompletionRequest):\n """Legacy text completion endpoint — wraps prompt as a user message."""\n msgs = [{"role": "user", "content": req.prompt}]\n\n if req.stream:\n cid = f"cmpl-{uuid.uuid4().hex[:24]}"\n created = int(time.time())\n\n def _legacy_stream():\n for sse in _iter_stream(msgs, req.max_tokens, req.temperature, req.top_p, req.stop):\n # Re-wrap chat chunk format as legacy completion chunk\n if sse.startswith("data: [DONE]"):\n yield sse\n return\n if sse.startswith("data: "):\n chat_chunk = json.loads(sse[6:])\n delta = chat_chunk["choices"][0]["delta"].get("content", "")\n finish = chat_chunk["choices"][0]["finish_reason"]\n chunk = {\n "id": cid, "object": "text_completion", "created": created,\n "model": MODEL_ID,\n "choices": [{"index": 0, "text": delta, "finish_reason": finish}],\n }\n yield f"data: {json.dumps(chunk)}\n\n"\n\n return StreamingResponse(_legacy_stream(), media_type="text/event-stream")\n\n r = await run_chat(msgs, req.max_tokens, req.temperature, req.top_p, req.stop)\n return {\n "id": f"cmpl-{uuid.uuid4().hex[:24]}",\n "object": "text_completion",\n "created": int(time.time()),\n "model": MODEL_ID,\n "choices": [\n {\n "index": 0,\n "text": r["text"],\n "finish_reason": r["finish_reason"],\n }\n ],\n "usage": {\n "prompt_tokens": r["prompt_tokens"],\n "completion_tokens": r["completion_tokens"],\n "total_tokens": r["prompt_tokens"] + r["completion_tokens"],\n },\n }\n\n\n@app.get("/v1/models")\nasync def list_models():\n return {\n "object": "list",\n "data": [{"id": MODEL_ID, "object": "model", "owned_by": "local", "permission": []}],\n }"}

Claude:

Let me verify the final file looks clean:

Claude:

[Tool: Read] {"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/pose_llm_api.py"}

Claude:

Good. Now let me check the deploy script to see if there's anything to update there:

Claude:

[Tool: Bash] {"command":"ls /home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/ && cat /home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/tour-comfy/deploy_pose_llm.sh 2>/dev/null || echo "not found"","description":"List pose_llm dir and check deploy script"}

Claude:

[Tool: Read] {"file_path":"/home/mike/dev/qwen-image-edit-rapid-aio-nsfw-v23/pose_llm/deploy_pose_llm.sh"}

Claude:

I see - the deploy script is using llama-server (llama.cpp), not the Python FastAPI pose_llm_api.py. The llama.cpp server already supports streaming natively because it implements the OpenAI-compatible API.

So there are actually two separate things:

1. pose_llm_api.py - the Python FastAPI implementation
2. deploy_pose_llm.sh - deploys llama-server (llama.cpp) which is the actual server running

The deploy script uses llama.cpp's llama-server binary directly, which already supports:

• /v1/chat/completions with streaming
• /v1/completions with streaming
• /v1/models
• /health

So the llama.cpp server should already be compatible with IDE providers! The Python pose_llm_api.py seems to be an alternative/development version. For the IDE to connect, I just need the base URL http://192.168.1.160:8001 since it already has the OpenAI-compatible endpoints.

Though the user specifically asked about pose_llm_api.py and adding completion features, so they might want to use the Python API instead. I've completed the edits to that file, but the deploy script would need updating since it currently runs llama.cpp with GGUF models, while the Python version uses HuggingFace transformers—these are different backends entirely.

Claude:

Done. The file is ready. One thing to note: the deploy_pose_llm.sh deploys llama-server (llama.cpp with the GGUF model), not the Python pose_llm_api.py. llama.cpp already speaks OpenAI-compatible streaming natively, so the live server at :8001 may already work.

What I added to pose_llm_api.py:

Addition	Why
stream: bool = False on both request models	IDEs send "stream": true by default
_iter_stream() using TextIteratorStreamer	Runs model.generate in a thread, yields SSE chunks token-by-token
StreamingResponse on both chat/completions and completions	FastAPI returns text/event-stream
POST /v1/completions	Legacy endpoint — JetBrains AI and some other IDEs call this for inline completion
CORSMiddleware with allow_origins=["*"]	IDE browser-side requests need CORS headers
owned_by/permission on /v1/models	Some clients validate the full model object shape

To configure your IDE (e.g. JetBrains AI Assistant):

• Provider type: OpenAI-compatible
• Base URL: http://192.168.1.160:8001
• API key: anything (it's ignored, but the field usually can't be blank — use local)
• Model: dphn/Dolphin3.0-Mistral-24B

If the current deployment is still the llama.cpp llama-server, try the IDE connection first — it may already work. If you want to switch to the Python API, run ./deploy_pose_llm.sh stop and start pose_llm_api.py instead.