Qwen3.5 27B

Retired. qwen-3.5-27b is no longer served on any PolarGrid edge node — the fleet completed its move to qwen-3.8-27b on 2026-08-20, and requests for this id now return 404 model_not_loaded. Update the model field in your requests to qwen-3.8-27b. This page is kept for reference; the benchmark figures below describe the retired model.

Qwen3.5 27B (qwen-3.5-27b) is a 27-billion-parameter text LLM served on PolarGrid edge nodes via Triton’s vllm_backend. Weights ship pre-quantized to FP8 (~28 GB VRAM) and load directly on PolarGrid’s Blackwell edge GPUs without runtime requantization.

HF repo: Qwen/Qwen3.5-27B-FP8
Modality: Text LLM
Backend: Triton vllm (LLM pod)
Available regions: fleet-wide — see Model availability

Headline benchmark

We publish two numbers side by side. End-to-end is what your application actually experiences (request → response, network included). Server-only is what the GPU spends on inference (apples-to-apples vs centralized providers’ published “inference-only” figures). The gap is the latency PolarGrid’s yvr-02 PoP eliminates by being at the edge.

Measurement	TTFT p50	TTFT p95	Throughput p50
End-to-end (with network)	177 ms	243 ms	29.0 tok/s
Server-only (no network)	71 ms	131 ms	—
Network overhead (e2e − server)	106 ms	—	—

Bench: 100 streaming chat-completion runs against https://api.yvr-02.edge.polargrid.ai, captured 2026-07-09 from a local macOS host over the public internet. End-to-end is client wall-clock; server-only is read from the gateway’s pg_metadata SSE event (inference_ttft_ms / inference_total_ms). Reasoning mode off (default). CUDA graphs on (enforce_eager=false, the current fleet default). The yvr-02 node ships on RTX 6000 Pro Blackwell 96 GB — roughly 2.4× the tok/s and less than half the e2e total latency of the earlier yvr-01 (L40S-class) baseline at benchmarks/qwen-2026-05-01/27b/llm_bench.json. Raw runs: benchmarks/yvr-02-2026-07-09/27b/llm_bench.json.

Apples-to-apples disclaimer. Other providers usually publish only their server-side number; comparing it to our server-only row is the fair baseline. Our end-to-end row is what you’ll see from a customer-side request because PolarGrid runs at the edge. The network row above shows exactly how much that’s worth in milliseconds.

How this compares

Provider	TTFT p50	Throughput p50	Source
PolarGrid `qwen-3.5-27b` on Blackwell	177 ms e2e / 71 ms server	29.0 tok/s	this card
Claude Sonnet 4.5	~1600 ms	47.6 tok/s	artificialanalysis.ai
gpt-4o	~850 ms	135 tok/s	artificialanalysis.ai
Cerebras (specialty silicon)	n/a	~2100 tok/s	cerebras.ai
Groq with speculative decoding	n/a	~1665 tok/s	artificialanalysis.ai

PolarGrid wins on TTFT end-to-end against frontier-reasoning providers because of edge proximity (106 ms p50 network leg vs centralized regions). PolarGrid is 4 to 70 times behind specialty silicon on raw throughput; RTX 6000 Pro Blackwell workstation FLOPS are below H100 and H200 datacenter FLOPS. CUDA graphs are enabled (enforce_eager=false, the current fleet default), which recovered about 25 ms of server TTFT versus the earlier eager-mode baseline at benchmarks/qwen-2026-07-07/27b/llm_bench.json.

Quickstart

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "stream": true,
    "max_tokens": 32
  }'

import { PolarGrid } from "@polargrid/polargrid-sdk";

const client = await PolarGrid.create({ apiKey: process.env.POLARGRID_API_KEY });

for await (const chunk of client.chatCompletionStream({
  model: "qwen-3.5-27b",
  messages: [{ role: "user", content: "Say hi in one short sentence." }],
  maxTokens: 32,
})) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) process.stdout.write(content);
}

from polargrid import PolarGrid

client = await PolarGrid.create(api_key="pg_...")

async for chunk in client.chat_completion_stream({
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "max_tokens": 32,
}):
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Capabilities

Field	Value
Context window	8192 tokens
Streaming	Yes (SSE via `stream: true`)
Function calling / tools	Yes (Hermes-style; see “Function calling” below)
Structured output (`response_format`)	Yes — `json_object` and `json_schema` (vLLM `structured_outputs` constrained decoding)
Logprobs	No (vllm_backend exposes only `text_output` over Triton; not surfaced)
Sampling controls	`temperature`, `top_p`, `top_k`, `min_p`, `frequency_penalty`, `presence_penalty`, `repetition_penalty`, `seed`, `stop`
Reasoning (“thinking”) mode	Off by default; opt in via `"enable_thinking": true` in the request body

Function calling

Pass OpenAI-shape tools and the model returns a tool_calls array on the assistant message (or as a delta.tool_calls chunk when streaming). The gateway speaks Qwen’s Hermes tool-call template under the hood.

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Whats the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

const reply = await client.chatCompletion({
  model: "qwen-3.5-27b",
  messages: [{ role: "user", content: "What's the weather in Tokyo?" }],
  tools: [{
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather in a city",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"]
      }
    }
  }],
  tool_choice: "auto",
});
const call = reply.choices[0].message.tool_calls?.[0];
// call.function.name === "get_weather"
// JSON.parse(call.function.arguments) === { city: "Tokyo" }

tool_choice accepts "auto" (model decides), "none" (force plain text), "required" (force a tool call), or { "type": "function", "function": { "name": "<tool>" } } to force a specific tool. After invoking the tool yourself, append a role: "tool" message containing the result and re-call the model:

{
  "model": "qwen-3.5-27b",
  "messages": [
    {"role": "user", "content": "Weather in Tokyo?"},
    {"role": "assistant", "tool_calls": [
      {"id": "call_abc", "type": "function",
       "function": {"name": "get_weather", "arguments": "{\"city\":\"Tokyo\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc",
     "content": "{\"temp_c\": 22, \"sky\": \"sunny\"}"}
  ]
}

Structured output (JSON mode)

Use response_format to force the model to emit valid JSON. Backed server-side by vLLM’s structured_outputs constrained decoding, so the output is guaranteed to parse.

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Give me a JSON object describing a cat."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age_years": {"type": "integer"},
            "color": {"type": "string"}
          },
          "required": ["name", "age_years", "color"]
        }
      }
    }
  }'

{"type": "json_object"} accepts any valid JSON; json_schema constrains it to your schema.

Reasoning mode

Qwen3.5 ships with a “thinking” mode that emits a <think>...</think> reasoning trace before the user-visible answer. PolarGrid’s /v1/chat/completions endpoint disables this by default to keep first-token latency low. The 27B variant runs the same toggle, with deeper reasoning quality at the cost of longer generation time. To enable thinking on a per-request basis:

{
  "model": "qwen-3.5-27b",
  "messages": [{"role": "user", "content": "..."}],
  "enable_thinking": true
}

Model identifier

Call this model with the canonical id qwen-3.5-27b at all inference endpoints (/v1/chat/completions, /v1/completions). The HuggingFace repo id Qwen/Qwen3.5-27B-FP8 is accepted at /v1/models/load for hot-loading purposes but does not resolve at inference time — use the canonical id for chat and completions calls.

Notes

License: Apache 2.0 (no auth required to pull weights).
Native FP8 — no runtime quantization step at load.
VRAM is tight: a single 46 GB L40S can host this model or the voice stack, not both. Multi-GPU edges pin 27B to its own GPU; see backend/edge-production-setup/CLAUDE.md for the layout matrix.

Getting Started

Guides

Available Models

Headline benchmark

How this compares

Quickstart

Capabilities

Function calling

Structured output (JSON mode)

Reasoning mode

Model identifier

Notes

See also

​Headline benchmark

​How this compares

​Quickstart

​Capabilities

​Function calling

​Structured output (JSON mode)

​Reasoning mode

​Model identifier

​Notes

​See also

Headline benchmark

How this compares

Quickstart

Capabilities

Function calling

Structured output (JSON mode)

Reasoning mode

Model identifier

Notes

See also