> ## Documentation Index
> Fetch the complete documentation index at: https://polargrid.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen3.5 27B

> Retired — superseded by Qwen3.8 27B. Kept for reference.

> [!WARNING]
>
> **Retired.** `qwen-3.5-27b` is no longer served on any PolarGrid edge node — the
> fleet completed its move to [`qwen-3.8-27b`](/models/qwen-3.8-27b) on
> 2026-08-20, and requests for this id now return `404 model_not_loaded`. Update
> the `model` field in your requests to `qwen-3.8-27b`. This page is kept for
> reference; the benchmark figures below describe the retired model.

Qwen3.5 27B (`qwen-3.5-27b`) is a 27-billion-parameter text LLM served on PolarGrid edge nodes via Triton's `vllm_backend`. Weights ship pre-quantized to FP8 (\~28 GB VRAM) and load directly on PolarGrid's Blackwell edge GPUs without runtime requantization.

* **HF repo:** [`Qwen/Qwen3.5-27B-FP8`](https://huggingface.co/Qwen/Qwen3.5-27B-FP8)
* **Modality:** Text LLM
* **Backend:** Triton `vllm` (LLM pod)
* **Available regions:** fleet-wide — see [Model availability](/guides/model-availability.md)

## Headline benchmark

We publish **two** numbers side by side. End-to-end is what your application actually experiences (request → response, network included). Server-only is what the GPU spends on inference (apples-to-apples vs centralized providers' published "inference-only" figures). The gap is the latency PolarGrid's `yvr-02` PoP eliminates by being at the edge.

| Measurement | TTFT p50 | TTFT p95 | Throughput p50 |
| - | - | - | - |
| **End-to-end (with network)** | **177 ms** | 243 ms | **29.0 tok/s** |
| **Server-only (no network)** | **71 ms** | 131 ms | — |
| *Network overhead (e2e − server)* | *106 ms* | — | — |

*Bench: 100 streaming chat-completion runs against `https://api.yvr-02.edge.polargrid.ai`, captured 2026-07-09 from a local macOS host over the public internet. End-to-end is client wall-clock; server-only is read from the gateway's `pg_metadata` SSE event (`inference_ttft_ms` / `inference_total_ms`). Reasoning mode off (default). CUDA graphs on (`enforce_eager=false`, the current fleet default). The yvr-02 node ships on RTX 6000 Pro Blackwell 96 GB — roughly 2.4× the tok/s and less than half the e2e total latency of the earlier yvr-01 (L40S-class) baseline at `benchmarks/qwen-2026-05-01/27b/llm_bench.json`. Raw runs: `benchmarks/yvr-02-2026-07-09/27b/llm_bench.json`.*

> **Apples-to-apples disclaimer.** Other providers usually publish only their server-side number; comparing it to our **server-only** row is the fair baseline. Our **end-to-end** row is what you'll see from a customer-side request because PolarGrid runs at the edge. The network row above shows exactly how much that's worth in milliseconds.

## How this compares

| Provider | TTFT p50 | Throughput p50 | Source |
| - | - | - | - |
| **PolarGrid `qwen-3.5-27b` on Blackwell** | **177 ms** e2e / **71 ms** server | **29.0 tok/s** | this card |
| Claude Sonnet 4.5 | \~1600 ms | 47.6 tok/s | [artificialanalysis.ai](https://artificialanalysis.ai/models/claude-4-5-sonnet) |
| gpt-4o | \~850 ms | 135 tok/s | [artificialanalysis.ai](https://artificialanalysis.ai/models/gpt-4o) |
| Cerebras (specialty silicon) | n/a | \~2100 tok/s | [cerebras.ai](https://www.cerebras.ai/blog/cerebras-inference-3x-faster) |
| Groq with speculative decoding | n/a | \~1665 tok/s | [artificialanalysis.ai](https://artificialanalysis.ai/models/llama-3-3-instruct-70b/providers) |

PolarGrid wins on TTFT end-to-end against frontier-reasoning providers because of edge proximity (106 ms p50 network leg vs centralized regions). PolarGrid is 4 to 70 times behind specialty silicon on raw throughput; RTX 6000 Pro Blackwell workstation FLOPS are below H100 and H200 datacenter FLOPS. CUDA graphs are enabled (`enforce_eager=false`, the current fleet default), which recovered about 25 ms of server TTFT versus the earlier eager-mode baseline at `benchmarks/qwen-2026-07-07/27b/llm_bench.json`.

## Quickstart

```bash cURL theme={null}
curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "stream": true,
    "max_tokens": 32
  }'
```

```typescript JavaScript theme={null}
import { PolarGrid } from "@polargrid/polargrid-sdk";

const client = await PolarGrid.create({ apiKey: process.env.POLARGRID_API_KEY });

for await (const chunk of client.chatCompletionStream({
  model: "qwen-3.5-27b",
  messages: [{ role: "user", content: "Say hi in one short sentence." }],
  maxTokens: 32,
})) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) process.stdout.write(content);
}
```

```python Python theme={null}
from polargrid import PolarGrid

client = await PolarGrid.create(api_key="pg_...")

async for chunk in client.chat_completion_stream({
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "max_tokens": 32,
}):
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
```

## Capabilities

| Field | Value |
| - | - |
| Context window | 8192 tokens |
| Streaming | Yes (SSE via `stream: true`) |
| Function calling / tools | Yes (Hermes-style; see "Function calling" below) |
| Structured output (`response_format`) | Yes — `json_object` and `json_schema` (vLLM `structured_outputs` constrained decoding) |
| Logprobs | No (vllm\_backend exposes only `text_output` over Triton; not surfaced) |
| Sampling controls | `temperature`, `top_p`, `top_k`, `min_p`, `frequency_penalty`, `presence_penalty`, `repetition_penalty`, `seed`, `stop` |
| Reasoning ("thinking") mode | Off by default; opt in via `"enable_thinking": true` in the request body |

## Function calling

Pass OpenAI-shape `tools` and the model returns a `tool_calls` array on the assistant message (or as a `delta.tool_calls` chunk when streaming). The gateway speaks Qwen's Hermes tool-call template under the hood.

```bash cURL theme={null}
curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Whats the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'
```

```typescript JavaScript theme={null}
const reply = await client.chatCompletion({
  model: "qwen-3.5-27b",
  messages: [{ role: "user", content: "What's the weather in Tokyo?" }],
  tools: [{
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather in a city",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"]
      }
    }
  }],
  tool_choice: "auto",
});
const call = reply.choices[0].message.tool_calls?.[0];
// call.function.name === "get_weather"
// JSON.parse(call.function.arguments) === { city: "Tokyo" }
```

`tool_choice` accepts `"auto"` (model decides), `"none"` (force plain text), `"required"` (force a tool call), or `{ "type": "function", "function": { "name": "<tool>" } }` to force a specific tool.

After invoking the tool yourself, append a `role: "tool"` message containing the result and re-call the model:

```json theme={null}
{
  "model": "qwen-3.5-27b",
  "messages": [
    {"role": "user", "content": "Weather in Tokyo?"},
    {"role": "assistant", "tool_calls": [
      {"id": "call_abc", "type": "function",
       "function": {"name": "get_weather", "arguments": "{\"city\":\"Tokyo\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc",
     "content": "{\"temp_c\": 22, \"sky\": \"sunny\"}"}
  ]
}
```

## Structured output (JSON mode)

Use `response_format` to force the model to emit valid JSON. Backed server-side by vLLM's `structured_outputs` constrained decoding, so the output is guaranteed to parse.

```bash theme={null}
curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.5-27b",
    "messages": [{"role": "user", "content": "Give me a JSON object describing a cat."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age_years": {"type": "integer"},
            "color": {"type": "string"}
          },
          "required": ["name", "age_years", "color"]
        }
      }
    }
  }'
```

`{"type": "json_object"}` accepts any valid JSON; `json_schema` constrains it to your schema.

### Reasoning mode

Qwen3.5 ships with a "thinking" mode that emits a `<think>...</think>` reasoning trace before the user-visible answer. PolarGrid's `/v1/chat/completions` endpoint disables this by default to keep first-token latency low. The 27B variant runs the same toggle, with deeper reasoning quality at the cost of longer generation time.

To enable thinking on a per-request basis:

```json theme={null}
{
  "model": "qwen-3.5-27b",
  "messages": [{"role": "user", "content": "..."}],
  "enable_thinking": true
}
```

## Model identifier

Call this model with the canonical id `qwen-3.5-27b` at all inference endpoints (`/v1/chat/completions`, `/v1/completions`). The HuggingFace repo id `Qwen/Qwen3.5-27B-FP8` is accepted at `/v1/models/load` for hot-loading purposes but does **not** resolve at inference time — use the canonical id for chat and completions calls.

## Notes

* License: [Apache 2.0](https://huggingface.co/Qwen/Qwen3.5-27B-FP8/blob/main/LICENSE) (no auth required to pull weights).
* Native FP8 — no runtime quantization step at load.
* VRAM is tight: a single 46 GB L40S can host this model **or** the voice stack, not both. Multi-GPU edges pin 27B to its own GPU; see `backend/edge-production-setup/CLAUDE.md` for the layout matrix.

## See also

* [Authentication](/authentication.md) — using your `pg_*` API key
* [`/v1/models`](/api-reference/models) — list all available models
* [`/v1/chat/completions`](/api-reference/chat-completions) — endpoint reference


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.