Qwen3.8 27B

Qwen3.8 27B (qwen-3.8-27b) is a 27-billion-parameter text LLM served on PolarGrid edge nodes via Triton’s vllm_backend. Weights ship pre-quantized to FP8 (~28 GB VRAM) and load directly on PolarGrid’s Blackwell edge GPUs without runtime requantization.

HF repo: Qwen/Qwen3.8-27B-FP8
Modality: Text LLM
Backend: Triton vllm (LLM pod)
Available regions: fleet-wide — see Model availability

Headline benchmark

We publish two numbers side by side. End-to-end is what your application actually experiences (request → response, network included). Server-only is what the GPU spends on inference (apples-to-apples vs centralized providers’ published “inference-only” figures). The gap is the latency PolarGrid’s yvr-02 PoP eliminates by being at the edge.

Measurement	TTFT p50	TTFT p95	Throughput p50
Server-only (no network)	72 ms	141 ms	29.5 tok/s
End-to-end (with network)	284 ms	363 ms	—
Network overhead (e2e − server)	210 ms	—	—

Bench: 60 streaming chat-completion runs (5 warmup, concurrency 1, max_tokens 48) against https://api.yvr-02.edge.polargrid.ai, captured 2026-08-19 from a macOS host over the public internet, paced at 1.2 req/s to stay under the per-key rate limit. Server-only is read from the gateway’s pg_metadata SSE event (inference_ttft_ms); end-to-end is client wall-clock. Reasoning mode off (default). CUDA graphs on (enforce_eager=false). Raw runs: benchmarks/fleet-2026-08-18-qwen-3.8/yvr-02-c1.json. Read the server-only row, not the end-to-end row, when comparing against the Qwen3.5 27B card. The two captures ran from different client locations, so their network legs differ (210 ms here vs 106 ms there) even though server-side latency is effectively identical. The end-to-end number tells you what that benchmark host saw, not what the model got slower at. Fleet-wide, not one node. All 13 production nodes were measured (780 runs, zero errors). The eleven 2-GPU nodes land at 87–91 ms server TTFT p50 with 30.8–31.4 tok/s decode — a 4 ms spread across eleven geographically separate machines. The two 4-GPU nodes (yto-01, yvr-02, which also carry telephony/livekit workloads) sit at 72 ms with 29.4–29.5 tok/s: lower TTFT, slightly lower decode. That is a machine-class difference, not a per-node regression.

Apples-to-apples disclaimer. Other providers usually publish only their server-side number; comparing it to our server-only row is the fair baseline. Our end-to-end row is what you’ll see from a customer-side request because PolarGrid runs at the edge. The network row above shows exactly how much that’s worth in milliseconds.

How this compares

Provider	TTFT p50	Throughput p50	Source
PolarGrid `qwen-3.8-27b` on Blackwell	72 ms server (284 ms e2e from the bench host)	29.5 tok/s	this card
Claude Sonnet 4.5	~1600 ms	47.6 tok/s	artificialanalysis.ai
gpt-4o	~850 ms	135 tok/s	artificialanalysis.ai
Cerebras (specialty silicon)	n/a	~2100 tok/s	cerebras.ai
Groq with speculative decoding	n/a	~1665 tok/s	artificialanalysis.ai

PolarGrid wins on TTFT against frontier-reasoning providers because of edge proximity — the server does the work in tens of milliseconds and the client is close to it. PolarGrid remains 4 to 70 times behind specialty silicon on raw throughput; RTX 6000 Pro Blackwell workstation FLOPS are below H100 and H200 datacenter FLOPS. CUDA graphs are enabled (enforce_eager=false, the fleet default). Against the Qwen3.5 27B it replaced, the swap is latency-neutral at concurrency 1 and better under load. Measured on the staging canary 2026-08-17: 88 / 121 / 118 / 143 ms at c=1 / 8 / 16 / 32 for 3.8, versus 88 / 129 / 144 / 173 ms for 3.5. Same first-token latency for a single caller; the gap opens as concurrency rises.

Quickstart

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.8-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "stream": true,
    "max_tokens": 32
  }'

import { PolarGrid } from "@polargrid/polargrid-sdk";

const client = await PolarGrid.create({ apiKey: process.env.POLARGRID_API_KEY });

for await (const chunk of client.chatCompletionStream({
  model: "qwen-3.8-27b",
  messages: [{ role: "user", content: "Say hi in one short sentence." }],
  maxTokens: 32,
})) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) process.stdout.write(content);
}

from polargrid import PolarGrid

client = await PolarGrid.create(api_key="pg_...")

async for chunk in client.chat_completion_stream({
    "model": "qwen-3.8-27b",
    "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
    "max_tokens": 32,
}):
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Capabilities

Field	Value
Context window	262,144 tokens (256K)
Streaming	Yes (SSE via `stream: true`)
Function calling / tools	Yes (Hermes-style; see “Function calling” below)
Structured output (`response_format`)	Yes — `json_object` and `json_schema` (vLLM `structured_outputs` constrained decoding)
Logprobs	No (vllm_backend exposes only `text_output` over Triton; not surfaced)
Sampling controls	`temperature`, `top_p`, `top_k`, `min_p`, `frequency_penalty`, `presence_penalty`, `repetition_penalty`, `seed`, `stop`
Reasoning (“thinking”) mode	Off by default; opt in via `"enable_thinking": true` in the request body

Function calling

Pass OpenAI-shape tools and the model returns a tool_calls array on the assistant message (or as a delta.tool_calls chunk when streaming). The gateway speaks Qwen’s Hermes tool-call template under the hood.

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.8-27b",
    "messages": [{"role": "user", "content": "Whats the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

const reply = await client.chatCompletion({
  model: "qwen-3.8-27b",
  messages: [{ role: "user", content: "What's the weather in Tokyo?" }],
  tools: [{
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather in a city",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"]
      }
    }
  }],
  tool_choice: "auto",
});
const call = reply.choices[0].message.tool_calls?.[0];
// call.function.name === "get_weather"
// JSON.parse(call.function.arguments) === { city: "Tokyo" }

tool_choice accepts "auto" (model decides), "none" (force plain text), "required" (force a tool call), or { "type": "function", "function": { "name": "<tool>" } } to force a specific tool. After invoking the tool yourself, append a role: "tool" message containing the result and re-call the model:

{
  "model": "qwen-3.8-27b",
  "messages": [
    {"role": "user", "content": "Weather in Tokyo?"},
    {"role": "assistant", "tool_calls": [
      {"id": "call_abc", "type": "function",
       "function": {"name": "get_weather", "arguments": "{\"city\":\"Tokyo\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc",
     "content": "{\"temp_c\": 22, \"sky\": \"sunny\"}"}
  ]
}

Structured output (JSON mode)

Use response_format to force the model to emit valid JSON. Backed server-side by vLLM’s structured_outputs constrained decoding, so the output is guaranteed to parse.

curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.8-27b",
    "messages": [{"role": "user", "content": "Give me a JSON object describing a cat."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age_years": {"type": "integer"},
            "color": {"type": "string"}
          },
          "required": ["name", "age_years", "color"]
        }
      }
    }
  }'

{"type": "json_object"} accepts any valid JSON; json_schema constrains it to your schema.

Reasoning mode

Qwen3.8 ships with a “thinking” mode that emits a <think>...</think> reasoning trace before the user-visible answer. PolarGrid’s /v1/chat/completions endpoint disables this by default to keep first-token latency low. The 27B variant runs the same toggle, with deeper reasoning quality at the cost of longer generation time. To enable thinking on a per-request basis:

{
  "model": "qwen-3.8-27b",
  "messages": [{"role": "user", "content": "..."}],
  "enable_thinking": true
}

Model identifier

Call this model with the canonical id qwen-3.8-27b at all inference endpoints (/v1/chat/completions, /v1/completions). The HuggingFace repo id Qwen/Qwen3.8-27B-FP8 is accepted at /v1/models/load for hot-loading purposes but does not resolve at inference time — use the canonical id for chat and completions calls.

Notes

License: Apache 2.0 (no auth required to pull weights).
Native FP8 — no runtime quantization step at load.
VRAM is tight: a single 46 GB L40S can host this model or the voice stack, not both. Multi-GPU edges pin 27B to its own GPU; see backend/edge-production-setup/CLAUDE.md for the layout matrix.

Getting Started

Guides

Available Models

Headline benchmark

How this compares

Quickstart

Capabilities

Function calling

Structured output (JSON mode)

Reasoning mode

Model identifier

Notes

See also

​Headline benchmark

​How this compares

​Quickstart

​Capabilities

​Function calling

​Structured output (JSON mode)

​Reasoning mode

​Model identifier

​Notes

​See also

Headline benchmark

How this compares

Quickstart

Capabilities

Function calling

Structured output (JSON mode)

Reasoning mode

Model identifier

Notes

See also