> ## Documentation Index
> Fetch the complete documentation index at: https://polargrid.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen3.8 27B

> Qwen3.8 27B text LLM on vLLM with native FP8 weights

Qwen3.8 27B (`qwen-3.8-27b`) is a 27-billion-parameter text LLM served on PolarGrid edge nodes via Triton's `vllm_backend`. Weights ship pre-quantized to FP8 (\~28 GB VRAM) and load directly on PolarGrid's Blackwell edge GPUs without runtime requantization.

* **HF repo:** [`Qwen/Qwen3.8-27B-FP8`](https://huggingface.co/Qwen/Qwen3.8-27B-FP8)
* **Modality:** Text LLM
* **Backend:** Triton `vllm` (LLM pod)
* **Available regions:** fleet-wide — see [Model availability](/guides/model-availability)

## Headline benchmark

We publish **two** numbers side by side. End-to-end is what your application actually experiences (request → response, network included). Server-only is what the GPU spends on inference (apples-to-apples vs centralized providers' published "inference-only" figures). The gap is the latency PolarGrid's `yvr-02` PoP eliminates by being at the edge.

| Measurement                       | TTFT p50  | TTFT p95 | Throughput p50 |
| --------------------------------- | --------- | -------- | -------------- |
| **Server-only (no network)**      | **72 ms** | 141 ms   | **29.5 tok/s** |
| **End-to-end (with network)**     | 284 ms    | 363 ms   | —              |
| *Network overhead (e2e − server)* | *210 ms*  | —        | —              |

*Bench: 60 streaming chat-completion runs (5 warmup, concurrency 1, `max_tokens` 48) against `https://api.yvr-02.edge.polargrid.ai`, captured 2026-08-19 from a macOS host over the public internet, paced at 1.2 req/s to stay under the per-key rate limit. Server-only is read from the gateway's `pg_metadata` SSE event (`inference_ttft_ms`); end-to-end is client wall-clock. Reasoning mode off (default). CUDA graphs on (`enforce_eager=false`). Raw runs: `benchmarks/fleet-2026-08-18-qwen-3.8/yvr-02-c1.json`.*

**Read the server-only row, not the end-to-end row, when comparing against the [Qwen3.5 27B card](/models/qwen-3.5-27b).** The two captures ran from different client locations, so their network legs differ (210 ms here vs 106 ms there) even though server-side latency is effectively identical. The end-to-end number tells you what *that* benchmark host saw, not what the model got slower at.

**Fleet-wide, not one node.** All 13 production nodes were measured (780 runs, zero errors). The eleven 2-GPU nodes land at **87–91 ms** server TTFT p50 with **30.8–31.4 tok/s** decode — a 4 ms spread across eleven geographically separate machines. The two 4-GPU nodes (`yto-01`, `yvr-02`, which also carry telephony/livekit workloads) sit at **72 ms** with **29.4–29.5 tok/s**: lower TTFT, slightly lower decode. That is a machine-class difference, not a per-node regression.

> **Apples-to-apples disclaimer.** Other providers usually publish only their server-side number; comparing it to our **server-only** row is the fair baseline. Our **end-to-end** row is what you'll see from a customer-side request because PolarGrid runs at the edge. The network row above shows exactly how much that's worth in milliseconds.

## How this compares

| Provider                                  | TTFT p50                                          | Throughput p50 | Source                                                                                         |
| ----------------------------------------- | ------------------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------- |
| **PolarGrid `qwen-3.8-27b` on Blackwell** | **72 ms** server (284 ms e2e from the bench host) | **29.5 tok/s** | this card                                                                                      |
| Claude Sonnet 4.5                         | \~1600 ms                                         | 47.6 tok/s     | [artificialanalysis.ai](https://artificialanalysis.ai/models/claude-4-5-sonnet)                |
| gpt-4o                                    | \~850 ms                                          | 135 tok/s      | [artificialanalysis.ai](https://artificialanalysis.ai/models/gpt-4o)                           |
| Cerebras (specialty silicon)              | n/a                                               | \~2100 tok/s   | [cerebras.ai](https://www.cerebras.ai/blog/cerebras-inference-3x-faster)                       |
| Groq with speculative decoding            | n/a                                               | \~1665 tok/s   | [artificialanalysis.ai](https://artificialanalysis.ai/models/llama-3-3-instruct-70b/providers) |

PolarGrid wins on TTFT against frontier-reasoning providers because of edge proximity — the server does the work in tens of milliseconds and the client is close to it. PolarGrid remains 4 to 70 times behind specialty silicon on raw throughput; RTX 6000 Pro Blackwell workstation FLOPS are below H100 and H200 datacenter FLOPS. CUDA graphs are enabled (`enforce_eager=false`, the fleet default).

**Against the Qwen3.5 27B it replaced, the swap is latency-neutral at concurrency 1 and better under load.** Measured on the staging canary 2026-08-17: 88 / 121 / 118 / 143 ms at c=1 / 8 / 16 / 32 for 3.8, versus 88 / 129 / 144 / 173 ms for 3.5. Same first-token latency for a single caller; the gap opens as concurrency rises.

## Quickstart

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "qwen-3.8-27b",
      "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
      "stream": true,
      "max_tokens": 32
    }'
  ```

  ```typescript JavaScript theme={null}
  import { PolarGrid } from "@polargrid/polargrid-sdk";

  const client = await PolarGrid.create({ apiKey: process.env.POLARGRID_API_KEY });

  for await (const chunk of client.chatCompletionStream({
    model: "qwen-3.8-27b",
    messages: [{ role: "user", content: "Say hi in one short sentence." }],
    maxTokens: 32,
  })) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) process.stdout.write(content);
  }
  ```

  ```python Python theme={null}
  from polargrid import PolarGrid

  client = await PolarGrid.create(api_key="pg_...")

  async for chunk in client.chat_completion_stream({
      "model": "qwen-3.8-27b",
      "messages": [{"role": "user", "content": "Say hi in one short sentence."}],
      "max_tokens": 32,
  }):
      content = chunk.choices[0].delta.content
      if content:
          print(content, end="", flush=True)
  ```
</CodeGroup>

## Capabilities

| Field                                 | Value                                                                                                                   |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Context window                        | 262,144 tokens (256K)                                                                                                   |
| Streaming                             | Yes (SSE via `stream: true`)                                                                                            |
| Function calling / tools              | Yes (Hermes-style; see "Function calling" below)                                                                        |
| Structured output (`response_format`) | Yes — `json_object` and `json_schema` (vLLM `structured_outputs` constrained decoding)                                  |
| Logprobs                              | No (vllm\_backend exposes only `text_output` over Triton; not surfaced)                                                 |
| Sampling controls                     | `temperature`, `top_p`, `top_k`, `min_p`, `frequency_penalty`, `presence_penalty`, `repetition_penalty`, `seed`, `stop` |
| Reasoning ("thinking") mode           | Off by default; opt in via `"enable_thinking": true` in the request body                                                |

## Function calling

Pass OpenAI-shape `tools` and the model returns a `tool_calls` array on the assistant message (or as a `delta.tool_calls` chunk when streaming). The gateway speaks Qwen's Hermes tool-call template under the hood.

<CodeGroup>
  ```bash cURL theme={null}
  curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "qwen-3.8-27b",
      "messages": [{"role": "user", "content": "Whats the weather in Tokyo?"}],
      "tools": [{
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get the current weather in a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }],
      "tool_choice": "auto"
    }'
  ```

  ```typescript JavaScript theme={null}
  const reply = await client.chatCompletion({
    model: "qwen-3.8-27b",
    messages: [{ role: "user", content: "What's the weather in Tokyo?" }],
    tools: [{
      type: "function",
      function: {
        name: "get_weather",
        description: "Get the current weather in a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"]
        }
      }
    }],
    tool_choice: "auto",
  });
  const call = reply.choices[0].message.tool_calls?.[0];
  // call.function.name === "get_weather"
  // JSON.parse(call.function.arguments) === { city: "Tokyo" }
  ```
</CodeGroup>

`tool_choice` accepts `"auto"` (model decides), `"none"` (force plain text), `"required"` (force a tool call), or `{ "type": "function", "function": { "name": "<tool>" } }` to force a specific tool.

After invoking the tool yourself, append a `role: "tool"` message containing the result and re-call the model:

```json theme={null}
{
  "model": "qwen-3.8-27b",
  "messages": [
    {"role": "user", "content": "Weather in Tokyo?"},
    {"role": "assistant", "tool_calls": [
      {"id": "call_abc", "type": "function",
       "function": {"name": "get_weather", "arguments": "{\"city\":\"Tokyo\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_abc",
     "content": "{\"temp_c\": 22, \"sky\": \"sunny\"}"}
  ]
}
```

## Structured output (JSON mode)

Use `response_format` to force the model to emit valid JSON. Backed server-side by vLLM's `structured_outputs` constrained decoding, so the output is guaranteed to parse.

```bash theme={null}
curl https://api.yto-01.edge.polargrid.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-3.8-27b",
    "messages": [{"role": "user", "content": "Give me a JSON object describing a cat."}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age_years": {"type": "integer"},
            "color": {"type": "string"}
          },
          "required": ["name", "age_years", "color"]
        }
      }
    }
  }'
```

`{"type": "json_object"}` accepts any valid JSON; `json_schema` constrains it to your schema.

### Reasoning mode

Qwen3.8 ships with a "thinking" mode that emits a `<think>...</think>` reasoning trace before the user-visible answer. PolarGrid's `/v1/chat/completions` endpoint disables this by default to keep first-token latency low. The 27B variant runs the same toggle, with deeper reasoning quality at the cost of longer generation time.

To enable thinking on a per-request basis:

```json theme={null}
{
  "model": "qwen-3.8-27b",
  "messages": [{"role": "user", "content": "..."}],
  "enable_thinking": true
}
```

## Model identifier

Call this model with the canonical id `qwen-3.8-27b` at all inference endpoints (`/v1/chat/completions`, `/v1/completions`). The HuggingFace repo id `Qwen/Qwen3.8-27B-FP8` is accepted at `/v1/models/load` for hot-loading purposes but does **not** resolve at inference time — use the canonical id for chat and completions calls.

## Notes

* License: [Apache 2.0](https://huggingface.co/Qwen/Qwen3.8-27B-FP8/blob/main/LICENSE) (no auth required to pull weights).
* Native FP8 — no runtime quantization step at load.
* VRAM is tight: a single 46 GB L40S can host this model **or** the voice stack, not both. Multi-GPU edges pin 27B to its own GPU; see `backend/edge-production-setup/CLAUDE.md` for the layout matrix.

## See also

* [Authentication](/authentication) — using your `pg_*` API key
* [`/v1/models`](/api-reference/models) — list all available models
* [`/v1/chat/completions`](/api-reference/chat-completions) — endpoint reference