Managing Context in Long Conversations

Stay under the model context window in multi-turn and voice agent workloads

Managing Context in Long Conversations

Every LLM on PolarGrid enforces a hard context limit covering the system prompt, the full message history, and the requested completion. For qwen-3.8-27b it is 262,144 tokens (256K). There is no server-side truncation: when a request exceeds the limit, the API returns:

{
  "error": {
    "message": "Request exceeds the model's context window of 262144 tokens. Reduce the prompt or conversation history (for example, truncate or summarize older turns) and retry.",
    "type": "invalid_request_error"
  }
}

A retry without shrinking the request can never succeed — treat this 400 as a signal to compact history, not as a transient failure.

Why this matters for voice agents

Voice sessions accumulate context fast: every user turn and assistant reply joins the history. A 256K window is unlikely to be exhausted by a single call, so the pressure here is latency and cost, not the hard cap — time-to-first-token grows with prompt length, and you pay for every input token on every turn. Compact history because it keeps the agent responsive and cheap, not because the session is about to die.

If you pin a lower max_model_len on a node, or move to a model with a smaller window, the hard cap becomes the binding constraint again — the patterns below cover both cases.

Rule of thumb for budgeting

English text averages roughly 4 characters (≈0.75 words) per token. qwen-3.8-27b’s 256K window is far larger than a voice conversation needs, so budget for responsiveness rather than to the ceiling:

Slice	Suggested budget
System prompt	≤ 1,000 tokens
Reserved for the reply (max_tokens)	512–1,024 tokens
Conversation history	~6,000 tokens (≈ 45 average voice turns) before compacting

These are latency targets, not limits. Track an estimate as you go (len(text) / 4 is adequate) and compact when history crosses your budget. If you are running against a genuinely small window, compact at ~80% of the hard limit rather than reacting to the 400.

Pattern 1: Sliding window (simplest, fits most voice agents)

Keep the system prompt plus the most recent N turns; drop the oldest user/assistant pairs first.

MAX_HISTORY_TOKENS = 6000

def estimate_tokens(messages):
    return sum(len(m["content"]) // 4 for m in messages)

def windowed(system_prompt, history):
    msgs = list(history)
    while msgs and estimate_tokens(msgs) > MAX_HISTORY_TOKENS:
        # Drop the oldest exchange; never drop the system prompt.
        msgs = msgs[2:] if len(msgs) >= 2 else []
    return [{"role": "system", "content": system_prompt}, *msgs]

Good enough whenever the conversation’s relevant state lives in recent turns — appointment booking, support triage, order taking.

Pattern 2: Running summary (when early context must survive)

When facts from early in the call matter at the end (caller name, account details, the original problem statement), fold older turns into a compact summary instead of dropping them:

1. When history crosses your threshold, send the oldest turns to the LLM with a “summarize the key facts in under 100 words” instruction.
2. Replace those turns with a single system-adjacent message: {"role": "system", "content": "Summary of earlier conversation: ..."}.
3. Keep the most recent turns verbatim.

This costs one extra LLM call per compaction but caps history growth permanently. For voice, run the compaction during the user’s speaking turn so it never adds response latency.

Pattern 3: Structured state instead of raw history

Voice agents that collect fields (name, phone, address, intent) often don’t need the transcript at all — extract entities into a structured object as you go and prompt with the state object plus only the last 2-3 turns. Smallest possible context, immune to call length.

Handling the overflow error defensively

Even with budgeting, guard the call:

try:
    response = client.chat.completions.create(model="qwen-3.8-27b", messages=msgs, max_tokens=512)
except APIError as e:
    if "context window" in str(e):
        msgs = compact(msgs)   # your sliding-window or summary step
        response = client.chat.completions.create(model="qwen-3.8-27b", messages=msgs, max_tokens=512)
    else:
        raise

Do not put context-overflow 400s through generic retry/backoff — they are deterministic.

Per-model limits

Check the model card for each model’s context window (Models); the limit also applies to /v1/completions prompts. For workloads that genuinely need very long context, contact us about enterprise configurations.