Managing Context in Long Conversations

Stay under the model context window in multi-turn and voice agent workloads

Managing Context in Long Conversations

Every LLM on PolarGrid enforces a hard context limit — for qwen-3.5-27b it is 8,192 tokens covering the system prompt, the full message history, and the requested completion. There is no server-side truncation: when a request exceeds the limit, the API returns:

{
  "error": {
    "message": "Request exceeds the model's context window of 8192 tokens. Reduce the prompt or conversation history (for example, truncate or summarize older turns) and retry.",
    "type": "invalid_request_error"
  }
}

A retry without shrinking the request can never succeed — treat this 400 as a signal to compact history, not as a transient failure.

Why this matters for voice agents

Voice sessions accumulate context fast: every user turn and assistant reply joins the history, and a long support call can cross 8,192 tokens mid-conversation. If your orchestration layer doesn’t manage the window, the session dies at its longest — and most engaged — moment. Build context management in from the first prototype.

Rule of thumb for budgeting

English text averages roughly 4 characters (≈0.75 words) per token. A practical budget for qwen-3.5-27b:

Slice	Budget
System prompt	≤ 1,000 tokens
Reserved for the reply (max_tokens)	512–1,024 tokens
Conversation history	whatever remains (~6,000 tokens ≈ 45 average voice turns)

Track an estimate as you go (len(text) / 4 is adequate) and compact before you hit the wall — at ~80% of the limit — rather than reacting to the 400.

Pattern 1: Sliding window (simplest, fits most voice agents)

Keep the system prompt plus the most recent N turns; drop the oldest user/assistant pairs first.

MAX_HISTORY_TOKENS = 6000

def estimate_tokens(messages):
    return sum(len(m["content"]) // 4 for m in messages)

def windowed(system_prompt, history):
    msgs = list(history)
    while msgs and estimate_tokens(msgs) > MAX_HISTORY_TOKENS:
        # Drop the oldest exchange; never drop the system prompt.
        msgs = msgs[2:] if len(msgs) >= 2 else []
    return [{"role": "system", "content": system_prompt}, *msgs]

Good enough whenever the conversation’s relevant state lives in recent turns — appointment booking, support triage, order taking.

Pattern 2: Running summary (when early context must survive)

When facts from early in the call matter at the end (caller name, account details, the original problem statement), fold older turns into a compact summary instead of dropping them:

1. When history crosses your threshold, send the oldest turns to the LLM with a “summarize the key facts in under 100 words” instruction.
2. Replace those turns with a single system-adjacent message: {"role": "system", "content": "Summary of earlier conversation: ..."}.
3. Keep the most recent turns verbatim.

This costs one extra LLM call per compaction but caps history growth permanently. For voice, run the compaction during the user’s speaking turn so it never adds response latency.

Pattern 3: Structured state instead of raw history

Voice agents that collect fields (name, phone, address, intent) often don’t need the transcript at all — extract entities into a structured object as you go and prompt with the state object plus only the last 2-3 turns. Smallest possible context, immune to call length.

Handling the overflow error defensively

Even with budgeting, guard the call:

try:
    response = client.chat.completions.create(model="qwen-3.5-27b", messages=msgs, max_tokens=512)
except APIError as e:
    if "context window" in str(e):
        msgs = compact(msgs)   # your sliding-window or summary step
        response = client.chat.completions.create(model="qwen-3.5-27b", messages=msgs, max_tokens=512)
    else:
        raise

Do not put context-overflow 400s through generic retry/backoff — they are deterministic.

Per-model limits

Check the model card for each model’s context window (Models); the limit also applies to /v1/completions prompts. For workloads that genuinely need very long context, contact us about enterprise configurations.