Context Overflow Crashed Your GPU? We Made That Mathematically Impossible.

A post went viral this week. A developer deployed Gemma 4 32B on a Vast.ai H100 for $1.50/hr. It worked great -- until a conversation hit 32,500 tokens. The context window overflowed. The GPU ran out of memory. The machine bricked. He had to SSH in, kill the process, re-provision, and start over.

His conclusion: "The infrastructure problem is solved. The developer experience problem is not."

He is right about the diagnosis. But we solved the developer experience problem six months ago. We just never wrote it up. This is that write-up.

The Five Failure Modes

What happened to Dan was not one failure. It was five failures that compounded because no layer caught any of them:

1. Context overflow -- The application sent more tokens than the model's context window could hold. vLLM tried to allocate KV cache memory that did not exist.
2. No graceful degradation -- Instead of truncating, compressing, or rejecting the request, the process crashed with an OOM error.
3. GPU bricked -- The crash left GPU memory in a corrupted state. The vLLM process could not restart without a full machine reboot.
4. No auto-recovery -- Nobody was watching. The machine sat dead until the developer noticed and manually intervened.
5. Tooling incompatibility -- His IDE agent (Cline) could not handle a self-hosted endpoint. It expected OpenAI-compatible responses and got connection errors.

Each of these is an engineering problem with an engineering solution. Here is how we eliminate all five.

Layer 1: The Hard Cap (Context Clamping)

The first layer is the simplest and most important. Every LLM request passes through our AitherLLMQueue, which enforces an absolute rule:

max_tokens can never exceed 60% of the context window. No exceptions.

def _safe_clamp_max_tokens(self, max_tokens, context_window, messages=None, prompt=None):
    # Rule 1: Hard cap at 60% of context window
    hard_cap = max(256, int(context_window * 0.60))
    clamped = min(max_tokens, hard_cap)

    # Rule 2: Estimate prompt tokens and tighten further
    approx_tokens = len(prompt_text) // 4
    if approx_tokens > 0:
        safe_max = max(256, context_window - approx_tokens - 512)
        clamped = min(clamped, safe_max)

    return clamped

If a model has a 32K context window, max output tokens is capped at 19,200. If the input prompt is estimated at 20K tokens, max output tightens further to 32K - 20K - 512 = 11,488. The math makes overflow impossible because input + output can never exceed the window.

This runs on every request. There is no way to bypass it. The caller can ask for 100,000 output tokens; they will get 19,200.

Why 60%? Because token estimation is imprecise. The tokenizer sees characters; the model sees tokens. A 512-token safety margin handles the estimation error. In practice, the soft cap (Layer 2) usually kicks in first, so the hard cap is a backstop for the backstop.

Layer 2: Input Truncation (Message Fitting)

The hard cap protects output. This layer protects input.

Before every inference call, we estimate the total input token count and compare it against the available budget:

def _truncate_messages_to_fit(messages, context_window, max_output_tokens):
    approx_input_tokens = int(total_chars / 3.2) + len(messages) * 4
    input_budget = context_window - max_output_tokens - 512

    if approx_input_tokens <= input_budget:
        return messages  # fits, no action needed

    overflow_chars = int(overflow_tokens * 3.2)
    # Truncate the system message (always the largest)
    messages[0]["content"] = messages[0]["content"][:len(sys_content) - overflow_chars]

The key insight: we truncate the system message, not the user's input. The system message is always the largest (it contains identity, context, memories, capabilities) and is the most compressible. The user's actual question is never touched.

This is applied at three separate call sites in the queue -- the primary vLLM path, the staged reasoning path, and the fallback path. Every road through the system hits this check.

Layer 3: Extractive Compression

Truncation cuts text at a boundary and throws the rest away. That is crude. Layer 3 replaces truncation with extractive compression: instead of cutting the last N characters, it scores every line by importance and keeps the highest-scoring ones.

def _synthesize_for_budget(text, budget_tokens, source_label, query=None):
    # Score each line by structural importance
    for i, line in enumerate(lines):
        score = _score_line(line, i, total_lines)
        if query:
            # Boost lines that share terms with the current query
            overlap = len(set(query_terms) & set(line_terms))
            score += overlap * 2.0
        scored_lines.append((score, i, line))

    # Keep highest-scoring lines until budget is filled
    scored_lines.sort(reverse=True)
    kept = []
    for score, idx, line in scored_lines:
        if running_tokens >= budget_tokens:
            break
        kept.append((idx, line))

    # Preserve original ordering
    kept.sort()

The scoring function prioritizes:

• Section headers and structural markers (score +5.0)
• Code signatures -- def, class, function (score +3.0)
• Key-value pairs -- short lines containing : (score +2.0)
• Content matching the current query (score +2.0 per overlapping term)
• Positional bias: first and last 10% of content (score +1.0)

This is the fast path -- no LLM call, pure heuristic. It runs in microseconds. The result is a compressed version of the system prompt that preserves the most important information while fitting within the token budget. A 40K-token system prompt compressed to 8K tokens retains all headers, code references, and query-relevant facts. The verbose explanations between them are what gets dropped.

Layer 4: VRAM Placement Manager

Layers 1-3 prevent the request from overflowing the context window. Layer 4 prevents the GPU from running out of VRAM in the first place.

The VRAMPlacementManager continuously monitors GPU memory and dynamically moves models between local GPU and cloud:

Priority (highest = last to offload, first to bring back):
  P0  orchestrator   ALWAYS local -- handles all chat traffic
  P1  vision         Local preferred -- needed for multimodal
  P2  coding         Offloadable when VRAM is tight
  P3  reasoning      First to offload -- most VRAM hungry (~10 GB)

When VRAM drops below 6 GB free (configurable), the placement manager starts offloading low-priority models to cloud backends. When VRAM recovers above 12 GB (hysteresis to prevent flapping), it brings them back.

The evaluation runs every 15 seconds. There is a 120-second cooldown between placement changes to prevent oscillation. A model must be absent for 60 seconds before it is eligible for restoration.

This means if you are running Gemma 4 32B as your orchestrator and someone starts a ComfyUI image generation job that consumes 8 GB of VRAM, the reasoning model automatically offloads to cloud. When the image job finishes, the reasoning model comes back. Nobody has to think about it.

Layer 5: Backend Failover Chain

If a backend crashes despite all the above protections, the system does not stop. It fails over.

Every backend has a defined fallback:

vllm_swap        → vllm_cloud       → vllm          → ollama
vllm_orchestrator → vllm
vllm_reasoning   → vllm_cloud_reasoning → vllm
vllm_vision      → vllm
vllm_coding      → vllm
ollama           → vllm

The failover is health-aware. Each backend has a circuit breaker that tracks consecutive failures. When a backend fails three times in a row, it is marked unhealthy and all requests route to the next in the chain. Health checks run continuously in the background; when the backend recovers, the circuit breaker resets and traffic flows back.

The key property: the fallback chain always terminates. The final link is Ollama running on CPU. It is slow -- maybe 5 tokens/second for a 3B model -- but it never OOMs because CPU memory is measured in hundreds of gigabytes.

A user whose H100 crashes mid-conversation sees a slower response on the next message, not an error. The system degrades gracefully, serves the response from a fallback backend, and meanwhile the primary backend is being recovered.

Layer 6: GPU Cluster Manager (Auto-Recovery)

The GPUClusterManager orchestrates three pools of GPU resources:

GPUClusterManager
├── GPUPool (Local)     -- RTX 5090/4080, priority 1
├── GPUPool (Remote)    -- Mesh nodes via ExoNodes, priority 2
├── GPUPool (Cloud)     -- Vast.ai/RunPod on-demand, priority 3
└── Unified Router      -- Health-aware, VRAM-fit matching, failover

Each pool has a health endpoint that is polled continuously. When a cloud GPU crashes (the exact scenario from the viral post), the cluster manager:

1. Detects the health check failure within 30 seconds
2. Marks the pool as DRAINING and migrates in-flight jobs
3. Provisions a replacement GPU from the same provider (or a different one if the provider is having issues)
4. Re-registers the new backend URL in the LLM queue
5. Emits a GPU_RECOVERED Flux event so monitoring dashboards update

The entire recovery happens without human intervention. The developer does not SSH into anything. They do not re-provision manually. The system notices the crash and fixes itself.

The OpenAI Compatibility Layer

The fifth failure Dan hit was tooling. His IDE agent could not talk to a raw vLLM endpoint. This is the easiest problem to solve but the one most self-hosted setups ignore.

Our inference proxy speaks the OpenAI /v1/chat/completions protocol:

client = OpenAI(
    base_url="https://gateway.aitherium.com/v1",
    api_key="aither_sk_..."
)
response = client.chat.completions.create(
    model="gemma-4-32b",
    messages=[{"role": "user", "content": "Hello"}]
)

Same request format. Same response format. Same streaming SSE protocol. Any tool that works with OpenAI works with our gateway. Cline, Continue, Cursor, LangChain, LlamaIndex -- they all just work because they all speak the same protocol.

The proxy handles model name mapping (external names to internal backends), authentication via API keys, metered billing, and request routing to the correct vLLM worker. It also applies all six context safety layers before the request reaches the GPU.

TurboQuant: 3.8x More Context in the Same VRAM

The context overflow problem has a second dimension: how much context can you actually fit before VRAM runs out?

Standard FP16 KV cache uses 512 bytes per token per layer. For a 32-layer model with 32K context, that is 512 MB just for attention state. Increase context to 128K and you need 2 GB of KV cache alone.

TurboQuant compresses KV cache to ~34 KB per token using 4-bit vector quantization with fused Triton kernels that read compressed data directly -- no decompression step. The same 32K context that consumed 512 MB now uses 135 MB. The same VRAM that held 32K tokens now holds ~122K tokens.

This is not an approximation technique. The quantization is applied to the key and value tensors after attention computation, using learned codebooks that minimize reconstruction error. In practice, perplexity increases by less than 0.3% compared to FP16 -- undetectable in real conversations.

The practical impact: Dan's Gemma 4 32B would have had a 122K effective context window instead of 32K. His 32,500-token conversation would have used 26% of the available budget, not 101%.

The Request Pool

All of these layers sit inside a priority queue that manages concurrent access to the GPU. The queue is not just a FIFO -- it is a multi-priority system with per-backend concurrency limits:

When all slots for a backend are full, new requests queue by priority: user-facing requests preempt agent requests, which preempt background neuron tasks. A scout exploring codebase documentation will yield its slot to a user who just typed a message.

This prevents the thundering herd problem that crashes self-hosted setups. If 8 agents all try to call the reasoning model simultaneously, 2 proceed and 6 queue. Nobody OOMs.

Why This Cannot Fail

Let us trace Dan's exact failure scenario through this system:

1. User sends a message that would create a 32,500-token context.
2. Layer 1 clamps max_tokens to 60% of the 32K window = 19,200.
3. Layer 2 estimates the input at 32,500 tokens, calculates input_budget = 32K - 19,200 - 512 = 12,288 tokens. The input exceeds the budget. The system message is truncated from ~30K tokens to ~10K tokens. Key headers, code references, and the user's actual question are preserved via extractive compression.
4. Layer 3 has already been applied during context assembly -- the 12-stage ContextPipeline built a prompt that fits within the model's budget before the request even reached the queue.
5. The request proceeds with ~10K input tokens and ~19K max output tokens. Total: 29K. Context window: 32K. Headroom: 3K tokens. No overflow possible.
6. If the GPU were under VRAM pressure (maybe someone ran ComfyUI), Layer 4 would have already offloaded the reasoning model and this request would route to the orchestrator or cloud backend.
7. If vLLM crashed for any other reason, Layer 5 would route to the next healthy backend.
8. If the entire cloud machine went down, Layer 6 would provision a replacement and recover.

At no point does the user see an error. At worst, they see a slower response or a slightly shorter system context. The conversation continues.

The Depth Router

There is one more piece that prevents context overflow before it starts: the depth router.

Not every message needs 32K tokens of context. "Hey, good morning" needs maybe 500 tokens. "Refactor the authentication module" might legitimately need 30K. Stuffing 32K tokens into a greeting is not just wasteful -- it increases the probability of hitting VRAM limits for no benefit.

The depth router classifies every incoming message into one of four depth levels:

A quick message gets a 500-token system prompt and responds in 200ms. A deep message gets the full 12-stage context pipeline, neuron context, memory recall, and routes to the reasoning model. The budget scales to the task.

This is not just a performance optimization. It is a safety mechanism. If 80% of messages are QUICK or BALANCED and only 20% are DEEP, the average VRAM utilization stays well below the ceiling. The system spends most of its time in the comfortable middle of the capacity curve, not at the edge.

The Uncomfortable Truth

None of this is rocket science. Every layer described in this post is straightforward engineering:

• Token counting is arithmetic.
• Extractive compression is sorting lines by score.
• VRAM monitoring is reading nvidia-smi.
• Failover chains are linked lists of URLs.
• Health checks are HTTP GETs on a timer.
• Priority queues are a heap with a comparator.

The reason self-hosted LLM setups crash is not that the safety engineering is hard. It is that nobody builds it. The default vLLM deployment has zero layers between the user's request and the GPU. If the request is too big, the GPU dies. If the GPU dies, the process dies. If the process dies, the user stares at a connection error until someone manually intervenes.

This is like shipping a car without seatbelts and blaming the driver when they go through the windshield. The fix is not "drive more carefully." The fix is seatbelts, airbags, crumple zones, ABS, and lane departure warnings. Defense in depth. Multiple independent layers, each of which can save you when the others fail.

That is what we built. Six layers. Any one of them prevents the crash Dan described. Together, they make it mathematically impossible.

Build It Yourself

If you are running a self-hosted LLM setup, here is the minimum viable safety stack:

1. Clamp max_tokens to 60% of context_window on every request. This is one line of code. Do it today.
2. Estimate input tokens and shrink max_tokens further when the input is large. Five lines of code.
3. Truncate the system message, not the user's message, when input exceeds the budget. Ten lines of code.
4. Monitor VRAM and stop accepting requests when free memory drops below a threshold. This prevents cascading failures.
5. Add one fallback backend. Even Ollama on CPU running a 3B model is better than a dead GPU.
6. Health-check your backends and route around failures automatically. A 10-line async loop that pings /health every 30 seconds.

You can implement the first three in an afternoon and eliminate 90% of context-related crashes. The remaining three take a week and eliminate the other 10%.

Or you can keep deploying raw vLLM and hoping your conversations stay short.