Loading
Loading
Point a HuggingFace model at a cloud GPU and get an OpenAI-compatible endpoint that physically cannot be overflowed. Six independent clamps sit between every request and the KV cache. No DevOps, no 3am pager.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.aitherium.com/v1",
api_key="aither_sk_...",
)
r = client.chat.completions.create(
model="gemma-4-32b",
messages=[...],
)drop-in for LangChain · LlamaIndex · Cline · Continue · Cursor
One oversized request exhausts the KV cache. The process dies mid-generation and takes the GPU with it.
A dead worker needs SSH, a process kill, and re-provisioning — at whatever hour it happened to fall over.
When one thing breaks, everything stops. No fallback, no slower-but-alive path to keep serving.
IDE agents speak OpenAI. Raw vLLM speaks almost-OpenAI. The gap surfaces as intermittent, hard-to-trace failures.
Every request descends through six independent layers before it reaches the GPU. Any one of them alone prevents a crash; stacked, they make context overflow arithmetically impossible. The rail bleeds from overflow to safe as pressure is shed, layer by layer.
Output tokens are clamped to 60% of the context window on every request. No flag disables it.
Input is measured against the remaining budget; oversized system prompts are trimmed while the user message is preserved intact.
Instead of a blunt cut, lines are scored by importance and the densest content is kept. Pure heuristic — microseconds, no model call.
A 15-second loop watches GPU memory. Below threshold, low-priority models offload to cloud; when it recovers, they return.
Every backend names its fallback. vLLM down → cloud. Cloud down → Ollama on CPU, which cannot OOM. The chain always terminates.
Health checks catch a dead GPU within 30 seconds and provision a replacement. No SSH, no pager at 3am.
FP16 KV cache burns 512 bytes per token per layer. TurboQuant packs it with 4-bit vector quantization and fused Triton kernels — under 0.3% perplexity drift, undetectable in real conversation.
shipped as aither-kvcache on PyPI
Paste a model ID. VRAM sizing, vLLM config, GPU provisioning, health monitoring, and the OpenAI endpoint are handled for you.
Catalog or any HF ID. We compute VRAM and find the cheapest GPU across Vast, RunPod, and Lambda that fits.
One click provisions the GPU, installs vLLM, loads the model, wires TurboQuant, and streams every step over SSE.
Point your OpenAI client at the gateway. All six clamps are live. Watch usage, cost, and health from the dashboard.
Provision a GPU and get an endpoint in minutes.
Every clamp, with code from the production system.