Gemma 4 12B on the DGX Spark: Parallel Co-Residence and Perceive-Then-Reason

Two months ago we wrote about hot-swapping Google's Gemma 4 26B MoE onto a single GPU. That model was a reasoning challenger — text in, chain-of-thought out. On June 3rd, Google shipped something different enough that it broke our mental model of where a model lives in the stack: Gemma 4 12B, an encoder-free unified model.

Encoder-free means there's no separate vision tower bolted onto a language model. Image pixels, audio waveforms, and text tokens are all projected into the same embedding space and run through one decoder. The model doesn't call a vision model — it is one. 256K context, native audio and video, Apache 2.0, and small enough that Google's marketing says it "fits in 16 GB."

That last claim is true, and also a trap. Here's what it actually took to run it in AitherOS.

The serving reality nobody puts in the launch post

Three constraints surfaced before we wrote a line of config:

1. It needs a nightly vLLM. The encoder-free 12B architecture landed in vLLM PR #44429, which merged into main on June 3rd — the same day Google shipped the model — and is not in the 0.22.0 stable release. We confirmed it: stock vllm/vllm-openai:v0.22.0 lists Gemma4ForConditionalGeneration but no Gemma4Unified; the nightly (0.22.1rc1.dev166) has it. The model's config.json declares Gemma4UnifiedForConditionalGeneration / model_type: gemma4_unified and is built against transformers 5.10 — our production aither-vllm-tq image (vLLM 0.19.1) refuses it outright. You need a pinned nightly, and that transformers pin matters more than you'd think (see the AWQ section).
2. FP8 weight quantization produces gibberish. vLLM issue #39049 — dynamic FP8 on Gemma 4 emits garbage tokens. So FP8 is for the KV cache only; the weights stay bf16.
3. There is no AWQ-4bit build yet. The community has AWQ for the 26B and 31B, but not the new 12B. bf16 weights mean ~24 GB resident, not 16.

That third point is the whole story. "Fits in 16 GB" assumes a quant that doesn't exist. At bf16 it's a 24 GB model, and 24 GB is a very awkward number on a 32 GB card.

The 32 GB arithmetic that doesn't close

Our desktop runs an RTX 5090 — 32 GB. The steady state holds three things that all want VRAM:

8 + 24 = 32. The orchestrator and Gemma alone saturate the card — before a single pixel of image generation. You cannot hold all three. This isn't a tuning problem; it's a wall.

We have two honest ways through it, and we're building both.

Path 1 (now): parallel co-residence on the DGX Spark

The DGX Spark (GB10) has 128 GB of unified memory. That changes everything. Instead of fighting over 32 GB, Gemma 4 12B and our flagship reasoner, Qwen3.6-27B, both stay resident with room to spare.

The only catch: vLLM reserves memory as a fraction of the device (--gpu-memory-utilization), and our Qwen primary was set to grab 0.65 — about 83 GB — to hoard KV cache for long-context jobs. We trimmed it to make a roommate:

Trimming Qwen from 0.65 to 0.50 costs almost nothing, because Qwen already runs TurboQuant TQ3.5 KV compression — its effective context is large even from a smaller slot. The win is that both models are hot: no loading, no swapping, no cold-start penalty on any request.

Why parallel, and not swap?

The obvious alternative is to share one big slot: sleep Gemma, wake Qwen, and vice versa. vLLM supports it (--enable-sleep-mode offloads weights to host RAM and reloads on demand), and AitherOS already orchestrates exactly this dance on the 5090 for image generation.

For this workload, swap is the wrong tool — and the reason is the pipeline shape.

The whole point of running Gemma alongside Qwen is a two-stage flow where one feeds the other on every multimodal turn. If the two models share a slot, every single image request pays a wake penalty: reloading 24–27 GB from host RAM is 10–20 seconds of dead air. Swap makes sense when two models serve disjoint workloads at different times. It's a disaster when one model's output is the next model's input.

So: parallel for the pipeline, swap as a fallback lever. We still enabled sleep mode on the Gemma container — not for steady state, but so we can temporarily put Gemma to sleep and hand Qwen the full ~85 GB for an occasional giant-context job, then wake it.

The pipeline: Gemma perceives so Qwen can reason

Here's the architectural insight that makes the parallel layout worth the effort.

Qwen3.6-27B is itself multimodal — it has vision. So today, when an image arrives, AitherOS routes it straight to Qwen, and Qwen does everything: it processes the raw pixels and reasons about them. That's our flagship reasoner spending its capacity — and a DGX round-trip — on pixel parsing.

But we now have a model whose entire identity is perception. So we split the work:

orchestrator → image → Gemma 4 (perceive) → orchestrator → Qwen 3.6 (reason)

• Stage 1 — perceive. Gemma 4 12B receives the image and the user's actual question, and returns question-conditioned visual context as text. Not a generic caption — "extract everything relevant to answering: «the user's question»." Fast, local to the DGX.
• Stage 2 — reason. We strip the image, inject Gemma's extraction as text, and hand it to Qwen3.6 for deep reasoning over that context plus everything else in the conversation. Qwen never touches a pixel.

Because both models live on the same DGX, the handoff is a localhost hop — not a network round-trip. Qwen gets to be a pure reasoner. Gemma gets to be the eyes and ears.

Where this lives in the code

AitherOS has exactly one choke point for multimodal routing — AitherLLMQueue._submit_internal. Today it's a single-stage decision: if the request has an image and the target model can see, keep it; otherwise redirect to a vision-capable model. The two-stage pipeline slots in around that decision, gated on effort: a casual "what's in this image?" still goes to Gemma alone (involving Qwen would just add latency), while a genuine reasoning task at effort ≥ 7 triggers perceive-then-reason. That gate is the difference between smart and slow.

The model registry already cooperates: model_supports_vision() recognizes any gemma4* name as multimodal, so the new gemma4-12b served-name is treated as a first-class perceiver, not mis-routed.

Eyes and ears: the same split for audio and video

We called Gemma "the eyes and ears" — then made it literal. The 12B is unified: raw audio waveforms project into the same embedding space as image patches, so the perceive→reason split generalizes from images to the whole perception stack.

One choke point, three modalities. _perceive_then_reason now detects image, audio, and video content parts and feeds them all to the same perceiver — honoring the model card's quirks:

• Modality order matters. The card specifies image before text, audio after text. A reorder step enforces it; get it wrong and quality drops.
• Audio is capped at 30 seconds. This is the hinge. Gemma does native ASR + speech translation for short, conversational clips — but a 45-minute meeting recording is not its job. So a duration probe routes anything over the cap to Whisper large-v3, which stays exactly where it was for long-form transcription, diarization, and word-level timestamps. Gemma didn't replace Whisper; it took the conversational front half and left the archival back half alone.
• Video is frames. Gemma has no video encoder — it reads video as a sequence of images (≤60s @ 1fps). A small PyAV sampler turns a clip into frames, which flow through the identical image path.

The output side never moved. Gemma is text-out only — it doesn't speak and it doesn't draw — so TTS (XTTS/Coqui) and image generation (Sana/ComfyUI) are untouched. The unification is purely on the input side: one model to perceive everything coming in, the right specialist for everything going out.

The same logic lives in two places now: the chat hot-path (AitherLLMQueue) for media in a conversation, and AitherVoice/AitherVision for explicit tool calls — both pointed at the same gemma4-12b backend, so there's one perceiver, not three implementations drifting apart.

The honest part: a unified perceiver is a single point of failure

Co-locating Gemma and Qwen on the DGX is great for the happy path — but it means the DGX is now a single point of failure for perception. Worth being clear-eyed about how it degrades:

• Perceiver down, reasoner up: the two-stage call fails, falls back to single-stage, and Qwen — itself multimodal — handles the image in one pass. You lose the optimization, not the answer.
• Reasoner down, perceiver up: this is the case the split improves. Because Gemma already turned the pixels into text, the reasoning step can land on any text model — the local orchestrator, a cloud reasoner — with no vision requirement. The handoff to text is exactly what makes the fallback flexible.
• Whole DGX down: here's the weak spot. Text reasoning still degrades to the always-resident 5090 orchestrator, but that orchestrator can't see. A vision request during a full DGX outage has nowhere local to go, and the text-only cloud fallback can't read the image.

The answer is the same AWQ track that frees the 5090: once an 8 GB Gemma runs locally next to the orchestrator, perception stops depending on one box. Cloud vision (Gemini, Vast.ai Gemma) is the other backstop. A unified perceiver is a powerful primitive — but "one model that sees and hears" is also "one model whose absence blinds you," and the deployment has to plan for both.

The roadmap: AWQ and AitherKVCache

Parallel-on-DGX is the production answer today. But the desktop 5090 is where most development happens, and we want Gemma there too. Two tracks bring it home:

AWQ-4bit (~8 GB). This is the unlock for the 5090: at 8 GB, orchestrator(8) + gemma-awq(8) + ComfyUI(16) = 32 finally co-resides with no sleep dance at all. No community quant exists for the 12B, so we set out to build our own — and the build is where the encoder-free novelty bites back. Four times.

The first wall is the toolchain. The obvious tool is llm-compressor — the modern, vLLM-blessed AWQ path — and that's what our first scaffold reached for. It can't touch this model: llm-compressor 0.11 pins transformers <= 4.57.6, but gemma4_unified only exists in transformers 5.10. Installing the quantizer silently downgrades transformers until the model won't even load (KeyError: 'gemma4_unified'). The model is one day old; the AWQ tooling hasn't caught up to the transformers release it needs. The tool that does span both is Intel's AutoRound (auto_round 0.12.3, which requires only transformers >= 4.38 — no upper cap — and ships an AutoRoundMLLM path for exactly this kind of model). There's already a published Gemma-4 W4A16 quant in the wild made with it. So: AutoRound, not llm-compressor.

The second wall is where to point it. Quantize the wrong modules and you either corrupt perception or quantize nothing at all. Instantiating the model on a meta device — no weights, no gated download, just the graph — shows the encoder-free layout plainly: the 48-layer language decoder lives at model.language_model.layers.* (328 Linear modules, the entire reasoning core), while perception is three small fused projectors — model.embed_vision.patch_dense, model.embed_vision.multimodal_embedder.embedding_projection, and model.embed_audio.embedding_projection. We quantize the decoder to 4-bit and keep those three projectors plus lm_head in bf16; the multimodal towers are tiny and AWQ kernels are unproven on them, so they're not worth the risk. Note the trap: a naive recipe targeting model.layers.* — the path every other model uses — matches nothing here, because the decoder is one level deeper, nested under language_model. You only catch that by reading the actual module tree before you spend a GPU-hour.

The third wall is calibration itself. AutoRound's flagship mode learns per-layer rounding by running calibration data through each decoder block — and that forward pass crashes on Gemma 4: RuntimeError: tensor a (512) must match tensor b (256) inside apply_rotary_pos_emb. Gemma 4 alternates attention head dimensions (256 and 512) across its sliding-window and global layers; AutoRound caches one block's rotary embeddings and reuses them for a block with a different head dim. The model runs fine in vLLM — it's the calibration harness that can't cope with the heterogeneous heads, the same 256/512 thorn the 26B MoE hit. The escape hatch is RTN mode (--iters 0): pure weight-rounding, no calibration forward, so it sidesteps the crash entirely. You trade a point or two of quality for a quant that actually completes — all 48 layers in under a minute, peak VRAM under 2 GB.

The fourth wall is the export format vs. the loader. AutoRound's auto_awq export and vLLM's loader have to agree on which modules are quantized — and they don't. The "right" format, compressed-tensors (what the 26B AWQ-Gemma serves on), crashes the same way calibration does: it runs a block-forward and dies on the 256/512 heads. So the only export that completes is auto_awq RTN — and vLLM's day-old gemma4_unified loader then refuses the result, because that loader builds the tiny vision tower as a quantized layer (it passes the AWQ config straight into Gemma4UnifiedVisionEmbedder) and demands vision_embedder.patch_dense.qweight, while the checkpoint correctly has bf16 .weight. No config — modules_to_not_convert with every name we could spell — makes vLLM honor the skip; it's hardcoded in the model graph.

The fix is a one-line vLLM patch: force the vision embedder's quant_config=None so the tower stays bf16. We overlay it onto the nightly image the same way the TurboQuant patch overlays its KV kernels — docker/Dockerfile.vllm-gemma4-awqfix, a sed and a FROM, no recompile. With that, the 7.3 GB AWQ 12B finally loads.

The fifth wall is where it fits. The whole point of AWQ was co-residence, and on the GB10 that's a knife-edge. Qwen3.6-27B's real footprint isn't its 28 GB of weights — with CUDA graphs and KV it eats ~90 GB, and trimming its gpu-memory-utilization below ~0.65 starves its own cache (No available memory for the cache blocks). Stack a second model's load on top and the two fight for the same unified memory at startup — Qwen OOM-crash-looped three times, and the crashes leaked memory that didn't reclaim until a clean docker restart. The 7.3 GB AWQ model fits with room; the loading is what collides. Which is the whole case for MicroScheduler: an admission gate (gpu_coordinator.request_vram() exists — nothing calls it before a vLLM container loads) plus a depends_on: service_healthy chain would have serialized the loads and turned every one of those OOMs into a polite wait. The pieces are in the tree; wiring them is this branch's job.

So it loads, it co-resides, and it reasons over text correctly. And then the sixth wall — the one that actually matters for a perceiver: 4-bit RTN is too lossy for vision. Text generation is clean; feed it an image and it emits pure padding tokens. The decoder is fine — it's that uncalibrated 4-bit rounding clips to the text weight distribution, and the image-token activations coming out of the bf16 vision tower fall outside it, so the quantized matmuls mangle them. The fix is a better quant — 8-bit, or 4-bit calibrated on image data — and both route straight back through the calibration forward that crashes on the 256/512 heads (wall three). We tried 8-bit: AutoRound even auto-selected its multimodal template and got further, then hit the identical apply_rotary_pos_emb mismatch. Round trip.

So we patched the calibration. The realization that broke it open: the model is right; the quantizer is wrong. Gemma 4's text model computes a dict of rotary embeddings — one cos/sin per layer type, sized to that type's head dim — and hands each block the tuple that matches. AutoRound (and AutoGPTQ, and llm-compressor — every block-wise quantizer) captures the auxiliary inputs from block 0 and replays that one tuple into every block. Block 0 is a sliding layer, so its cos/sin are head-dim 256; replay them into a global block and you get exactly the 512-vs-256 crash from wall three. The fix is a thirty-line shim — scripts/gemma4_calibration_patch.py — that makes the attention self-correct: if the rotary it was handed doesn't match its own head dim, it recomputes the right one for its layer type from position_ids. In a normal forward the handed rotary always matches, so the shim is a no-op in production; only the quantizer's bad replay triggers it. A CPU-only A/B test (scripts/gemma4_calib_validate.py) proves it: a tiny gemma4_unified decoder that keeps the real 256/512 heads reproduces the crash without the shim and runs clean with it.

Running the real quant surfaced a second instance of the same bug class. AutoRound's blockwise RTN also runs a propagating forward, and Gemma 4's last-of-type layers do a bookkeeping write — shared_kv_states[layer_type] = … — that the quantizer feeds None, crashing on item assignment at layer 46. Same root cause (block-wise replay doesn't carry a gemma4-specific auxiliary input), same shim, one more line: hand those layers a throwaway dict. Two architectural specials — per-type rotary and cross-layer KV bookkeeping — both invisible to a quantizer built for a homogeneous model. With both handled, the shim carried all 48 layers through to a clean export.

And then the landing we were after: we quantized an 8-bit (W8A16) build with the shim, served it on the DGX, and it reads images. Same fruit-stand price list the 4-bit build emitted padding for — the 8-bit model returns "Apples $1.20, Oranges$0.80, Bananas $0.45." The fix wasn't more calibration; it was more precision. Text calibration never sees an image token, so no amount of it teaches the scales the vision distribution — but 8 bits of headroom simply don't clip the bf16 vision-tower activations the way 4 bits do. That's why a forward-free --iters 0 RTN at 8-bit was enough: the shim only had to survive the propagating forward, not tune anything. The 13 GB W8A16 build now serves on :8124 (vLLM's MarlinLinearKernel for CompressedTensorsWNA16, projectors bf16 via the same vision patch), co-resident with Qwen3.6-27B, and the perceive→reason flow runs end to end: Gemma reads the pixels, Qwen reasons over the text it extracts. Seven walls in, the decoder quant was still never the hard part — the quantizer's assumptions about a homogeneous model were, and once we stopped fighting them and made the model self-correct, the perceiver finally perceives.

AitherKVCache / TurboQuant for Gemma. Qwen's DGX slot already runs TQ3.5 KV compression. Teaching those same hooks to compress Gemma 4 12B's KV cache — on the nightly vLLM, against its sliding-window-plus-global attention layout — makes long context nearly free and the parallel layout roomier. Active R&D, but it's the multiplier.

What shipped

In the tree and committed: the AutoRound recipe (scripts/quantize_gemma4_awq_autoround.py — RTN, the model.language_model.layers targeting, projectors kept fp), the cloud-quant runner (3122_Quantize-Gemma4-AWQ-Cloud.ps1, no write-token on the rented box), the vLLM serving patch (docker/Dockerfile.vllm-gemma4-awqfix), the calibration shim (scripts/gemma4_calibration_patch.py — per-type rotary and the KV-bookkeeping coercion) with its CPU-only A/B proof (scripts/gemma4_calib_validate.py) and the quant driver that applies it (scripts/quantize_gemma4_calibrated.py), and the perceive→reason pipeline — wired and unit-tested across image, audio (30 s → Whisper guard), and video (frame sampling), routing through one perceiver. Live on the DGX: the 13 GB W8A16 8-bit 12B serving on :8124 alongside Qwen3.6-27B — vision verified (reads a price-list image the 4-bit build padded on), text reasoning verified, perceive→reason runs end to end. The shim carried all 48 layers through both crash points (per-type rotary at the global heads, KV bookkeeping at layer 46) to a clean compressed-tensors export. The 4-bit AWQ build remains the lighter, text-only understudy; 8-bit is the perceiver.

The lesson that keeps repeating with frontier open models: the launch post tells you what the model can do; the toolchain tells you what you can actually run. A unified model that sees, hears, and reasons is genuinely new — and on real hardware, the weeks before the quantizers catch up, "runs in 8 GB" still has to survive a vLLM loader that quantizes the wrong layer, a 27B roommate on a unified-memory GPU, and a 4-bit rounding that text forgives and vision does not.