Gemma 4 on one RTX 5090 — Stoneforge Labs

Verified results

The useful numbers

RTX 5090 · WSL2 · vLLM 0.25.1 · batch-one decode unless noted

82.86%MMLU-Pro, 26B FP8
82.6% published

180tokens/s, 26B FP8
64 experts offloaded

189tokens/s, 26B NVFP4
64k context

99.6%prefix-cache hit rate
repeated agent prompts

Measurement	Result	Method
26B MoE FP8 + expert offload	180 tok/s @ 32k	`bench_serve.py`, quiet card
26B MoE NVFP4	189 tok/s @ 64k	Same serving benchmark
31B dense NVFP4	63 tok/s @ 32k	Same serving benchmark
MMLU-Pro, 26B FP8	82.86%	lm-eval, 5-shot, chat template
MMLU-Pro, 26B NVFP4	79.71%	Identical protocol
NVFP4 accuracy cost	−3.15 points	Difference between the two rows above
Expert offload	97 → 180 tok/s	Freed VRAM enabled full CUDA-graph capture
NVFP4 execution path	`FlashInferCutlassNvFp4LinearKernel`	vLLM startup log

Model profiles

Three operating points

FP8 KV cache is the verified default for every profile.

Profile	Checkpoint	Decode	Context	Use
`fast`	26B-A4B NVFP4	189 tok/s	64k	Agent and coding work
`quality`	26B-A4B FP8 dynamic	180 tok/s	32k	Default; evaluation and general work
`smart`	31B dense NVFP4	63 tok/s	32k	Harder reasoning; slower decode

Additional checkpoints tested

System

Local model, complete agent path

Gemma Gateway is open source. The 1,300-line Rust bridge keeps upstream Gemini CLI intact and has 31 translation tests plus a live-model verifier.

Gemini CLI / gemma
        │
        ▼
gemma-gateway :8899  ───  rig-tools MCP
        │
        ▼
vLLM :8891  ───  Gemma 4 checkpoint

Workspace  ───  CodeGraph index

Launcher: gemma selects profiles, starts vLLM, attaches files or images, reports status, and keeps local logs.
Agent driver: A work-wave ledger gives each unit a verification command, retry budget, session capture, and commit requirement. Gemini CLI can replace the session driver without changing those gates.
Gateway: Gemma Gateway translates Gemini REST to OpenAI-compatible vLLM calls, including tools, schemas, streaming, reasoning, and token counts. It discovers the active model from /models.
Tools: rig-tools exposes web retrieval, defect location, syntax checks, and unit status through MCP.
CodeGraph: A workspace-side index returns relevant source, call paths, and change blast radius before edits. It is tooling, not part of the model weights.

Reproduction

Current run configuration

Serve a profile

gemma --boot
GEMMA_PROFILE=fast gemma --boot
GEMMA_PROFILE=smart gemma --boot
GEMMA_CTX=16384 gemma --boot

Connect Gemini CLI

gemma-gateway --listen 127.0.0.1:8899 \
  --upstream http://127.0.0.1:8891/v1

GEMINI_API_KEY=local \
GEMINI_CLI_TRUST_WORKSPACE=true \
GOOGLE_GEMINI_BASE_URL=http://127.0.0.1:8899 \
gemini

Run a gated work wave

WAVE_AGENT=gemini \
GEMMA_GATEWAY=http://127.0.0.1:8899 \
python3 scripts/pi_work_wave.py \
  --workspace /path/to/project \
  --no-baseline-gate \
  --final-verify true

Inspect an indexed codebase

codegraph init
codegraph explore "how does a Gemini request reach vLLM?"
codegraph sync

The measured report is public first. Reusable harness source is being separated from unrelated private code before release.

Engineering findings

What changed after measurement

Finding	Evidence	Result
Expert routing is workload-specific	The hottest expert received 5.6× the selections of the coldest; no expert was unused.	Offload the coldest 64, not arbitrary layers.
Offload can improve speed	FP8 decode rose from 97 to 180 tok/s while context doubled.	VRAM released for CUDA graphs outweighed transfer cost.
TurboQuant does not serve here	All four presets failed when multimodal Gemma 4 selected `TRITON_ATTN`: KV dtype unsupported.	Reverted every profile to FP8 KV.
The broad sweep was invalid	8k output truncation, a 7,200-second timeout, and concurrent boots corrupted different rows.	Removed the matrix; retained only independently reproduced results.
Reasoning changes TTFT accounting	First tokens arrive in `reasoning_content`, not `content`.	The speed harness now counts the first reasoning token.
Boot and sweep must be mutually exclusive	An interactive boot killed the model being measured and produced false serve failures.	The launcher now refuses to evict an active sweep.

Evidence policy: a vendor table is context, not a local result. Failed and interrupted runs are excluded from headline numbers.