A health check is not a warm model.
Where the GPU mattered
Qwen’s warm short response median fell from 3.32 seconds on CPU to 1.25 seconds on T4. The long-prompt first token fell from 51.03 to 1.39 seconds. Yet the GPU’s first request took 32.84 seconds after process readiness. Representative warm-up is essential before admitting users.
Why speed needs an answer check
DeepSeek’s small reasoning model often consumed its 256-token budget without a final answer. A later mixed run with a larger explicit budget completed all 24 requests, but had sizeable first-round and later latency spikes. Neither fast reasoning tokens nor a brief successful burst establishes usable sustained capacity.
The product consequence
Separate loading, first-use kernel setup, queue wait and useful-answer latency. Use a real text and concurrent warm-up before routing pilot traffic. The results helped choose a compact T4 for further integration; they were not a production launch benchmark.
Results, in context.
| Measurement | CPU (s) | T4 (s) |
|---|---|---|
| Readiness | 5.0 | 40.14 |
| First request | 1.4 | 32.84 |
| Warm short first token | 0.93 | 0.081 |
| Warm short full response | 3.32 | 1.25 |
| Long-prompt first token | 51.03 | 1.39 |
Four warm short tasks. Loading excludes VM launch and downloads.
How we ran it.
- AWS Cape Town: r8g.2xlarge Arm CPU (8 vCPUs, 64 GiB) versus g4dn.xlarge T4 (4 vCPUs, 16 GiB). Pinned llama.cpp b11429, DeepSeek R1 Distill Qwen 1.5B and Qwen3-8B Q4_K_M; Qwen thinking disabled.
- Two 4,096-token slots, temperature 0.6, seed 42, uncached prompts and 256 output tokens. Four warm short tasks, a repetitive 1,557-token Qwen prompt and separate cold/concurrent probes. A subsequent mixed 24-request burst kept both models resident.
The limits matter.
- Small diagnostic samples, not a production SLA or a population accuracy estimate. Different output lengths affect completion time.
- CPU/GPU builds and architectures differed. All measurements were loopback; no customer or cross-cloud networking. First concurrent shapes had setup overhead.
- No claim that the full model catalogue fits together or that a short mixed burst predicts an SLA.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
631520a2ea8c274f2effdc2321758814f824147340e3ccbd0d65641dd990f5bab47ec65b06f711b159c632bcd633b514cd606880691ecf8410b115e3480f1b06fd1ce83a521379457e020ada9f26f4458af4fa667f1db97c878734a7ea7c330cSuggested citation: Embiro Labs Research. “Warm GPUs, cold reality.” Published 12 Oct 2026. embiro.ai/research/cape-town-cpu-versus-t4
Report a correction
