{
  "formatVersion": 1,
  "scope": "Curated public research extract; not the full raw run.",
  "canonical": "https://embiro.ai/research/cape-town-cpu-versus-t4",
  "fieldNote": 2,
  "slug": "cape-town-cpu-versus-t4",
  "title": "Warm GPUs, cold reality.",
  "description": "Our first Cape Town CPU-versus-T4 comparison explains why readiness needs a real inference probe.",
  "category": "Infrastructure",
  "art": "local-speed",
  "thumbnail": {
    "src": "/research/thumbnails/cape-town-cpu-versus-t4.webp",
    "alt": "Cold and warm compute towers connected by a copper timing track."
  },
  "date": "2026-10-10",
  "published": "2026-10-12",
  "updated": "2026-10-12",
  "author": "Embiro Labs Research",
  "status": "Exploratory benchmark",
  "headline": "A health check is not a warm model.",
  "metric": "51.03 → 1.39s",
  "metricLabel": "Qwen long-prompt first token · CPU to T4",
  "method": [
    "AWS Cape Town: r8g.2xlarge Arm CPU (8 vCPUs, 64 GiB) versus g4dn.xlarge T4 (4 vCPUs, 16 GiB). Pinned llama.cpp b11429, DeepSeek R1 Distill Qwen 1.5B and Qwen3-8B Q4_K_M; Qwen thinking disabled.",
    "Two 4,096-token slots, temperature 0.6, seed 42, uncached prompts and 256 output tokens. Four warm short tasks, a repetitive 1,557-token Qwen prompt and separate cold/concurrent probes. A subsequent mixed 24-request burst kept both models resident."
  ],
  "sections": [
    {
      "title": "Where the GPU mattered",
      "paragraphs": [
        "Qwen’s warm short response median fell from 3.32 seconds on CPU to 1.25 seconds on T4. The long-prompt first token fell from 51.03 to 1.39 seconds. Yet the GPU’s first request took 32.84 seconds after process readiness. Representative warm-up is essential before admitting users."
      ]
    },
    {
      "title": "Why speed needs an answer check",
      "paragraphs": [
        "DeepSeek’s small reasoning model often consumed its 256-token budget without a final answer. A later mixed run with a larger explicit budget completed all 24 requests, but had sizeable first-round and later latency spikes. Neither fast reasoning tokens nor a brief successful burst establishes usable sustained capacity."
      ]
    },
    {
      "title": "The product consequence",
      "paragraphs": [
        "Separate loading, first-use kernel setup, queue wait and useful-answer latency. Use a real text and concurrent warm-up before routing pilot traffic. The results helped choose a compact T4 for further integration; they were not a production launch benchmark."
      ]
    }
  ],
  "limitations": [
    "Small diagnostic samples, not a production SLA or a population accuracy estimate. Different output lengths affect completion time.",
    "CPU/GPU builds and architectures differed. All measurements were loopback; no customer or cross-cloud networking. First concurrent shapes had setup overhead.",
    "No claim that the full model catalogue fits together or that a short mixed burst predicts an SLA."
  ],
  "tables": [
    {
      "title": "Qwen3-8B practical paths",
      "columns": [
        "Measurement",
        "CPU (s)",
        "T4 (s)"
      ],
      "rows": [
        [
          "Readiness",
          "5.0",
          "40.14"
        ],
        [
          "First request",
          "1.4",
          "32.84"
        ],
        [
          "Warm short first token",
          "0.93",
          "0.081"
        ],
        [
          "Warm short full response",
          "3.32",
          "1.25"
        ],
        [
          "Long-prompt first token",
          "51.03",
          "1.39"
        ]
      ],
      "note": "Four warm short tasks. Loading excludes VM launch and downloads."
    }
  ],
  "references": [],
  "audio": [],
  "evidence": [
    {
      "file": "README.md",
      "sha256": "631520a2ea8c274f2effdc2321758814f824147340e3ccbd0d65641dd990f5ba"
    },
    {
      "file": "summary.json",
      "sha256": "b47ec65b06f711b159c632bcd633b514cd606880691ecf8410b115e3480f1b06"
    },
    {
      "file": "mixed-summary.json",
      "sha256": "fd1ce83a521379457e020ada9f26f4458af4fa667f1db97c878734a7ea7c330c"
    }
  ]
}