{ "formatVersion": 1, "scope": "Curated public research extract; not the full raw run.", "canonical": "https://embiro.ai/research/kokoro-cpu-rightsizing", "fieldNote": 18, "slug": "kokoro-cpu-rightsizing", "title": "How small can our speech server go?", "description": "We compared Kokoro on Azure D8 and D32 CPUs, including queued traffic and live AWS fallback. The smaller VM was cheaper, but it missed our latency gate.", "category": "Infrastructure", "art": "speech-wave", "thumbnail": { "src": "/research/thumbnails/kokoro-cpu-rightsizing.webp", "alt": "Two sculpted server forms connected by a copper speech wave, with mint request markers on a cream background." }, "date": "2026-10-17", "published": "2026-10-17", "updated": "2026-10-17", "author": "Embiro Labs Research", "status": "Exploratory benchmark", "headline": "Low average CPU use was a reason to test a smaller server. It was not enough evidence to move the pilot.", "metric": "Keep D32", "metricLabel": "Decision from this bounded comparison", "method": [ "One isolated Azure South Africa North VM and disk, measured sequentially at Standard_D8s_v5 (8 vCPUs, 32 GiB) and Standard_D32s_v5 (32 vCPUs, 128 GiB). Production stayed on D32. Each layout used two Uvicorn workers; we tested 4 and 2 threads per worker on D8, then 16 and 4 on D32.", "Kokoro 82M v1.0 on FP32 CPU, model revision f3ff3571791e39611d31c381e3a41a3af07b4987. The service source, weights and voice hashes matched production, and its Python package versions were pinned: PyTorch 2.8.0+cpu, Kokoro 0.9.4 and Misaki 0.9.4.", "Three synthetic voice/language cases: af_heart and bf_emma in English, ff_siwis in French. Nine prompts were approximately 150, 600 and 1,479 characters. After three voice warmups, each layout ran nine sequential requests and three mixed batches at each concurrency of 1, 2, 4 and 8: 54 measured requests and three warmups per layout, 216 measured requests overall.", "Timing covers the complete loopback JSON/WAV response and decoding. This is full-buffer generation, not time to first streamed audio or end-user network latency. Two synthesis locks remain; higher concurrency deliberately queues. The schedule is identical across layouts, but requests and voice warmups are not explicitly pinned or balanced between worker processes; some per-worker first-use effects may remain.", "Before comparison, we required zero test errors and each sequential length median plus concurrency-two p95 to stay within 25% of the D32 production-layout reference. Nearest-rank p95 at concurrency two is the maximum of only six observations.", "A separate five-request live API probe used normal admission on the existing production D32, including three concurrent long requests to exercise AWS fallback. Failed earlier probes, the adapter fix and recovery steps were retained." ], "sections": [ { "title": "The savings were real. The promotion gate was not met.", "paragraphs": [ "Our earlier CPU speech experiments established that Kokoro could serve the pilot without a dedicated GPU. This follow-up asked a narrower question: could an eight-vCPU Azure instance replace the current 32-vCPU host without noticeably slowing requests?", "Neither tested D8 layout passed the latency gate. All four controlled layouts completed their measured requests without serving errors, but successful audio delivery and acceptable waiting time are different requirements. The pilot stays on D32; no production VM resize was made.", "The lower-thread D8 control and matched four-thread D32 control help show why a VM size cannot be evaluated independently of its worker layout. These are observations from one allocation per size, not a universal ranking of CPUs or a production capacity guarantee." ] }, { "title": "Concurrency exposes what idle averages hide", "paragraphs": [ "A low average CPU chart mixes idle time with short bursts of work. A person waiting for generated speech experiences the burst. Reducing the instance size can save money while making that wait longer, even when memory use remains modest.", "We deliberately sent more requests than the two synthesis locks could execute at once. The four- and eight-request batches include queueing inside the service. They are stress observations, not a recommendation to raise production admission limits. A small p95 sample should be read alongside the full schedule and individual results.", "The next useful candidate is D16 or a separately validated runtime optimisation. Neither was measured here. Any later resize needs a bounded canary, realistic traffic and a rollback plan." ] }, { "title": "Fallback needed the same speech contract", "paragraphs": [ "The first live probe found a limit mismatch: the AWS adapter allowed 1,000 input characters and 60 seconds of audio, while the public speech contract allowed 1,500 characters and 180 seconds. We aligned the fallback limits while retaining authentication, finite-sample checks and incremental output bounds. Five targeted boundary/authentication tests and all 115 backend tests passed.", "Our initial fail-fast harness exited after the fallback error and cancelled two Azure requests. Their capacity slots were quarantined because backend completion was uncertain. A subsequent fallback-only probe completed three requests but rejected two long requests after the existing 30-second admission wait. We corrected the harness to await every request, restarted the Azure speech service, verified readiness and reconciled only the known test-owned claims.", "The final live check completed all five requests and recorded exactly one usage receipt per request. Three simultaneous long requests finished in 6.83 and 18.05 seconds on Azure and 58.66 seconds on AWS. These are public-path measurements from a different prompt, not additional samples for the loopback comparison.", "AWS fallback remains useful for continuity, but its single CPU speech slot is slower and cannot replace the full Azure pool during an outage. Cancellation with uncertain backend completion still requires verified recovery before releasing quarantined capacity. This study does not solve automatic recovery." ] }, { "title": "What we checked in the audio", "paragraphs": [ "All measured responses were valid 24 kHz mono PCM16 WAVs within the benchmark's 180-second bound. We compared the nine saved sequential outputs per other layout with the D32 production-layout reference: 27 of 27 comparisons had equal frame counts; 0 were byte-identical. The private benchmark archive retains frame counts and waveform correlations.", "Matching duration or waveform correlation is not a human listening assessment. We did not measure pronunciation accuracy, naturalness or a mean opinion score, and this run says nothing about untested languages." ] }, { "title": "Cost decisions without spending the pilot experience", "paragraphs": [ "Azure's Linux consumption list prices checked on 17 October 2026 were $0.508 per hour for D8s v5 and $2.032 for D32s v5 in South Africa North. At 730 hours, those are $370.84 and $1,483.36 for the VM component: a potential 75% reduction, or $1,112.52 per month. These savings were not realised because production remains D32.", "Disks, IP addresses, traffic, taxes and other services are outside that comparison. Credits do not change measured performance or the underlying list price. The experiment is too small to establish cost per customer, sustained throughput or a profitable speech price.", "Inference keeps its own instance, separate from application hosting and databases. Other migrated applications default to their own instances; consolidation needs a specific operational benefit and an agreed plan. Spare memory on a speech server is not a migration plan.", "The benchmark host used an isolated resource group, restricted SSH and an absolute shutdown deadline. Its results were collected before cleanup. The AWS fallback and GPU runtime controls remain in place." ] } ], "limitations": [ "One host allocation per size, fixed test order and no independent repetitions of the whole experiment. Both allocations reported Xeon Platinum 8573C processors, but they were not guaranteed to use the same physical host; topology is retained in the private benchmark archive.", "Three voices, synthetic repetitive prompts and a short run do not represent the full customer workload. No long-duration soak, cancellation storm or human speech-quality evaluation.", "Service startup was measured with model files already on disk; it excludes VM provisioning and model downloads, and the OS page cache was not controlled.", "The isolated D32 matched the production service/runtime but was not the production VM. The five-request live probe is reported separately.", "The initial live probes failed and required scoped service recovery. A completed final probe does not prove full-capacity failover or automatic cancellation recovery.", "A trailing launcher error occurred after the first D8 run had saved all 57 successful results; no measurements were discarded. Later launcher files were not edited while running." ], "tables": [ { "title": "Warmed full-response latency", "columns": [ "Layout: 2 workers × threads", "Short median", "Medium median", "Long median", "Concurrency 2 p95", "Errors / measured" ], "rows": [ [ "D8 / 2 × 4", "1.65 s", "7.83 s", "19.17 s", "29.89 s", "0 / 54" ], [ "D8 / 2 × 2", "2.87 s", "14.02 s", "35.43 s", "35.77 s", "0 / 54" ], [ "D32 / 2 × 16", "0.58 s", "3.22 s", "8.23 s", "12.33 s", "0 / 54" ], [ "D32 / 2 × 4", "1.53 s", "7.54 s", "19.10 s", "19.36 s", "0 / 54" ] ], "note": "Seconds. Each sequential length median has three voice samples. Concurrency-two p95 has six requests and is their maximum. These are isolated loopback measurements, not public API timings." }, { "title": "Queueing under mixed traffic", "columns": [ "Layout", "Concurrency 1 p95", "Concurrency 2 p95", "Concurrency 4 p95", "Concurrency 8 p95" ], "rows": [ [ "D8 / 2 × 4", "19.02 s", "29.89 s", "53.72 s", "72.65 s" ], [ "D8 / 2 × 2", "34.96 s", "35.77 s", "102.01 s", "86.27 s" ], [ "D32 / 2 × 16", "8.75 s", "12.33 s", "30.45 s", "40.03 s" ], [ "D32 / 2 × 4", "18.88 s", "19.36 s", "38.53 s", "51.81 s" ] ], "note": "Three deterministic batches per concurrency, with 3, 6, 12 and 24 requests respectively. The prompt mix differs by concurrency, so comparisons are between layouts at the same concurrency; this is not a pure scaling curve." }, { "title": "Potential VM price difference", "columns": [ "Size", "USD / hour", "USD / 730 hours" ], "rows": [ [ "D8s v5", "0.508", "370.84" ], [ "D32s v5", "2.032", "1,483.36" ] ], "note": "South Africa North Linux consumption prices, checked 17 October 2026. VM only; excludes disk, IP, traffic and tax. No live resize or realised saving." } ], "references": [ { "label": "Kokoro model card and v1.0 weights", "url": "https://huggingface.co/hexgrad/Kokoro-82M" }, { "label": "Azure Dsv5 sizes and CPU generations", "url": "https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dsv5-series" }, { "label": "Azure Retail Prices API", "url": "https://learn.microsoft.com/en-us/rest/api/cost-management/retail-prices/azure-retail-prices" }, { "label": "Earlier study: Can a CPU keep up with a conversation?", "url": "https://embiro.ai/research/kokoro-conversation-canary" }, { "label": "Earlier study: Does every AI workload need a GPU?", "url": "https://embiro.ai/research/african-inference-on-cpu" } ], "audio": [], "evidence": [ { "file": "README.md", "sha256": "bf1e06a8ab18b3e68c67dfdb6524b862aef04b77235c7b4e447e4d6a367f1f33" }, { "file": "REPORT.md", "sha256": "706dfdb8f5e8dd8800b0b5d73f08080d6f7fc35f6e5eb1774f3ddb932a9ea2b5" }, { "file": "summary.json", "sha256": "157fa8a984b3b796558ea6537e0a1b0274c2b866e940985861030a5d01703bae" } ] }