Accuracy, speed and listening belong together.
The transcription choice
Optimised current Sunflower reached 12.5% Swahili WER, versus 24.3% for Crane and 38.0% for Whisper on the same twelve clips. Native Sunflower reached 13.3%. The fresh conversion offered a useful speed/accuracy tradeoff and was selected for the Swahili pilot.
Luganda remained difficult: native Sunflower reached 50.5% WER and the current conversion 54.7%. Better than a baseline does not mean dependable transcription. The published Faster-Whisper conversion used an older checkpoint/preprocessing, so differences cannot be attributed to quantization alone.
The voice choice
The product owner preferred Orpheus in English, Luganda and Swahili and found the Spark samples rushed and less natural. Spark was excluded from the pilot despite its shorter generation times. Kokoro and Crane were retained in their respective generation/transcription roles; Crane TTS was not established by this test.
This is qualitative owner listening, not a blind panel or a naturalness score. Orpheus provides a separate voice option at a substantial speed and compute cost. Later long-text runtime work is documented in its own note.
Results, in context.
| Model / path | Luganda WER | Swahili WER | LG median (s) | SW median (s) |
|---|---|---|---|---|
| Crane | 61.7% | 24.3% | 0.67 | 0.69 |
| Whisper large-v3 | 108.9% | 38.0% | 1.53 | 1.19 |
| Native Sunflower | 50.5% | 13.3% | 1.56 | 1.6 |
| Published Faster-Whisper | 56.1% | 17.6% | 1.12 | 1.24 |
| Current Sunflower · CT2 | 54.7% | 12.5% | 1.13 | 1.14 |
WER is corpus-level after normalisation; lower is better. More than 100% is possible through insertions. Whisper Luganda was an exploratory probe.
| Model | Language | Median (s) | Busy USD / 1M chars |
|---|---|---|---|
| Kokoro | EN | 1.86 | 6.52 |
| Orpheus | EN | 13.1 | 46.43 |
| Orpheus | LG | 5.31 | 63.28 |
| Orpheus | SW | 14.39 | 71.41 |
| Spark | LG | 3.17 | 26.09 |
| Spark | SW | 5.31 | 26.09 |
Small prompt set; different voices and output lengths. Spark results are historical evidence, not a recommendation. Busy-compute estimates allocate the whole host rate to the measured operation. They exclude idle time, startup, storage, network and taxes; they are not customer API prices or marginal savings. Rates are the recorded October 2026 inputs, not a current quote.
How we ran it.
- Same Cape Town g4dn.xlarge T4. ASR: twelve FLEURS clips each in Luganda and Swahili; pinned native Sunflower, published Faster-Whisper conversion, fresh CTranslate2 conversion of current Sunflower, Crane and Whisper.
- TTS: two unique short prompts per language plus a seed repeat, and a separate long English prompt. Orpheus used FP16 with CPU SNAC decoding; Spark used FP32 with GPU BiCodec; Kokoro ran on CPU. These are practical serving profiles, not matched architectures.
The limits matter.
- Twelve clips per language; possible training overlap, no independent native-speaker accuracy study. Noise tests used synthetic white noise, not real call-centre conditions.
- ASR round-trip scoring is an intelligibility proxy, not a naturalness metric. Configurations and output durations differ.
- Busy-compute estimates allocate the whole host rate to the measured operation. They exclude idle time, startup, storage, network and taxes; they are not customer API prices or marginal savings. Rates are the recorded October 2026 inputs, not a current quote.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
b4971323352a7e32ef1c46f724749ead08532b02b864ad20db8616b654af35cf2af09f544f163f5b11652788ba4c475b1956847a5c8fc3747efd28e2a126166eSuggested citation: Embiro Labs Research. “Choosing speech models for local languages.” Published 12 Oct 2026. embiro.ai/research/sunbird-african-speech
Report a correction
