A language token is a starting point.
Fast enough is only half the answer
Warm inference processed 195.37 seconds of audio in 25.81 seconds, with a median clip time of 0.594 seconds. Yet combined word error rate was 35.83%. The difference between Common Voice and MAK also shows how dataset choice changes the picture.
What we decided
Keep Kinyarwanda as an evaluation candidate. Review representative errors with fluent speakers and test independent real recordings before enabling a general transcription offering. Do not compare this WER directly with Swahili or Luganda measured on FLEURS.
Orpheus also generated two short Kinyarwanda voice samples. Valid WAV files establish technical execution, not correct pronunciation or naturalness; fluent listening review remains necessary.
Results, in context.
| Subset | Clips | WER ↓ | CER ↓ | Audio (s) | Inference (s) |
|---|---|---|---|---|---|
| Common Voice | 20 | 42.16% | 19.36% | 81.55 | 12.37 |
| MAK | 20 | 31.58% | 12.68% | 113.82 | 13.44 |
| Combined | 40 | 35.83% | 15.28% | 195.37 | 25.81 |
Combined error rates are weighted by reference counts, not averages of the two percentages.
How we ran it.
- Cape Town T4; production Sunflower weights converted to CTranslate2 INT8_FLOAT16, beam five, voice-activity detection and waveform normalisation.
- First twenty test rows from each of kin_commonvoice and kin_makbenchmark in the gated Sunbird speech benchmark, revision 4747e3bd3a1915de1cdf5fb032ec12e0cb68c19d. Forty recordings total 195.37 seconds. Access was granted before evaluation.
- WER/CER use NFKC, lowercasing, punctuation-to-space and collapsed whitespace, weighted by reference words/characters. Warm latency excludes model loading.
The limits matter.
- Small publisher-provided benchmark; independence from training data was not established. No gated recordings or transcripts are redistributed here.
- Two short TTS samples are insufficient to endorse a voice. Kinyarwanda was not enabled by this diagnostic.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
8987348ef6dbd613f1573fa8570d2fd7ae1aa5bb1d5a52c8d06750438561cade0f3cd7e4b51c3ee3372d3ebc072756868862a0b27f18658de15e0b583228ed7fSuggested citation: Embiro Labs Research. “Kinyarwanda: promising is not ready.” Published 12 Oct 2026. embiro.ai/research/kinyarwanda-speech-diagnostic
Report a correction
