{
  "formatVersion": 1,
  "scope": "Curated public research extract; not the full raw run.",
  "canonical": "https://embiro.ai/research/sunbird-african-speech",
  "fieldNote": 12,
  "slug": "sunbird-african-speech",
  "title": "Choosing speech models for local languages.",
  "description": "Sunflower, Crane and Whisper for transcription; Orpheus, Spark and Kokoro for generation. The fastest option did not always sound best.",
  "category": "Speech",
  "art": "speech-wave",
  "thumbnail": {
    "src": "/research/thumbnails/sunbird-african-speech.webp",
    "alt": "Three sculptural microphones surrounding a shared listening ring."
  },
  "date": "2026-10-12",
  "published": "2026-10-12",
  "updated": "2026-10-12",
  "author": "Embiro Labs Research",
  "status": "Exploratory benchmark",
  "headline": "Accuracy, speed and listening belong together.",
  "metric": "12.5%",
  "metricLabel": "optimised Sunflower Swahili WER",
  "method": [
    "Same Cape Town g4dn.xlarge T4. ASR: twelve FLEURS clips each in Luganda and Swahili; pinned native Sunflower, published Faster-Whisper conversion, fresh CTranslate2 conversion of current Sunflower, Crane and Whisper.",
    "TTS: two unique short prompts per language plus a seed repeat, and a separate long English prompt. Orpheus used FP16 with CPU SNAC decoding; Spark used FP32 with GPU BiCodec; Kokoro ran on CPU. These are practical serving profiles, not matched architectures."
  ],
  "sections": [
    {
      "title": "The transcription choice",
      "paragraphs": [
        "Optimised current Sunflower reached 12.5% Swahili WER, versus 24.3% for Crane and 38.0% for Whisper on the same twelve clips. Native Sunflower reached 13.3%. The fresh conversion offered a useful speed/accuracy tradeoff and was selected for the Swahili pilot.",
        "Luganda remained difficult: native Sunflower reached 50.5% WER and the current conversion 54.7%. Better than a baseline does not mean dependable transcription. The published Faster-Whisper conversion used an older checkpoint/preprocessing, so differences cannot be attributed to quantization alone."
      ]
    },
    {
      "title": "The voice choice",
      "paragraphs": [
        "The product owner preferred Orpheus in English, Luganda and Swahili and found the Spark samples rushed and less natural. Spark was excluded from the pilot despite its shorter generation times. Kokoro and Crane were retained in their respective generation/transcription roles; Crane TTS was not established by this test.",
        "This is qualitative owner listening, not a blind panel or a naturalness score. Orpheus provides a separate voice option at a substantial speed and compute cost. Later long-text runtime work is documented in its own note."
      ]
    }
  ],
  "limitations": [
    "Twelve clips per language; possible training overlap, no independent native-speaker accuracy study. Noise tests used synthetic white noise, not real call-centre conditions.",
    "ASR round-trip scoring is an intelligibility proxy, not a naturalness metric. Configurations and output durations differ.",
    "Busy-compute estimates allocate the whole host rate to the measured operation. They exclude idle time, startup, storage, network and taxes; they are not customer API prices or marginal savings. Rates are the recorded October 2026 inputs, not a current quote."
  ],
  "tables": [
    {
      "title": "Same clips, different ASR paths",
      "columns": [
        "Model / path",
        "Luganda WER",
        "Swahili WER",
        "LG median (s)",
        "SW median (s)"
      ],
      "rows": [
        [
          "Crane",
          "61.7%",
          "24.3%",
          "0.67",
          "0.69"
        ],
        [
          "Whisper large-v3",
          "108.9%",
          "38.0%",
          "1.53",
          "1.19"
        ],
        [
          "Native Sunflower",
          "50.5%",
          "13.3%",
          "1.56",
          "1.6"
        ],
        [
          "Published Faster-Whisper",
          "56.1%",
          "17.6%",
          "1.12",
          "1.24"
        ],
        [
          "Current Sunflower · CT2",
          "54.7%",
          "12.5%",
          "1.13",
          "1.14"
        ]
      ],
      "note": "WER is corpus-level after normalisation; lower is better. More than 100% is possible through insertions. Whisper Luganda was an exploratory probe."
    },
    {
      "title": "Short speech generation",
      "columns": [
        "Model",
        "Language",
        "Median (s)",
        "Busy USD / 1M chars"
      ],
      "rows": [
        [
          "Kokoro",
          "EN",
          "1.86",
          "6.52"
        ],
        [
          "Orpheus",
          "EN",
          "13.1",
          "46.43"
        ],
        [
          "Orpheus",
          "LG",
          "5.31",
          "63.28"
        ],
        [
          "Orpheus",
          "SW",
          "14.39",
          "71.41"
        ],
        [
          "Spark",
          "LG",
          "3.17",
          "26.09"
        ],
        [
          "Spark",
          "SW",
          "5.31",
          "26.09"
        ]
      ],
      "note": "Small prompt set; different voices and output lengths. Spark results are historical evidence, not a recommendation. Busy-compute estimates allocate the whole host rate to the measured operation. They exclude idle time, startup, storage, network and taxes; they are not customer API prices or marginal savings. Rates are the recorded October 2026 inputs, not a current quote."
    }
  ],
  "references": [
    {
      "label": "Google FLEURS · dataset and attribution",
      "url": "https://huggingface.co/datasets/google/fleurs"
    },
    {
      "label": "SunflowerASR · model card",
      "url": "https://huggingface.co/Sunbird/SunflowerASR-51-african-languages"
    },
    {
      "label": "Kokoro · model card",
      "url": "https://huggingface.co/hexgrad/Kokoro-82M"
    }
  ],
  "audio": [],
  "evidence": [
    {
      "file": "README.md",
      "sha256": "b4971323352a7e32ef1c46f724749ead08532b02b864ad20db8616b654af35cf"
    },
    {
      "file": "summary.json",
      "sha256": "2af09f544f163f5b11652788ba4c475b1956847a5c8fc3747efd28e2a126166e"
    }
  ]
}