Language coverage is useful. Reliability still decides the route.
The model fits the machine
The quantised 9B checkpoint loaded in 40.76 seconds and peaked at 7,731 MiB of allocated GPU memory, leaving meaningful room on the 16 GB T4. Across fourteen fast-mode fixtures, median full-response time was 2.99 seconds and median decode throughput was 8.93 tokens per second.
That is a practical deployment result: an African-language-focused model of this size can run on the affordable Cape Town hardware already used by the pilot. Fit, however, only answers whether the service can start. It does not answer whether customers can trust its responses.
The quality gate stopped the release
Six fixtures passed, six failed and two translations still need native-speaker review. The model handled exact JSON extraction, abstention, Swahili clock convention and negation. It also produced wrong totals, reversed a supplied refund rule and invented a customer-service phone number.
These are release-blocking failures for an API used in commerce or operations. A fluent local-language answer is not enough when the model changes the rule or fabricates the next action. We therefore did not add a public route, price or billing entry.
More thinking did not reliably repair it
Thinking mode recovered the Swahili refund-policy case, but the two arithmetic tests and the French refund-policy test still failed. The targeted retest passed one of four cases. Longer reasoning therefore increased work without making the weak categories dependable.
The result is a useful routing lesson. Reasoning mode should be enabled because a measured workload benefits from it, not used as a blanket repair for a model that misses grounded constraints.
Naive batching was slower
The two prompts completed sequentially in 4.20 seconds. A padded two-prompt batch took 13.26 seconds, 3.16 times longer, and produced 2.71 corrected output tokens per second. This specific fallback runtime does not turn simple batching into useful concurrent capacity.
At the recorded $0.698 hourly compute rate, the measured median decode speed implies a theoretical floor of about $21.71 per million output tokens at continuous full decode utilisation. Real serving cost would be higher. That is not competitive enough to offset the quality risk, so the existing Gemma route remains the safer pilot default.
Results, in context.
| Measurement | Result | Interpretation |
|---|---|---|
| Model load | 40.76 s | Cold model initialisation |
| Median response | 2.99 s | Fourteen fast-mode fixtures |
| p95 response | 6.74 s | Small-sample tail |
| Median decode | 8.93 tok/s | Generated output tokens |
| Peak GPU memory | 7,731 MiB | Allocated memory on a 16 GB T4 |
Full-response latency includes generation after the model was loaded. It is not time to first token.
| Language | Pass | Fail | Native review | Observed risk |
|---|---|---|---|---|
| English | 3 | 2 | 0 | Wrong arithmetic; incomplete support triage |
| Swahili | 2 | 3 | 1 | Policy contradiction; wrong arithmetic; invented phone number |
| Kinyarwanda | 1 | 0 | 1 | Translation wording needs native review |
| French | 0 | 1 | 0 | Reversed supplied refund rule |
| Total | 6 | 6 | 2 | Did not clear pilot release gates |
Counts reflect the published assistant-rubric review. Review items are deliberately not counted as passes.
| Retest | Fast result | Thinking result | Outcome |
|---|---|---|---|
| English arithmetic | Fail · UGX 40,000 | Fail · UGX 45,000 | Expected UGX 47,000 |
| Swahili policy | Fail | Pass | Recovered supplied rule |
| Swahili arithmetic | Fail | Fail | Totals remained inconsistent |
| French policy | Fail | Fail | Still allowed excluded refund |
Thinking recovered one of four targeted failures; it was not a reliable repair strategy.
| Path | Elapsed | Corrected output | Rate |
|---|---|---|---|
| Sequential | 4.20 s | 36 tokens | 8.58 tok/s combined |
| Padded batch | 13.26 s | 36 tokens | 2.71 tok/s combined |
Corrected output excludes padding. The batch was 3.16 times slower than sequential execution in this runtime.
How we ran it.
- AWS Cape Town: one NVIDIA T4 16 GB on a g4dn.xlarge. Sunbird/Sunflower-Qwen3.5-9B-bnb-4bit at revision 37246b26caf4e8ed8f883fbff1e51dbf00815104, 4-bit weights, PyTorch 2.8.0, Transformers 5.14.1 and bitsandbytes 0.47.0. The optional fast linear-attention extension was unavailable, so the run used the Transformers PyTorch fallback.
- Fourteen synthetic business fixtures covered English, Swahili, Kinyarwanda and French: structured extraction, arithmetic, supplied-policy grounding, support triage, abstention, translation, time interpretation and negation. Fast mode ran all fixtures; thinking mode reran four failed arithmetic or policy cases. An assistant rubric scored exact or factual requirements. Swahili and Kinyarwanda naturalness were left for native-speaker review.
- Performance measurements covered model load, full-response latency, decoded output rate, peak allocated GPU memory and a two-prompt padded batch proxy. The batch comparison used the same two prompts sequentially and together; it is a scheduler diagnostic rather than a production concurrency claim. The cost floor multiplies the measured hourly compute rate by decode time at full utilisation and excludes idle time, input processing, orchestration, storage, network, tax and margin.
The limits matter.
- The fourteen fixtures are a small synthetic screening set, not a comprehensive language benchmark or a population study. Two translation samples require native-speaker review and are not counted as passes.
- The assistant rubric checked explicit factual and formatting requirements. It is not a substitute for blind human evaluation, native-language review or domain-specific safety testing.
- The optional fast linear-attention runtime was unavailable. Results describe the PyTorch fallback on one T4 and should not be treated as the model's best possible throughput on other kernels or accelerators.
- The two-prompt padded batch is a narrow proxy, not a production serving engine, sustained-load test or end-user concurrency measurement.
- The cost figure is a theoretical busy-compute floor from a dated hourly rate. It excludes idle capacity, input tokens, queueing, operations, storage, network, tax and commercial margin.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
9f76c826b8e0f77418b277316b426810a27afb5883b3210e50fca33fb38e872ebc753b96ea5a25d1e513e153d682dfdc0613890e105391d82db08bda4c5c8c93fa8b47b2dea1a006e20dce671f118703cad4f49c27f5d6c7842e59f6d6e1f0ebSuggested citation: Embiro Labs Research. “It fit on the GPU. It missed the rule.” Published 13 Oct 2026. embiro.ai/research/sunflower-qwen35-benchmark
Report a correction
