Model choice is more than response speed.
Why Gemma moved forward
Gemma 4 E4B combined a 1.16-second English median with correct outcomes on all five image fixtures and a modest memory footprint. It still invented shipment status, failed a Swahili clock translation and needed language review. It was a candidate for the next integration test, not an accuracy winner.
Llama 3.1 8B was a useful fast English alternative. Qwen3.5-9B handled the image fixtures well but its thinking profile was slower. Every model missed the full Swahili clock translation in this screen.
Runtime compatibility is part of quality
All streams completed, yet eleven replies were truncated and seven Kimi requests lacked a final answer after separating its native reasoning markers. A 200 response cannot stand in for task completion. Kimi’s configuration was not an equivalent controlled thinking-budget test.
What we cannot rank
Four reasoning questions and two French questions do not establish advanced reasoning or multilingual ability. Human-like fluency, short completion times and small active-parameter counts do not prove task quality or total weight-memory requirements.
Results, in context.
| Profile | English (s) | Swahili (s) | Reasoning (s) | French (s) | Vision (s) |
|---|---|---|---|---|---|
| Qwen3.5-4B | 10.25 | 12.32 | 13.84 | 10.78 | 8.29 |
| Qwen3.5-9B | 16.42 | 16.29 | 20.77 | 16.62 | 14.70 |
| Gemma 4 E4B | 1.16 | 10.62 | 9.69 | 0.35 | 5.77 |
| Ministral 3 8B | 7.99 | 15.39 | 22.86 | 12.20 | 12.61 |
| GLM-4.6V-Flash | 7.28 | 18.29 | 15.00 | 13.66 | 3.29 |
| Llama 3.1 8B | 1.23 | 1.43 | 2.47 | 1.59 | Not tested |
| Kimi-VL-A3B | 3.70 | 24.98 | 17.70 | 15.26 | 4.78 |
Llama is text-only; Kimi’s requested reasoning cap was ineffective. Compare practical configurations, not theoretical model capability.
How we ran it.
- Each model ran separately on one Cape Town T4 using pinned llama.cpp b11429 and Q4_K_M weights; F16 projectors where supported. One 4,096-token slot, four CPU threads, 2,048 output tokens and a requested 512-token thinking cap.
- Seven English chat, nine Swahili, four reasoning, two French and five vision fixtures (vision-capable models only). Model-specific sampling, one seed. Seven separate cold requests and 184 measured requests. Kimi’s reasoning delimiters were not parsed, so its requested cap did not apply.
The limits matter.
- Small diagnostic samples, not a production SLA or a population accuracy estimate. Different output lengths affect completion time.
- Assistant review of synthetic fixtures, not blinded or native-speaker validation. Answer lengths and sampling differ. Vision includes the first image request. Memory readings were snapshots, not peak co-residency measurements.
- Kimi cap/parser incompatibility prevents an equivalent reasoning comparison. No tool-call contract or long-context accuracy was tested.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
bca9ec8bc569aff8622a3188520c8a690da552fec07e05983dcb8d20d4a63d578be6b3a2774e4cb1169e604b881c2e2a5cb5b52dd5fcb05726ebdfd9254137fde3370982319ea9d79962ee9f4b47076cbabb9a5a2011cba75a1fb290b8d63529Suggested citation: Embiro Labs Research. “Seven small models. No universal winner.” Published 12 Oct 2026. embiro.ai/research/seven-small-model-families
Report a correction
