Measure the task, not the parameter count.
What the replies revealed
Qwen3-8B was strongest on the seven English fixtures in the initial screen. Qwen3.5-9B recovered all five image tasks’ required facts, but contradicted its own English arithmetic. Qwen3.5-4B omitted delivery from an invoice calculation. Gemma 3 also made arithmetic and policy errors.
All 182 measured requests produced visible replies without truncation or transport errors. That did not make every reply correct.
Sampling is part of the result
The separate recommended-sampling check corrected some 9B English mistakes, but Swahili translations and clock conversions remained wrong. One corrected seeded answer does not establish reliability. Repeated identical greedy outputs are not independent quality samples.
Next experiment
The later seven-family study evaluated thinking-enabled profiles with different token budgets and one slot. Keep the two studies separate: merging their latency distributions would conceal the configuration change.
Results, in context.
| Model | English median (s) | Swahili median (s) | Vision median (s) |
|---|---|---|---|
| Qwen3.5-4B | 1.10 | 1.60 | 1.86 |
| Qwen3.5-9B | 1.69 | 1.08 | 3.35 |
| Qwen3-8B | 1.01 | 0.85 | Not supported |
| Gemma 3 4B | 0.94 | 1.38 | 1.43 |
Readiness, download and separate cold text requests are excluded. Vision includes first-image initialization.
How we ran it.
- Cape Town T4, llama.cpp b11429, Q4_K_M and F16 vision projectors; two 4,096-token slots, four CPU threads, 768 output tokens, thinking disabled and initial temperature zero.
- Seven English, nine Swahili and five vision fixtures repeated twice, plus three concurrent pairs per model and four separate cold requests. Qwen3-8B is text-only. A separate Qwen sensitivity run used temperature 0.7, top-p 0.8, top-k 20 and seeds 42/43.
The limits matter.
- Small diagnostic samples, not a production SLA or a population accuracy estimate. Different output lengths affect completion time.
- Synthetic tasks assessed by assistant review; no native-speaker validation. First image calls remain in vision medians. All timings exclude public networking.
- Qwen GGUFs were third-party Unsloth conversions; results do not describe original full-precision weights.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
cc1dbb7d1f411399950dba8846064352c4ae5cd17183f8a43fc65f2f4d382953e01e0d924ff6b2c34e8d77e41444da2c9f264edcee77628122c92d4db34472f3b683387ca29aeb8871e55d83ed5369686fbf3582006cecac96c377b0b3610e11Suggested citation: Embiro Labs Research. “Bigger did not always answer better.” Published 12 Oct 2026. embiro.ai/research/qwen35-small-multimodal
Report a correction
