Test the contract and the content.
What worked mechanically
All 24 warm chat requests returned nonempty answers and completed streams. Qwen3-8B and Gemma 3 4B had roughly one-second medians; DeepSeek R1 Distill Qwen 7B needed 5.57 seconds. DeepSeek’s first reasoning token arrived much earlier than its first visible answer.
What failed semantically
All three chat models mistranslated the simple Swahili fixture. Gemma contradicted a policy and invented a support timeline. Retrieval reached 9/10 top-one matches with embeddings and 8/10 with reranking, but the Swahili subset was weaker. Adding a reranker was not automatically better.
Qwen emitted the expected lookup_order tool-call structure. No real business tool was executed, so this tested a contract rather than an end-to-end business action.
Keep incomplete work visible
The initial transcription path hit a readiness timeout. That is an incomplete speech measurement, not evidence that the model cannot run. The isolated Whisper retry is a separate note, preserving the original failure instead of replacing it.
Results, in context.
| Model | Median response (s) | Median first visible answer (s) |
|---|---|---|
| Qwen3-8B | 0.917 | 0.074 |
| Gemma 3 4B | 0.953 | 0.074 |
| DeepSeek 7B | 5.575 | 4.502 |
Eight warm requests per model; transport success did not imply correctness.
How we ran it.
- Cape Town T4, llama.cpp b11429, pinned Q4_K_M weights and Gemma F16 projector. Two 4,096-token slots, temperature zero, seed 42, maximum 768 output tokens; Qwen thinking disabled.
- Eight shared English/chat fixtures per model, ten retrieval queries over six fixed documents (seven English, three Swahili), vision/media feasibility and a gateway tool-call probe. Loopback timing.
The limits matter.
- Eight chat fixtures and a six-document retrieval corpus are screening tools, not broad quality estimates. Generated code was not executed.
- Different reasoning behavior affects time to visible output. Incomplete media stages are not successful benchmarks.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
fd102390aab9d919b4a77425bcbbaa20863ebf97cba9688ffd7e76c0346e8e629c9b9f56836de1b2c46e20ab4c737836e1ab68b8edd5dedd9908d1fd62e50dcbSuggested citation: Embiro Labs Research. “An API response is not an answer.” Published 12 Oct 2026. embiro.ai/research/capability-screening
Report a correction
