Count useful answers, not just tokens.
The useful warning
Median streamed throughput looked encouraging, but all ten short runs reached the output limit before closing their reasoning segment. The first streamed token was reasoning/output, not necessarily useful answer text. This became an early reason to track final-answer completion separately.
What the gateway did prove
A later arithmetic probe answered 391 correctly but ignored the integer-only format; a summary returned two sentences instead of one. The async addition job completed. Nine measured gateway receipts totalled 1,829 tokens with no unresolved holds. This verified a development integration, not durable cloud operation.
Why we moved beyond a laptop
The gateway appeared faster than direct calls in this tiny sample, which is scheduling variation rather than evidence of acceleration. A separate CPU microbenchmark was slower. Background applications and thermal conditions were uncontrolled, so this was a feasibility baseline for the Cape Town tests.
Results, in context.
| Measurement | Direct | Gateway |
|---|---|---|
| Median first output (s) | 0.144 | 0.171 |
| Median completion (s) | 3.866 | 3.625 |
| Median end-to-end tokens/s | 33.11 | 35.31 |
| Output-cap finishes | 5 / 5 | 5 / 5 |
Every short run ended before completing its reasoning. Lower latency here is not a completed-answer quality result.
How we ran it.
- Local Apple M1 MacBook Pro, 16 GiB unified memory, CPU-only, four threads; not an African cloud deployment. Metal initialization failed in this execution environment; this does not establish general GPU incompatibility.
- DeepSeek R1 Distill Qwen 1.5B via a pinned third-party Q4_K_M GGUF, llama.cpp b11429. A 2,048-token context, temperature 0.6, top-p 0.95, min-p 0.05. Five direct and five gateway requests alternated after warm-up; the same 22-token prompt and 128-output-token cap, no fixed seed.
The limits matter.
- Five samples per path, shared laptop, unseeded sampling, uncontrolled file cache and background load. No reliable tail latency, gateway overhead or sustained-capacity claim.
- No cloud meter was involved; local electricity was not measured.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
32d0f695805999597d8c129d5764e27779abb8302d63824c06792c989b49f3cfbc1c71eecedeea065a66de79e441a6c6f8517ebac10239e04cab76cb5b11ef5944dc98fe58e3affe53383ae88f78eb986f1ec6a51c71a83c5f342eada93518acSuggested citation: Embiro Labs Research. “Our first benchmark: fast tokens, unfinished answers.” Published 12 Oct 2026. embiro.ai/research/local-reasoning-baseline
Report a correction
