A healthy queue beats a bigger number.
The saturation signal
Moving from four to eight concurrent clients changed throughput from 0.777 to 0.775 requests per second while nearly doubling chat and vision medians. More callers mainly added waiting. That is a queue design signal, not a reason to advertise eight-client capacity.
What the first run taught us
The initial eight-client phase had two retrieval-capacity rejections and four socket disconnects. The corrected run used a bounded retrieval queue, a socket timeout beyond the inference deadline and explicit busy responses. All 183 final measured requests completed; one arithmetic answer was wrong. Transport success and answer quality are separate.
Durability and accounting
All sixteen queued jobs survived restart; idempotent replay did not duplicate inference. Receipt totals reconciled at 54,966 accounted tokens, with no open reservations. The synthetic rate card verified arithmetic, not actual cloud charges. This run supported taking Gemma chat/vision and Qwen retrieval into the compact pilot.
Results, in context.
| Request | 2 clients (s) | 4 clients (s) | 8 clients (s) |
|---|---|---|---|
| Chat | 5.03 | 9.46 | 16.93 |
| Vision | 6.09 | 11.07 | 20.49 |
| Embedding batch of 6 | 0.099 | 0.09 | 0.094 |
| Rerank 6 documents | 0.091 | 0.09 | 0.097 |
Equal-category mixes; different prompts from earlier single-model tests. Sampled GPU peak: 7,225 MiB, not an absolute memory maximum.
How we ran it.
- One Cape Town T4 host with four vCPUs, 16 GiB RAM; Node gateway, PostgreSQL, independent durable worker, Gemma 4 E4B Q4_K_M plus F16 projector, Qwen3 embedding/reranker FP16.
- Two 4,096-token Gemma slots, 512-token thinking cap and 2,048 total output tokens. Baseline sixteen requests; three 48-request equal-category phases at two, four and eight closed-loop clients. Eight test tenants.
- Sixteen durable jobs were enqueued while the worker was stopped, the API restarted, then the worker drained them. Repeated idempotency keys were checked. Three separate retrieval probes and warm-ups complete the measured receipt set.
The limits matter.
- Short closed-loop synthetic bursts, not sustained arrival-rate testing. Twelve requests per category per mixed phase are too few for an SLA.
- All networking was local to the instance. Expected-answer checks are smoke tests, not a general accuracy leaderboard.
Evidence & sources.
The public extract contains the methods, selected results and caveats in this note. It is a curated summary, not the full raw run. Fingerprints identify the retained source artifacts; gated audio and private operational records are not included.
Download results extractSource artifact fingerprints · SHA-256
02472410d910866a42fd07e0f793a41d8b8b1d317aaa265cb84b979ace1e5159b3fe9997f59857b29d68d6c9a74c5bdd29a4987c02ae41bbb163f2effb26bc282fc73c112b414844448a47bc2fb21b3373484a3dfe14c17caaf92e3fa96c048eSuggested citation: Embiro Labs Research. “More concurrency is not more capacity.” Published 12 Oct 2026. embiro.ai/research/gemma-pilot-queue-and-load
Report a correction
