{
  "formatVersion": 1,
  "scope": "Curated public research extract; not the full raw run.",
  "canonical": "https://embiro.ai/research/qwen35-small-multimodal",
  "fieldNote": 5,
  "slug": "qwen35-small-multimodal",
  "title": "Bigger did not always answer better.",
  "description": "A non-thinking Qwen3.5 comparison against Qwen3-8B and Gemma 3 4B, including a separate sampling check.",
  "category": "Models",
  "art": "vision-lens",
  "thumbnail": {
    "src": "/research/thumbnails/qwen35-small-multimodal.webp",
    "alt": "A camera, speech bubble and code brackets orbiting a compact compute cube."
  },
  "date": "2026-10-10",
  "published": "2026-10-12",
  "updated": "2026-10-12",
  "author": "Embiro Labs Research",
  "status": "Exploratory benchmark",
  "headline": "Measure the task, not the parameter count.",
  "metric": "182",
  "metricLabel": "measured requests in the main run",
  "method": [
    "Cape Town T4, llama.cpp b11429, Q4_K_M and F16 vision projectors; two 4,096-token slots, four CPU threads, 768 output tokens, thinking disabled and initial temperature zero.",
    "Seven English, nine Swahili and five vision fixtures repeated twice, plus three concurrent pairs per model and four separate cold requests. Qwen3-8B is text-only. A separate Qwen sensitivity run used temperature 0.7, top-p 0.8, top-k 20 and seeds 42/43."
  ],
  "sections": [
    {
      "title": "What the replies revealed",
      "paragraphs": [
        "Qwen3-8B was strongest on the seven English fixtures in the initial screen. Qwen3.5-9B recovered all five image tasks’ required facts, but contradicted its own English arithmetic. Qwen3.5-4B omitted delivery from an invoice calculation. Gemma 3 also made arithmetic and policy errors.",
        "All 182 measured requests produced visible replies without truncation or transport errors. That did not make every reply correct."
      ]
    },
    {
      "title": "Sampling is part of the result",
      "paragraphs": [
        "The separate recommended-sampling check corrected some 9B English mistakes, but Swahili translations and clock conversions remained wrong. One corrected seeded answer does not establish reliability. Repeated identical greedy outputs are not independent quality samples."
      ]
    },
    {
      "title": "Next experiment",
      "paragraphs": [
        "The later seven-family study evaluated thinking-enabled profiles with different token budgets and one slot. Keep the two studies separate: merging their latency distributions would conceal the configuration change."
      ]
    }
  ],
  "limitations": [
    "Small diagnostic samples, not a production SLA or a population accuracy estimate. Different output lengths affect completion time.",
    "Synthetic tasks assessed by assistant review; no native-speaker validation. First image calls remain in vision medians. All timings exclude public networking.",
    "Qwen GGUFs were third-party Unsloth conversions; results do not describe original full-precision weights."
  ],
  "tables": [
    {
      "title": "Non-thinking profiles",
      "columns": [
        "Model",
        "English median (s)",
        "Swahili median (s)",
        "Vision median (s)"
      ],
      "rows": [
        [
          "Qwen3.5-4B",
          "1.10",
          "1.60",
          "1.86"
        ],
        [
          "Qwen3.5-9B",
          "1.69",
          "1.08",
          "3.35"
        ],
        [
          "Qwen3-8B",
          "1.01",
          "0.85",
          "Not supported"
        ],
        [
          "Gemma 3 4B",
          "0.94",
          "1.38",
          "1.43"
        ]
      ],
      "note": "Readiness, download and separate cold text requests are excluded. Vision includes first-image initialization."
    }
  ],
  "references": [],
  "audio": [],
  "evidence": [
    {
      "file": "README.md",
      "sha256": "cc1dbb7d1f411399950dba8846064352c4ae5cd17183f8a43fc65f2f4d382953"
    },
    {
      "file": "summary.json",
      "sha256": "e01e0d924ff6b2c34e8d77e41444da2c9f264edcee77628122c92d4db34472f3"
    },
    {
      "file": "sampled-results.json",
      "sha256": "b683387ca29aeb8871e55d83ed5369686fbf3582006cecac96c377b0b3610e11"
    }
  ]
}