{
  "formatVersion": 1,
  "scope": "Curated public research extract; not the full raw run.",
  "canonical": "https://embiro.ai/research/capability-screening",
  "fieldNote": 3,
  "slug": "capability-screening",
  "title": "An API response is not an answer.",
  "description": "Our first capability screen separated transport success, retrieval quality, vision and incomplete media tests.",
  "category": "Models",
  "art": "semantic-library",
  "thumbnail": {
    "src": "/research/thumbnails/capability-screening.webp",
    "alt": "A mint prism splitting one signal into four capability tiles."
  },
  "date": "2026-10-10",
  "published": "2026-10-12",
  "updated": "2026-10-12",
  "author": "Embiro Labs Research",
  "status": "Exploratory benchmark",
  "headline": "Test the contract and the content.",
  "metric": "24",
  "metricLabel": "warm chat responses · three models",
  "method": [
    "Cape Town T4, llama.cpp b11429, pinned Q4_K_M weights and Gemma F16 projector. Two 4,096-token slots, temperature zero, seed 42, maximum 768 output tokens; Qwen thinking disabled.",
    "Eight shared English/chat fixtures per model, ten retrieval queries over six fixed documents (seven English, three Swahili), vision/media feasibility and a gateway tool-call probe. Loopback timing."
  ],
  "sections": [
    {
      "title": "What worked mechanically",
      "paragraphs": [
        "All 24 warm chat requests returned nonempty answers and completed streams. Qwen3-8B and Gemma 3 4B had roughly one-second medians; DeepSeek R1 Distill Qwen 7B needed 5.57 seconds. DeepSeek’s first reasoning token arrived much earlier than its first visible answer."
      ]
    },
    {
      "title": "What failed semantically",
      "paragraphs": [
        "All three chat models mistranslated the simple Swahili fixture. Gemma contradicted a policy and invented a support timeline. Retrieval reached 9/10 top-one matches with embeddings and 8/10 with reranking, but the Swahili subset was weaker. Adding a reranker was not automatically better.",
        "Qwen emitted the expected lookup_order tool-call structure. No real business tool was executed, so this tested a contract rather than an end-to-end business action."
      ]
    },
    {
      "title": "Keep incomplete work visible",
      "paragraphs": [
        "The initial transcription path hit a readiness timeout. That is an incomplete speech measurement, not evidence that the model cannot run. The isolated Whisper retry is a separate note, preserving the original failure instead of replacing it."
      ]
    }
  ],
  "limitations": [
    "Eight chat fixtures and a six-document retrieval corpus are screening tools, not broad quality estimates. Generated code was not executed.",
    "Different reasoning behavior affects time to visible output. Incomplete media stages are not successful benchmarks."
  ],
  "tables": [
    {
      "title": "Chat screening",
      "columns": [
        "Model",
        "Median response (s)",
        "Median first visible answer (s)"
      ],
      "rows": [
        [
          "Qwen3-8B",
          "0.917",
          "0.074"
        ],
        [
          "Gemma 3 4B",
          "0.953",
          "0.074"
        ],
        [
          "DeepSeek 7B",
          "5.575",
          "4.502"
        ]
      ],
      "note": "Eight warm requests per model; transport success did not imply correctness."
    }
  ],
  "references": [],
  "audio": [],
  "evidence": [
    {
      "file": "summary.json",
      "sha256": "fd102390aab9d919b4a77425bcbbaa20863ebf97cba9688ffd7e76c0346e8e62"
    },
    {
      "file": "category-results.json",
      "sha256": "9c9b9f56836de1b2c46e20ab4c737836e1ab68b8edd5dedd9908d1fd62e50dcb"
    }
  ]
}