· 3 min read
The Language of Surrender: Larger Model Measurements
Three models one size class up — Qwen3-14B, Phi-4, and Mistral-Small-24B at Q4_K_M quantization — measured on the same 1,505-sentence limitation gold set, same prompt, same harness as the earlier six-model comparison. Qwen3-14B scores F1 0.8272 as a single model, above the previous two-model union (0.8143) and the published rule-system reference (0.800). VRAM at load: 10.6 GB of 16.3 GB. Every number is a full-split measurement.
This measurement extends the earlier model-selection article. That work compared six models of 7–9B parameters on a fixed gold set and deployed the best pair. This work adds three models one size class up, under identical conditions, and reports what changed.
Conditions
- Gold set: the Kilicoglu limitation test split — 1,505 human-labeled sentences, 303 positive (Dryad doi:10.5061/dryad.06ds7). Frozen; identical to the earlier article.
- Prompt: identical few-shot classification prompt, one-word POS/NEG output.
- Harness: identical serving stack — llama.cpp behind a FastAPI agent on one RTX 5070 Ti (16,303 MiB); models hot-swapped by config call; Q4_K_M quantization throughout.
- Candidates: Qwen3-14B (9,001,753,632 bytes), Phi-4 14B (9,053,114,816 bytes), Mistral-Small-24B-Instruct-2501 (14,333,908,672 bytes).
Results
| model | parameters | precision | recall | F1 | run time |
|---|---|---|---|---|---|
| Qwen3-14B | 14B | 0.8778 | 0.7822 | 0.8272 | 320 s |
| Mistral-Small-24B | 24B | 0.9760 | 0.5380 | 0.6936 | 390 s |
| Phi-4 | 14B | 0.9524 | 0.5281 | 0.6794 | 758 s |
| Qwen3-8B (prior best single) | 8B | 0.8875 | 0.7030 | 0.7845 | 248 s |
| Qwen3-8B ∨ Gemma-2-9B (prior best pair) | 8B+9B | 0.887 | 0.753 | 0.8143 | — |
| published rule system (reference) | — | 0.758 | 0.848 | 0.800 | — |
Confusion counts: Qwen3-14B 237/33/66 (tp/fp/fn); Mistral-Small-24B 163/4/140; Phi-4 160/8/143. Mistral-Small-24B’s 4 false positives in 1,505 sentences is the lowest false-positive count of any model measured in either article.
Memory fit
| model | file size | VRAM at load | card capacity |
|---|---|---|---|
| Qwen3-14B | 9.0 GB | 10,587 MiB | 16,303 MiB |
| Phi-4 | 9.1 GB | 11,007 MiB | 16,303 MiB |
| Mistral-Small-24B | 14.3 GB | 15,689 MiB | 16,303 MiB |
Measurement status notes
- The Qwen3-8B figure is the mean of three identical runs (identical confusion matrices; zero run-to-run variance at serving temperature). The 14B figures are single runs; the triplicate protocol applies before any deployment number is treated as final.
- Pair combinations were computed from the retained per-sentence outputs after the single-model runs; see the pair table below.
- All figures above are against clinical-register gold. Per-register and estate-scale performance are separate measurements; estate-scale accuracy is unmeasurable until human-labeled estate gold exists.
Pair combinations
Computed from per-sentence outputs on the same split.
| combination | precision | recall | F1 |
|---|---|---|---|
| Qwen3-14B ∨ Gemma-2-9B | 0.8714 | 0.8053 | 0.8370 |
| Qwen3-14B ∨ Qwen3-8B | 0.8469 | 0.8218 | 0.8342 |
| Qwen3-14B ∨ Mistral-Small-24B | 0.8791 | 0.7921 | 0.8333 |
| Qwen3-14B ∧ Gemma-2-9B | 0.9521 | 0.5908 | 0.7291 |
| Qwen3-14B ∧ Mistral-Small-24B | 0.9756 | 0.5281 | 0.6852 |
Production state
The serving rule in effect: the best-measured model on the frozen gold stays loaded. As of this measurement that is Qwen3-14B. The pair configuration with the highest measured union F1 is Qwen3-14B with Gemma-2-9B (0.8370). Mistral-Small-24B is retained for the agreement gate (precision 0.9756). All earlier models remain staged; a swap is one configuration call.
Reproduction
Data, prompt, and harness are unchanged from the earlier article’s reproduction section. The added artifacts: the three Q4_K_M files named above from their bartowski HuggingFace repositories, and a parameterized measurement runner that swaps a named model, waits for load, runs the full split, and writes a JSON result. Per-sentence outputs are retained for every run.
References
- Kilicoglu, H., Rosemblat, G., Malički, M., & ter Riet, G. (2018). Automatic recognition of self-acknowledged limitations in clinical research literature. JAMIA 25(7), 855–861. doi:10.1093/jamia/ocy038
- Kilicoglu, H., et al. (2018). Dataset. Dryad. doi:10.5061/dryad.06ds7
- Qwen3-14B, Phi-4, Mistral-Small-24B-Instruct-2501 — Q4_K_M GGUF quantizations, bartowski, HuggingFace.
- The earlier measurement article: The Language of Surrender: Model Selection Observations.
