A 1.7B model beat my larger models at following instructions

I added qwen3:1.7b to Ralph Lab mostly to find out how small was too small. In my late-March 2026 tests, it beat qwen2.5-coder and qwen3:8b on instruction-following-medium.

That mattered because I was using the results to choose models for Kit, my builder agent. Picking one good code model for every job would have hidden the result.

The small model fits in about 1.1GB. Across all task types it averaged 3.8, below granite4:tiny-h's 5.3 and qwen3:8b's 8.2. Its overall score gave me little reason to expect a useful specialist.

What Ralph Lab measured

By late March, the harness had accumulated 395 runs across 8 models and two experimental waves. It ran pytest-style evaluations with LLM-as-judge scoring on a 0–10 scale. Tasks included code generation, bug detection, bug repair, schema generation, SQL, instruction following and math.

The important design choice was separating tasks that sound similar. Finding a bug and fixing it had different labels. So did following a constrained instruction and generating code.

The small model's useful job

On instruction-following-medium, qwen3:1.7b averaged 6.08 over 12 runs. qwen2.5-coder scored 5.63. qwen3:8b scored 4.51, with zero variance across 12 runs. The routing spec recorded High confidence for the replicated instruction-following route.

Those repeated scores described what happened on these tasks. They didn't explain why. My hypothesis from reading the outputs was that the larger models kept adding something helpful: an explanation, a caveat or a reformatted answer. The task required a constrained response and nothing else.

qwen3:1.7b followed those constraints more successfully. Whether that came from fewer competing tendencies or something else wasn't established by the scores.

Its limits were just as useful. It averaged 1.5 on bug detection and 1.35 on string manipulation. Those became hard exclusions in the routing table. This was a model for a particular instruction-following job, not a replacement for the larger models across the board.

Finding a bug was a different job from repairing it

Across 6 runs each on detect-only and fix-only tasks, the larger models split:

Routing both phases to the same model would have given up 1.45 points on detection or 1.1 points on repair in this test. A single combined code-review score would have obscured the distinction.

I could speculate about code-review training helping qwen2.5-coder find errors, or reasoning helping qwen3:8b repair them. The experiment measured the split, not those causes.

The model that didn't survive

I added qwen3:4b because qwen3:8b was doing well on code tasks. A smaller sibling seemed a reasonable candidate for simpler work.

It averaged 0.2 across 4 runs, with 3/4 completion. Outputs were incomplete or malformed: responses cut off mid-sentence or failed to follow the requested structure. That was enough for me to retire it from this routing table rather than keep it as an experimental fallback.

I didn't establish whether training, quantisation or another cause explained the failure. Sharing a family name with qwen3:8b hadn't made it a reliable smaller substitute. Nor did it explain why the still-smaller qwen3:1.7b had a useful specialty.

granite4:tiny-h was another reason not to equate small with unusable. At 4.2GB it served as the floor model, with 100% task completion in a single iteration.

How I used the results

The routing spec built from those 395 runs gave each build phase a task_type. Firefly recorded the label in BUILD_PLAN.md; Kit read it before choosing a model. At the time, the relevant entries were:

code_generation → qwen2.5-coder (92 runs, 7.60 avg)
instruction_following → qwen3:1.7b (6.08 avg, 12 runs; fallback: qwen2.5-coder if fails twice)
bug_detection → qwen2.5-coder (9.05 avg, 6 runs)
bug_repair → qwen3:8b (9.00 avg, zero variance, 6 runs)
boilerplate / floor → granite4:tiny-h (100% completion, single iteration)
everything else → qwen2.5-coder (7.60 avg, 100% completion, 92 runs)
skip entirely → qwen3:4b (0.2 avg, 4 runs, retired)

Maintaining that table added work. In this experiment it exposed useful differences that a single default would have missed. The result I hadn't expected was the instruction-following route: a 1.7B model beating models roughly 4× and 5× its size on one specific job.