← Blog

Decisions APIs for agent routing, Part 4: the same model scored 20 and 24

Claude Opus 5.5 picked the right agent lane on 24 of 24 sealed tasks as a plain router, and 20 in our Workbench planner role. Sonnet also passed; Sol, Terra, a free floor model and every decisions API did not.

Nissan Dookeran4 min readCoding AgentsAI EvaluationDeveloper Tools

Decisions APIs for agent routing, Part 4: the same model scored 20 and 24

Claude Opus 5.5 scored 24 of 24 when we asked it to pick a lane from a list. The same model, running our Workbench planner role, scored 20 on the same tasks.

Watch the 65-second walkthrough on YouTube.

Part 3 tested three decisions APIs against that planner on 24 fresh, sealed tasks. Calibrated Jev and Gemini matched the planner at 20, and none of the three passed. Two gaps were left in the comparison. We had not rerun the ordinary models on the new tasks, and we had never tried OpenAI's Sol and Terra, or Opus itself as a plain router. Part 4 closes both.

Ordinary models on the fair test

Every model saw the same 24 tasks, the same allowed lanes and the same locked planner answers. GPT-6 Luna, a different family from the planner, had written the answer key. Each model got one structured question: which lane, if any, should run this task? To pass, a model had to make no regressions, meaning it could not get wrong any task the planner got right.

MethodCorrect / 24Wrong laneUnneeded reviewRegressions
Current Workbench planner (Opus 5.5)2022—
Claude Opus 5.5, plain router, medium24000
Claude Sonnet, medium23100
GPT-6.2 Sol, medium23011
GPT-6 Luna, low21300
Calibrated Jev 1.14 (Part 3)20041
Nemotron 3.5 Super, free19233
GPT-6.1 Terra, medium18063
Claude Haiku, low16174

Opus and Sonnet passed. Sol came close, with no wrong lanes but one regression. Luna at medium reasoning scored 20. The free floor, NVIDIA's Nemotron 3.5 Super on OpenRouter, reached 19 but picked two wrong lanes. Cohere was on our list for the floor, but OpenRouter only offers it as a paid model, so we left it out.

What the planner got wrong

The planner's four misses weren't reckless. In two analysis tasks it chose a high-tier model where the answer key said a mid-tier one was enough, so it would have spent more. In the other two, the allowed lane didn't match the tier the planner wanted. One was a simple edit where only a high-reasoning lane was allowed. The other was an analysis task where only a mid-tier lane was allowed. The planner sent both to a human. The answer key accepted the allowed lane, and so did Opus as a router.

So the 24 is agreement with one answer key, written by one model. The key's view of how much model a task deserves differs from our planner policy's view in a few places. Our guess, which we have not tested, is that the planner role's extra policy and free-text reasoning push it toward caution, and a narrow choice from a list does not.

Where the evidence stops

This is one run on 24 synthetic tasks. Nobody has reviewed the labels by hand. Sol, Terra and Luna share the answer key's model family, which may flatter them, the same effect that flattered our planner in Part 1. Nothing was executed, so there is no cost-savings claim. Every run here used included subscription usage or a free tier.

On this evidence we're not changing Workbench routing, and we're not adopting a decisions API for it. The code, data and method are open. If you'd like to support more of these tests, you can buy us a coffee.