Decisions APIs for agent routing, Part 4: the same model scored 20 and 24
Claude Opus 5.5 scored 24 of 24 when we asked it to pick a lane from a list. The same model, running our Workbench planner role, scored 20 on the same tasks.
Watch the 65-second walkthrough on YouTube.
Part 3 tested three decisions APIs against that planner on 24 fresh, sealed tasks. Calibrated Jev and Gemini matched the planner at 20, and none of the three passed. Two gaps were left in the comparison. We had not rerun the ordinary models on the new tasks, and we had never tried OpenAI's Sol and Terra, or Opus itself as a plain router. Part 4 closes both.
Ordinary models on the fair test
Every model saw the same 24 tasks, the same allowed lanes and the same locked planner answers. GPT-6 Luna, a different family from the planner, had written the answer key. Each model got one structured question: which lane, if any, should run this task? To pass, a model had to make no regressions, meaning it could not get wrong any task the planner got right.
| Method | Correct / 24 | Wrong lane | Unneeded review | Regressions |
|---|---|---|---|---|
| Current Workbench planner (Opus 5.5) | 20 | 2 | 2 | — |
| Claude Opus 5.5, plain router, medium | 24 | 0 | 0 | 0 |
| Claude Sonnet, medium | 23 | 1 | 0 | 0 |
| GPT-6.2 Sol, medium | 23 | 0 | 1 | 1 |
| GPT-6 Luna, low | 21 | 3 | 0 | 0 |
| Calibrated Jev 1.14 (Part 3) | 20 | 0 | 4 | 1 |
| Nemotron 3.5 Super, free | 19 | 2 | 3 | 3 |
| GPT-6.1 Terra, medium | 18 | 0 | 6 | 3 |
| Claude Haiku, low | 16 | 1 | 7 | 4 |
Opus and Sonnet passed. Sol came close, with no wrong lanes but one regression. Luna at medium reasoning scored 20. The free floor, NVIDIA's Nemotron 3.5 Super on OpenRouter, reached 19 but picked two wrong lanes. Cohere was on our list for the floor, but OpenRouter only offers it as a paid model, so we left it out.
What the planner got wrong
The planner's four misses weren't reckless. In two analysis tasks it chose a high-tier model where the answer key said a mid-tier one was enough, so it would have spent more. In the other two, the allowed lane didn't match the tier the planner wanted. One was a simple edit where only a high-reasoning lane was allowed. The other was an analysis task where only a mid-tier lane was allowed. The planner sent both to a human. The answer key accepted the allowed lane, and so did Opus as a router.
So the 24 is agreement with one answer key, written by one model. The key's view of how much model a task deserves differs from our planner policy's view in a few places. Our guess, which we have not tested, is that the planner role's extra policy and free-text reasoning push it toward caution, and a narrow choice from a list does not.
Where the evidence stops
This is one run on 24 synthetic tasks. Nobody has reviewed the labels by hand. Sol, Terra and Luna share the answer key's model family, which may flatter them, the same effect that flattered our planner in Part 1. Nothing was executed, so there is no cost-savings claim. Every run here used included subscription usage or a free tier.
On this evidence we're not changing Workbench routing, and we're not adopting a decisions API for it. The code, data and method are open. If you'd like to support more of these tests, you can buy us a coffee.