← Blog

Decisions APIs for agent routing, Parts 2 and 3: three APIs, then calibration

OpenAI Decisions and Jev joined Gemini; uncalibrated, all three over-escalated. Calibrated on development data, Jev matched our Workbench planner at 20 of 24 fresh sealed tasks with zero wrong picks.

Nissan Dookeran3 min readCoding AgentsAI EvaluationDeveloper Tools

Decisions APIs for agent routing, Parts 2 and 3: three APIs, then calibration

Calibrated Jev matched our planner on 24 fresh tasks and made no wrong picks. It took two rounds to get there.

Watch the 94-second walkthrough on YouTube.

In Part 1, our Workbench planner beat every alternative at choosing a provider, model and reasoning level for 24 synthetic agent tasks. Gemini's Decisions API never chose a wrong lane, but it sent seven routable tasks to review. That left two open questions: would the other decisions APIs behave the same way, and was our 0.80 confidence threshold the real problem?

Uncalibrated, all three APIs over-escalated

We ran OpenAI's Decisions API (GPT-6 Luna) and TypeSafe's Jev 1.14 on the same sealed tasks, against the same locked planner baseline and the same thresholds.

MethodCorrect / 24Wrong laneUnneeded review
Current Workbench planner2301
Gemini Decision 11707
OpenAI Decisions13011
Jev 1.1411013

None of the three picked a wrong lane. All three sent a large share of perfectly routable work to a human. At a shared 0.80 threshold, a decisions API behaved like a cautious filter rather than a router.

Calibrating thresholds changed the picture

Next we tuned each API's thresholds on 36 development cases. The rule was zero wrong lanes first, then as many correct decisions as possible, and we froze the values before writing any new test tasks.

We then wrote 24 new tasks. In Part 1 the labeler and our Claude-based planner came from the same model family, so this time GPT-6 Luna labelled the answers. We sealed the labels, locked the planner's decisions first, and locked every API's output before scoring.

MethodCorrect / 24Wrong laneUnneeded review
Current Workbench planner2022
Jev 1.14, calibrated2004
Gemini Decision 1, calibrated2022
OpenAI Decisions, calibrated13011

Calibrated Jev matched the planner's 20 and avoided both of the planner's wrong picks, at the cost of two extra reviews. Gemini also reached 20, wrong picks included. OpenAI couldn't loosen its threshold on the development set without a wrong lane, so it stayed cautious.

The planner's own score moved too. With a labeler from a different model family it scored 20, not 23. That's the caveat we raised in Part 1, and it held up.

On this data, the threshold mattered more than which API we used. That's an inference from one run on 24 tasks, not a general rule.

Where the evidence stops

Every method still had at least one case the planner got right and it got wrong, so none passed our bar. The labels are model-generated and haven't been reviewed by a person, and at 24 tasks a one-task difference is noise. Calibration reused the Part 1 tasks. Nothing was executed, so there's no cost-savings claim.

The code, data and method are open if you want to run it on your own tasks. If you'd like to support more of these tests, you can buy us a coffee.