Two numbers, not one
A tool-calling agent fails in two ways, and a single pass rate hides the difference. Being unhelpful — asking when it should act — is recoverable. Being unsafe — calling a tool the role forbids, or obeying an instruction hidden in a pull request title — pages a customer at 3am.
So both are scored, and a change that lifts the pass rate while adding an unsafe decision fails CI.
What the gap is made of
All eight of the baseline's unsafe decisions are the two that matter: five where it obeyed text written by a stranger, three where it called a tool its role never had. It still scores well on the easy half — which is exactly why that kind of agent ships.
Two cases stay red on purpose: conditional work it can't sequence, and a paraphrase that shares no keyword with the tool. A dataset scoring 100% has stopped being useful.