55 cases × 10 tools, scored four ways.
Tool-match, args-match, no-forbidden, must-succeed. Pass on tool-match ≥ 0.5 and clean must-succeed. CI fails on any > 2pp regression vs. the pinned baseline.
Regression-catcher. The heuristic regex was tuned on these exact phrasings, so the number moves only when something genuinely breaks.
Paraphrases like "identify my whales" or "bucket my buyers" — none of the regex's keywords appear. The heuristic falls off a cliff.
Same paraphrases through Gemini. ~4× the heuristic — meaningful generalization, with room to grow on prompt tuning.
Pass-rate over time
Every full-corpus run lands here. One series per backend.
No runs yet — fire one off below and the chart will populate.
Per-tool pass rate
Per-case detail
| id | question | planned | tool | args | overall | |
|---|---|---|---|---|---|---|