eval harness · public

55 cases × 10 tools, scored four ways.

Tool-match, args-match, no-forbidden, must-succeed. Pass on tool-match ≥ 0.5 and clean must-succeed. CI fails on any > 2pp regression vs. the pinned baseline.

main · heuristic tuned
100.0%
55 / 55 cases · heuristic

Regression-catcher. The heuristic regex was tuned on these exact phrasings, so the number moves only when something genuinely breaks.

holdout · heuristic novel
7.1%
1 / 14 paraphrases

Paraphrases like "identify my whales" or "bucket my buyers" — none of the regex's keywords appear. The heuristic falls off a cliff.

holdout · LLM generalization
28.6%
4 / 14 paraphrases

Same paraphrases through Gemini. ~4× the heuristic — meaningful generalization, with room to grow on prompt tuning.

Pass-rate over time

Every full-corpus run lands here. One series per backend.

Run config