All submissions
4 entries · v1.0.0 · updated 2026-10-18
| # | Model | Score ↑↓ | Correct call ↑↓ | Abstention ↑↓ | Spurious ↓ ↑↓ | n ↑↓ | By protocol | Notes |
|---|---|---|---|---|---|---|---|---|
| 1 | oracleall protocols · reads the expectations out of the corpus | 100.0% 98.9% – 100.0% |
100.0% | 100.0% | 0.0% | 342 | json
100.0% xml
100.0% plain
100.0% |
grader self-check |
| 2 | HardcoreAI/Nova-1.2-0.8Ball protocols · LoRA on Nova-1.1 · greedy · thinking off | 77.2% 72.5% – 81.3% |
71.8% | 88.9% | 4.6% | 342 | json
82.5% xml
73.7% plain
75.4% |
LoRA · bf16 · batch 8 |
| 3 | HardcoreAI/Nova-1.1-0.8Ball protocols · greedy · thinking off | 64.9% 59.7% – 69.8% |
59.8% | 75.9% | 16.7% | 342 | json
67.5% xml
65.8% plain
61.4% |
base model · bf16 · batch 8 |
| 4 | Qwen/Qwen3.5-0.8Ball protocols · greedy · thinking off | 59.7% 54.4% – 64.7% |
48.3% | 84.3% | 6.5% | 342 | json
67.5% xml
68.4% plain
43.0% |
base model · bf16 · batch 8 |
No model results yet. Open a discussion to submit the first one.
What counts
A case passes when the emitted call set matches the expectation: the right tool, semantically correct arguments, and — on no-call cases — nothing emitted at all. Wilson 95% interval in parentheses.
Recall on tool cases
Share of call-eligible cases where the model produced the expected invocation with correct arguments. High abstention at low correct-call means the model is too shy.
Calibration
Share of no-call cases where the model emitted *some* call anyway. High abstention at high spurious means it is noisy, not cautious — the two are not interchangeable.
Submit a model
Run the harness, keep the raw results JSON, and open a discussion on the dataset repository. Rows are added by hand — include the model id, protocols, case count, decoding configuration and the JSON attachment.
python -m tooluse1 selftest # oracle must score exactly 100%
python -m tooluse1 run --model <id> --backend transformers \
--protocol all --batch-size 8 --out results/