ToolUse.1 — Tool-Calling Benchmark with Abstention

Overall score

77.2%

Passed cases over all cases, three protocols combined. Each entry reports a Wilson 95% interval.

Correct call

71.8%

Right tool with semantically correct arguments when a tool applies. Expressions are evaluated — 45+12 equals 45 + 12.

Abstention

88.9%

No call emitted when no tool applies: knowledge questions, absent capabilities, explicit no-tool instructions.

Current leader: HardcoreAI/Nova-1.2-0.8B — the figures above are its run.

What it measures

Right call, right time

Nine tools across arithmetic, weather, search, translation, conversion, timers, notes and time. Every case is a realistic user turn — no trivia, no synthetic templates.

Three protocols

JSON · XML · plain-text

The output protocol is a run-time axis, not a data axis. Each case is rendered in all three; grading uses canonical call semantics, so a model is judged on intent, not formatting.

Deterministic grading

No LLM judge

Arguments are compared as values with alias tables for cities, currencies, units and timezones. Exactly one call is required for call cases. The oracle self-check scores 100%.

Leaderboard — top of the board

ToolUse.1 — Leaderboard

4 entries · v1.0.0 · updated 2026-10-18

#	Model	Score	Correct call	Abstention	Spurious ↓	n
1	oracleall protocols · reads the expectations out of the corpus	100.0% 98.9% – 100.0%	100.0%	100.0%	0.0%	342	json 100.0% xml 100.0% plain 100.0%	grader self-check
2	HardcoreAI/Nova-1.2-0.8Ball protocols · LoRA on Nova-1.1 · greedy · thinking off	77.2% 72.5% – 81.3%	71.8%	88.9%	4.6%	342	json 82.5% xml 73.7% plain 75.4%	LoRA · bf16 · batch 8
3	HardcoreAI/Nova-1.1-0.8Ball protocols · greedy · thinking off	64.9% 59.7% – 69.8%	59.8%	75.9%	16.7%	342	json 67.5% xml 65.8% plain 61.4%	base model · bf16 · batch 8
4	Qwen/Qwen3.5-0.8Ball protocols · greedy · thinking off	59.7% 54.4% – 64.7%	48.3%	84.3%	6.5%	342	json 67.5% xml 68.4% plain 43.0%	base model · bf16 · batch 8

Open the full leaderboard →

Submit a model

Run the benchmark yourself, keep the raw results JSON the harness writes, and open a discussion on the dataset repository with the model id and configuration.

git clone https://huggingface.co/datasets/HardcoreAI/ToolUse.1
pip install -r requirements.txt         # transformers 5.x for Nova / Qwen3.5

python -m tooluse1 selftest             # oracle must score exactly 100%
python -m tooluse1 run --model <id> \
    --backend transformers --protocol all --batch-size 8 --out results/

Verify the harness — run the self-check first; it validates the grader against the corpus.
Run all three protocols and keep the results JSON the harness emits.
Open a discussion on HardcoreAI/ToolUse.1 with the model id, protocols, case count, decoding config, and the raw JSON attachment.

Recommended: greedy decoding (do_sample=False), max_new_tokens=256, unmodified system prompt, no few-shot examples, reasoning mode disabled where supported. One dtype per row — bfloat16 on Ampere+, float16 on T4.