Voice agents that act
Measure completed tasks, state changes, recovery, confirmation, reversibility, and truthful status—not just valid tool-call JSON.
Open voice agent research
Choose a voice agent benchmark by the job it measures.
“Voice AI” is not one evaluation problem. A benchmark for speech recognition may say nothing about task completion; a tool-call suite may never execute the action; and a transcript benchmark may start after the microphone work is already done.
This guide maps 13 public starting points reviewed from first-party sources on September 1, 2026. It is a landscape, not a leaderboard, and inclusion does not imply endorsement.
Start with one of three product outcomes
Select the track that matches the user result you need to verify. Then use component metrics to explain why that result succeeded or failed.
Voice typing and dictation
Combine recognition accuracy with entity handling, formatting, latency, correction burden, and reliable insertion into the target app.
Meeting understanding
Keep capture, transcription, diarization, summary faithfulness, decisions, and action-item quality as separate layers.
Public benchmark starting points
The matrix below answers a narrow question: which public project is a reasonable first stop for a given evaluation need? “Partial” means the capability appears in only part of a suite or is measured indirectly.
| Benchmark | Best starting point for | Spoken I/O | Tools | Task completion | Computer action |
|---|---|---|---|---|---|
| EVA | End-to-end conversational agent accuracy and experience | Input + output | Yes | Yes | No |
| tau2-bench / tau-Voice | Dynamic user-agent-tool interaction in realistic domains | Input + output | Yes | Yes | No |
| VoiceComputerBench / TalkAct | Real-time voice conversation plus browser action | Input + output | Yes | Yes | Yes |
| NVIDIA voice-agent evaluation | Reproducible live harness over EVA and tau2 scenarios | Input + output | Yes | Yes | No |
| VoiceAgentBench | Speech-based tool selection, orchestration, and safety | Input only | Yes | Partial | No |
| Audio2Tool | Spoken tool-call selection across intent complexity | Input only | Yes | No | No |
| Voice Agent Latency | Caller-perceived time to first agent audio | Input + output | No | No | No |
| OpenBench | Reproducible ASR, diarization, and streaming tests | Input only | No | No | No |
| ELITR-Bench | Long-context reasoning over meeting transcripts | Transcript text | No | Partial | No |
Capability coverage, evidence notes, licenses, and links for all 13 reviewed projects are available in the machine-readable JSONL dataset.
Use a layered evaluation plan
A single composite score hides failure modes. Keep these four layers visible and publish the result for each one.
User outcome
Did the user get the intended result in the target system?
Interaction quality
Was the exchange timely, understandable, interruptible, and recoverable?
Component quality
Measure ASR, diarization, entities, tool arguments, or generated content without treating them as the final outcome.
Safety and trust
Test consent, confirmation, reversibility, privacy, and whether the agent reports completion truthfully.
Important gap: no public benchmark in this reviewed landscape fully measures continuous desktop dictation inside arbitrary applications. Teams should combine a public ASR baseline with a private, consented corpus that reflects their real microphones, languages, apps, and vocabulary.
Do not compare scores that mean different things
- Tool-call accuracy does not prove that the downstream action succeeded or changed the right object.
- Word error rate does not capture punctuation, formatting, entity accuracy, editing effort, or insertion reliability.
- Transcript-only meeting tests cannot reveal microphone, overlap, diarization, reconnect, or device failures.
- Latency measurements need explicit start and end boundaries; caller audio, first token, final text, and visible task progress are different clocks.
Record the benchmark commit, model and provider versions, prompts, tool schemas, audio preprocessing, trial count, sampling settings, timing boundaries, judge rubric, timeouts, refusals, and every dataset license. Failed runs should remain in the report rather than disappear from the denominator.
Open data and reproducibility
The underlying landscape is maintained as an independent community resource by Sophon LLC. It does not rank Cue against the listed projects, and no benchmark owner has sponsored inclusion.
GitHub · MITInspect, query, or contributeJSONL data, schema, dependency-free CLI, tests, source audit, and contribution workflow.
Hugging Face datasetLoad the structured dataPublic dataset card and a convenient starting point for analysis with common data libraries.
Zenodo DOICite a preserved releaseVersioned archives for reproducible research; use the release-specific DOI when pinning an exact snapshot.
Separate evidence: this landscape does not reproduce Cue's private 227-sample production benchmark. That narrower Apple Silicon measurement is documented in the Gemma 4 dictation case-study guide, with Google DeepMind's original Gemmaverse page as its primary source.
Evaluate the job, then choose the tool.
Start with the open landscape if you are designing an evaluation. If you are evaluating a desktop voice agent in practice, explore Cue's voice typing, meeting, and cross-app workflows separately.