Benchmarks
Choose a task suite, then follow its configuration and environment guide. FlashRSI registers 30 suites; years, variants, and domains within one adapter count as one suite.
Prepare the environment
| Task environment | Start here |
|---|---|
| Text, QA, and reasoning | Configuration index for data, grading, and optional model roles. |
| Web research | Search and reader services. |
| Repository and container tasks | Coding setups, Terminal-Bench, and the native verifier guide. |
| Interactive environments | τ²-bench, ALFWorld, or the configuration index for τ³. |
Start with the complete configuration index for the exact YAML example and dependencies of each adapter.
Tasks, metrics, and splits
configured means the run chooses the count with benchmark.split or split_counts; native means the benchmark exposes a fixed source split; — means that role is not part of the native collection. Details link to the corresponding integration or configuration guide.
| Domain | Benchmark | Task | Metric | Train | Dev | Test | Details |
|---|---|---|---|---|---|---|---|
| QA | SearchQA | Search-snippet short-answer QA | EM / F1 | configured | configured | configured | Setup |
| QA | HotpotQA | Multi-hop Wikipedia QA | EM / F1; joint F1 | configured | configured | configured | Setup |
| QA | MuSiQue | Compositional multi-hop QA | EM / F1 | configured | configured | configured | Setup |
| QA | TriviaQA | Wikipedia trivia QA | Alias-aware EM / F1 | configured | configured | configured | Setup |
| QA | Natural Questions | Short-answer and yes/no QA | EM / F1 | configured | configured | configured | Index |
| QA | 2WikiMultiHopQA | Multi-hop QA with context | EM / F1 | configured | configured | configured | Setup |
| Reasoning | BBEH | BIG-Bench Extra Hard tasks | Per-answer accuracy | — | — | native | Guide |
| Reasoning | SuperGPQA | Graduate-level multiple choice | Answer accuracy | — | — | native | Guide |
| Reasoning | HLE | Text-only Humanity's Last Exam | Judge accuracy | configured | configured | configured | Index |
| Math | MATH | Competition mathematics | Math-Verify accuracy | configured | configured | configured | Setup |
| Math | AIME | AIME 2025 / 2026 | Exact integer accuracy | — | — | native | Guide |
| Math | LiveMathBench | Repeated-sampling mathematics | G-Pass@k / mG-Pass@k | configured | configured | configured | Index |
| Coding | SWE-bench Verified | Repository issue repair | Verifier resolved rate | configured | configured | configured | Setup |
| Coding | SWE-bench Pro V2 | Repository repair | Container regrade rate | configured | configured | configured | Guide |
| Coding | DeepSWE | DeepSWE repository tasks | Container verifier reward | configured | configured | configured | Guide |
| Coding | Terminal-Bench 2.0 / 2.1 | Linux terminal tasks | Native verifier reward | configured | configured | configured | Guide |
| Web research | BrowseComp | Broad web research | Official judge accuracy | — | — | native | Services |
| Web research | DeepSearchQA | Single- and set-answer search | Autorater F1 | — | — | native | Services |
| Web research | WideSearch | Tabular broad search | Official table score | — | — | native | Services |
| Documents | OfficeQA-Pro | Document research | Numeric / text accuracy | configured | configured | configured | Index |
| Documents | Harvey LAB | Legal document work | Dual-judge all-pass rate | configured | configured | configured | Index |
| Documents | JobBench | Professional file delivery | Weighted rubric reward | configured | configured | configured | Guide |
| Documents | SpreadsheetBench 2 | Spreadsheet debugging and modeling | LibreOffice verifier | configured | configured | configured | Guide |
| Tool use | AppWorld | Application and API automation | Task-completion reward | configured | configured | configured | Setup |
| Tool use | BFCL V4 | Multi-turn function calling | Conversation success | configured | configured | configured | Index |
| Tool use | WorkBench | Workplace database tasks | Database-state correctness | configured | configured | configured | Guide |
| Tool use | TravelPlanner | Constraint-aware itineraries | Constraint pass rate | native | native | native | Index |
| Interactive | τ²-bench | Airline, retail, telecom dialogue | Mean native reward | configured | configured | native | Guide |
| Interactive | τ³ Banking | Multi-turn banking support | Native reward | configured | configured | native | Index |
| Interactive | ALFWorld | Text-only embodied tasks | TextWorld success | native | native | native | Guide |
Check coverage before a large run
DeepSWE currently has an interface adapter; official downloads and real execution remain unverified. SpreadsheetBench 2 covers its first three task categories and excludes Visualization. Check each guide for the supported track and validation status.
A registered benchmark is not evidence of a completed model run. Inspect task failures and trajectories alongside its aggregate metric. Report custom experimental splits separately from official leaderboard settings.
To integrate another task suite, see Add a benchmark.