391,249 responses from 115 respondents to 4,402 items.
One of 21 tables from the same source; this one measures Omni-MATH olympiad mathematics. The others are listed under Related tables.
| Description | AI models as respondents: 115 large language models scored on 4,402 items of Omni-MATH, olympiad-level mathematics problems with reference answers. resp is 1 if the judge ruled the model's final answer equivalent to the reference. OpenEval's cross-judge average is not kept. rater = the LLM judge; each judged response has one row per judge. Judges: anthropic/claude-3-5-sonnet-20241022, meta/llama-3.1-405b-instruct-turbo, openai/gpt-4o-2024-05-13, openai/gpt-5-2025-08-07, openai/gpt-5-mini-2025-08-07. Items were sampled per model, so most models answered only a subset. From the OpenEval archive (Hugging Face Open-Eval-Commons/OpenEval, revision 23a1ded9), scores only, one run per model x item. id = model name, verbatim (aliases and case variants not merged). Respondents are not independent: many models share base weights, decoding temperature varies across models (cov_temperature), and items may appear in training data. |
|---|---|
| Reference | Jiang, H., Zhang, S., Zhu, D., Bai, Y., Truong, S. T., Yi, X., Koyejo, S., Xie, X., & Xiao, Z. (2026). AI Evaluation Should Require Standardized Item-Level Data Releases. arXiv:2604.03244 (v2). Benchmark: Gao, B., et al. (2025). Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models. ICLR 2025. https://iclr.cc/virtual/2025/poster/27714 |
| DOI | 10.48550/arXiv.2604.03244 |
| Licence | CC BY-NC 4.0 |
| Source data | https://huggingface.co/datasets/Open-Eval-Commons/OpenEval/tree/23a1ded985b3c2cdaaddb27581a35bfabe0ad7e7 |
| Responses | 391,249 |
|---|---|
| Respondents | 115 |
| Items | 4,402 |
| Response categories | 2 |
| Responses per respondent | 3402.165 |
| Responses per item | 88.880 |
| Density | 0.773 |
| Longitudinal | FALSE |
| age range | Not applicable (non-person) |
|---|---|
| sample | AI/model |
| construct type | Cognitive/educational |
| measurement tool | Test, Observational rating |
| item format | Constructed Response |
| primary language(s) | eng |
| construct name | Omni-MATH olympiad mathematics |
This codebook is reconstructed by the IRW from the IRW data standard and the script that built it, jiang_2026_openeval.py. It is our best reading, not the source’s own codebook, and it may contain mistakes. Where the two disagree, the source data are the authority. If you find an error, please tell us.
| Column | What it is | Source column |
|---|---|---|
| IRW standard columns | ||
id | IRW standard Identifier for the focal unit being measured — typically a person, but sometimes another entity (e.g., a word in a lexical task). | made in the build script |
item | IRW standard Item identifier. | made in the build script |
rater | IRW standard Observer identifier in scenarios where items are rated by an external observer rather than self-reported. | made in the build script |
resp | IRW standard Response value. | made in the build script |
| Respondent covariates | ||
cov_model_size | Covariate | made in the build script |
cov_temperature | Covariate | made in the build script |
What item codes and covariate values stand for is in the codebook released with the source data.
Licence: CC BY-NC 4.0 — non-commercial use only.
Or load it directly in R or Python (both need a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("remotes")
remotes::install_github("itemresponsewarehouse/Rpkg")
library(irw)
df <- irw_fetch("jiang_2026_openeval_omni_math")
# Python
pip install irw
import irw
df = irw.fetch("jiang_2026_openeval_omni_math")
@misc{https://doi.org/10.48550/arxiv.2604.03244,
doi = {10.48550/ARXIV.2604.03244},
url = {https://arxiv.org/abs/2604.03244},
author = {Jiang, Han and Zhang, Susu and Zhu, Dongyao and Bai, Yuzhuo and Truong, Sang T. and Yi, Xiaoyuan and Koyejo, Sanmi and Xie, Xing and Xiao, Ziang},
keywords = {Artificial Intelligence (cs.AI), Computers and Society (cs.CY), Databases (cs.DB), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {AI Evaluation Should Require Standardized Item-Level Data Releases},
publisher = {arXiv},
year = {2026},
copyright = {Creative Commons Attribution 4.0 International}
}
| IRW version | v584 |
|---|---|
| Redivis dataset | item_response_warehouse_6 v5.11 |
| Redivis dataset DOI | 10.57761/qahq-j855 |
| Manifest pin for this IRW version | v5.11 |
| Metadata source | irw_meta v44.0 |