66,152 responses from 124 respondents to 541 items.
| Description | AI models as respondents: 124 large language models (id = model name, verbatim, aliases not merged) scored on the 541 prompts of IFEval, an instruction-following benchmark whose prompts each carry 1-3 programmatically verifiable instructions (e.g. no commas, at least 300 words). resp is the number of the prompt's instructions the model's response satisfied under IFEval's strict check (0 to itemcov_n_instructions); resp == itemcov_n_instructions is prompt-level strict accuracy. From the OpenEval archive (Hugging Face Open-Eval-Commons/OpenEval, revision 23a1ded9), one run, scores only. Respondents are not independent: many models share base weights, decoding temperature varies across models (cov_temperature), and items may appear in training data. |
|---|---|
| Reference | Jiang, H., Zhang, S., Zhu, D., Bai, Y., Truong, S. T., Yi, X., Koyejo, S., Xie, X., & Xiao, Z. (2026). AI Evaluation Should Require Standardized Item-Level Data Releases. arXiv:2604.03244 (v2). Benchmark: Zhou, J., et al. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911. |
| DOI | 10.48550/arXiv.2604.03244 |
| Licence | CC BY-NC 4.0 |
| Source data | https://huggingface.co/datasets/Open-Eval-Commons/OpenEval/tree/23a1ded985b3c2cdaaddb27581a35bfabe0ad7e7 |
| Responses | 66,152 |
|---|---|
| Respondents | 124 |
| Items | 541 |
| Response categories | 4 |
| Responses per respondent | 533.484 |
| Responses per item | 122.277 |
| Density | 0.986 |
| Longitudinal | FALSE |
| age range | Not applicable (non-person) |
|---|---|
| sample | AI/model |
| construct type | Cognitive/educational |
| measurement tool | Test |
| item format | Constructed Response |
| primary language(s) | eng |
| construct name | IFEval instruction following (verifiable instructions, strict check) |
cov_model_sizecov_temperatureiditemitemcov_n_instructionsresp
Licence: CC BY-NC 4.0 — non-commercial use only.
Or load it directly in R or Python:
# R
install.packages("remotes")
remotes::install_github("itemresponsewarehouse/Rpkg")
library(irw)
df <- irw_fetch("jiang_2026_openeval_ifeval")
# Python
pip install irw
import irw
df = irw.fetch("jiang_2026_openeval_ifeval")
@misc{https://doi.org/10.48550/arxiv.2604.03244,
doi = {10.48550/ARXIV.2604.03244},
url = {https://arxiv.org/abs/2604.03244},
author = {Jiang, Han and Zhang, Susu and Zhu, Dongyao and Bai, Yuzhuo and Truong, Sang T. and Yi, Xiaoyuan and Koyejo, Sanmi and Xie, Xing and Xiao, Ziang},
keywords = {Artificial Intelligence (cs.AI), Computers and Society (cs.CY), Databases (cs.DB), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {AI Evaluation Should Require Standardized Item-Level Data Releases},
publisher = {arXiv},
year = {2026},
copyright = {Creative Commons Attribution 4.0 International}
}
| IRW version | v407 |
|---|---|
| Redivis dataset | item_response_warehouse_6 v3.2 |
| Redivis dataset DOI | 10.57761/yvkw-nj32 |
| Manifest pin for this IRW version | v3.2 |
| Metadata source | irw_meta v24.0 |