101,805 responses from 5,265 respondents to 5 items.
| Description | ICLR 2024 peer-review scores from OpenReview: 20,361 reviews of 5,265 submissions, each scored on rating (1-10), confidence (1-5), soundness, presentation and contribution (1-4). id = submission, rater = anonymous reviewer id within the submission (reviewer ids do not link across submissions), cov_decision = outcome. 101,805 rows. |
|---|---|
| Reference | Neumann, M. (2025). SNOR v1: Structured and Normalized OpenReview Reviews and Comments (Version 1) [Data set]. Zenodo. Review content: OpenReview.net, ICLR 2024 Conference. |
| Licence | CC BY 4.0 |
| Source data | https://doi.org/10.5281/zenodo.15866613 |
| Responses | 101,805 |
|---|---|
| Respondents | 5,265 |
| Items | 5 |
| Response categories | 8 |
| Responses per respondent | 19.336 |
| Responses per item | 20,361 |
| Density | 3.867 |
| Longitudinal | FALSE |
This codebook is reconstructed by the IRW from the IRW data standard and the script that built it, iclr_2024_peer_review.py. It is our best reading, not the source’s own codebook, and it may contain mistakes. Where the two disagree, the source data are the authority. If you find an error, please tell us.
| Column | What it is | Source column |
|---|---|---|
| IRW standard columns | ||
id | IRW standard Identifier for the focal unit being measured — typically a person, but sometimes another entity (e.g., a word in a lexical task). | made in the build script |
item | IRW standard Item identifier. | made in the build script |
rater | IRW standard Observer identifier in scenarios where items are rated by an external observer rather than self-reported. | made in the build script |
resp | IRW standard Response value. | made in the build script |
| Respondent covariates | ||
cov_decision | Covariate | made in the build script |
What item codes and covariate values stand for is in the codebook released with the source data.
Or load it directly in R or Python (both need a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("remotes")
remotes::install_github("itemresponsewarehouse/Rpkg")
library(irw)
df <- irw_fetch("iclr_2024_peer_review")
# Python
pip install irw
import irw
df = irw.fetch("iclr_2024_peer_review")
@misc{https://doi.org/10.5281/zenodo.15866613,
doi = {10.5281/ZENODO.15866613},
url = {https://zenodo.org/doi/10.5281/zenodo.15866613},
author = {Neumann, Mark},
keywords = {Peer Review, OpenReview, Machine Learning Conferences},
language = {enc},
title = {SNOR v1: Structured and Normalized OpenReview Reviews and Comments},
publisher = {Zenodo},
year = {2025},
copyright = {Creative Commons Attribution 4.0 International}
}
| IRW version | v584 |
|---|---|
| Redivis dataset | item_response_warehouse_5 v8.5 |
| Redivis dataset DOI | 10.57761/41xy-5k56 |
| Manifest pin for this IRW version | v8.5 |
| Metadata source | irw_meta v44.0 |