514,200 responses from 2,571 respondents to 10,000 items.
A version of this table that keeps the option each respondent chose is cifar10h_nom.
One of 2 tables from the same source; this one measures Image classification (CIFAR-10 object categories). The others are listed under Related tables.
| Description | CIFAR-10H: 2,571 MTurk annotators each labelled 200 CIFAR-10 test images with one of the ten CIFAR-10 classes (514,200 labels, 10,000 images); resp = 1 if the chosen class matches the CIFAR-10 label, resp_raw = the class chosen. |
|---|---|
| Reference | Peterson, J. C., Battleday, R. M., Griffiths, T. L., & Russakovsky, O. (2019). Human uncertainty makes classification more robust. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 9617-9626. |
| DOI | 10.1109/ICCV.2019.00971 |
| Licence | CC BY-NC-SA 4.0 |
| Source data | https://github.com/jcpeterson/cifar-10h |
These notes describe the source data, which the IRW reproduces as released. All notes are listed on the Data Notes page.
| Responses | 514,200 |
|---|---|
| Respondents | 2,571 |
| Items | 10,000 |
| Response categories | 2 |
| Responses per respondent | 200 |
| Responses per item | 51.420 |
| Density | 0.020 |
| Longitudinal | FALSE |
| age range | Adult (18+) |
|---|---|
| sample | Internet-based |
| construct type | Cognitive/educational |
| measurement tool | Test |
| item format | Likert Scale/selected response |
| primary language(s) | eng |
| construct name | Image classification (CIFAR-10 object categories) |
This codebook is reconstructed by the IRW from the IRW data standard and the script that built it, cifar10h.py. It is our best reading, not the source’s own codebook, and it may contain mistakes. Where the two disagree, the source data are the authority. If you find an error, please tell us.
| Column | What it is | Source column |
|---|---|---|
| IRW standard columns | ||
id | IRW standard Identifier for the focal unit being measured — typically a person, but sometimes another entity (e.g., a word in a lexical task). | made in the build script |
item | IRW standard Item identifier. | made in the build script |
resp | IRW standard Response value. | made in the build script |
resp_raw | IRW standard The response as originally recorded, where resp holds a scored or recoded version of it — the letter a candidate marked (A–E), an option label, a categorical string. | made in the build script |
rt | IRW standard Response time. | made in the build script |
| Other columns | ||
itemcov_subcategory | Not documented Covariates invariant to measurement probes (item-level attributes). | made in the build script |
itemcov_true_category | Not documented Covariates invariant to measurement probes (item-level attributes). | made in the build script |
trial_index | Not documented Trial-level details in repeated-trial designs: the trial index (trial_number), block, list, and so on. | made in the build script |
What item codes and covariate values stand for is in the codebook released with the source data.
Licence: CC BY-NC-SA 4.0 — non-commercial use only; adaptations must be shared under the same licence.
Or load it directly in R or Python (both need a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("remotes")
remotes::install_github("itemresponsewarehouse/Rpkg")
library(irw)
df <- irw_fetch("cifar10h")
# Python
pip install irw
import irw
df = irw.fetch("cifar10h")
@inproceedings{Peterson_2019, title={Human Uncertainty Makes Classification More Robust}, url={http://dx.doi.org/10.1109/ICCV.2019.00971}, DOI={10.1109/iccv.2019.00971}, booktitle={2019 IEEE/CVF International Conference on Computer Vision (ICCV)}, publisher={IEEE}, author={Peterson, Joshua and Battleday, Ruairidh and Griffiths, Thomas and Russakovsky, Olga}, year={2019}, month=Oct, pages={9616–9625} }
| IRW version | v581 |
|---|---|
| Redivis dataset | item_response_warehouse_6 v5.11 |
| Redivis dataset DOI | 10.57761/qahq-j855 |
| Manifest pin for this IRW version | v5.11 |
| Metadata source | irw_meta v44.0 |