8,244 responses from 1,188 respondents to 7 items.
One of 8 tables from the same source; this one measures Tendency to override intuitive answers using the Cognitive Reflection Test (CRT). The others are listed under Related tables.
| Description | This dataset measures the tendency to override intuitive answers using the 7-item Cognitive Reflection Test (CRT). |
|---|---|
| Reference | Himmelstein, Mark, Sophie Ma Zhu, Nikolay Petrov, Ezra Karger, Jessica Helmer, Sivan Livnat, Amory Bennett, Page Hedley, and Phil Tetlock. “The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability”. PsyArXiv, November 23, 2025. doi:10.31234/osf.io/a7kdx_v8. |
| DOI | 10.31234/osf.io/a7kdx_v8 |
| Licence | CC BY 4.0 |
| Source data | https://github.com/forecastingresearch/fpt/tree/main/data_cognitive_tasks |
| Responses | 8,244 |
|---|---|
| Respondents | 1,188 |
| Items | 7 |
| Response categories | 2 |
| Responses per respondent | 6.939 |
| Responses per item | 1177.714 |
| Density | 0.991 |
| Longitudinal | TRUE |
| age range | Adult (18+) |
|---|---|
| construct type | Behavioral |
| measurement tool | Test |
| item format | Likert Scale/selected response |
| primary language(s) | eng |
| construct name | Tendency to override intuitive answers using the Cognitive Reflection Test (CRT) |
These are classifications the IRW records, not the study’s own words. Some were annotated by hand and some written by an automated tagger whose output is published only where it clears a measured accuracy bar; which values came from which is not recorded per table. Coverage and accuracy per column, and what is held back, are on Tag quality.
This table is in 1 collection. Each membership records how it was decided:
longitudinal — by ruleMembership is the IRW’s judgment and is worth checking before you use a collection as a sample. A hand-curated list is a snapshot: it is not re-checked when tables are added, so a table that would qualify may be missing. See Collections.
This codebook is reconstructed by the IRW from the IRW data standard. It is our best reading, not the source’s own codebook, and it may contain mistakes. Where the two disagree, the source data are the authority. If you find an error, please tell us.
| Column | What it is | Source column |
|---|---|---|
| IRW standard columns | ||
id | IRW standard Identifier for the focal unit being measured — typically a person, but sometimes another entity (e.g., a word in a lexical task). | not traced |
item | IRW standard Item identifier. | not traced |
resp | IRW standard Response value. | not traced |
wave | IRW standard Longitudinal wave indicator. | not traced |
| Respondent covariates | ||
cov_session_id | Covariate | not traced |
What item codes and covariate values stand for is in the source’s documentation, data_cognitive_reflection.docx.
Or load it directly in R (no account needed with irw 1.4.0 or later) or Python (needs a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("irw")
library(irw)
df <- irw_fetch("himmelstein-cognitive_reflection-2025")
# Python
pip install irw
import irw
df = irw.fetch("himmelstein-cognitive_reflection-2025")
Or in Stata, which reads the same no-account CSV as the button above:
* Stata
net install irw, from("https://raw.githubusercontent.com/itemresponsewarehouse/Stata-pkg/main/")
irw_fetch himmelstein-cognitive_reflection-2025, clear
@article{Himmelstein_2025, title={The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability}, url={http://dx.doi.org/10.31234/osf.io/a7kdx_v8}, DOI={10.31234/osf.io/a7kdx_v8}, publisher={Center for Open Science}, author={Himmelstein, Mark and Zhu, Sophie Ma and Petrov, Nikolay and Karger, Ezra and Helmer, Jessica and Livnat, Sivan and Bennett, Amory and Hedley, Page and Tetlock, Phil}, year={2025}, month=nov }
| IRW version | v674 |
|---|---|
| Redivis dataset | item_response_warehouse_2 v35.0 |
| Redivis dataset DOI | 10.57761/kpjk-f829 |
| Manifest pin for this IRW version | v35.0 |
| Metadata source | irw_meta v59.0 |