70,103 responses from 1,194 respondents to 90 items.
One of 8 tables from the same source; this one measures Performance on a mix of general-knowledge and impossible (unknowable) questions. The others are listed under Related tables.
| Description | This dataset measures performance on a mix of general-knowledge and impossible. |
|---|---|
| Reference | Himmelstein, Mark, Sophie Ma Zhu, Nikolay Petrov, Ezra Karger, Jessica Helmer, Sivan Livnat, Amory Bennett, Page Hedley, and Phil Tetlock. “The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability”. PsyArXiv, November 23, 2025. doi:10.31234/osf.io/a7kdx_v8. |
| DOI | 10.31234/osf.io/a7kdx_v8 |
| Licence | CC BY 4.0 |
| Source data | https://github.com/forecastingresearch/fpt/tree/main/data_cognitive_tasks |
| Responses | 70,103 |
|---|---|
| Respondents | 1,194 |
| Items | 90 |
| Response categories | 2 |
| Responses per respondent | 58.713 |
| Responses per item | 778.922 |
| Density | 0.652 |
| Longitudinal | TRUE |
| age range | Adult (18+) |
|---|---|
| construct type | Cognitive/educational |
| measurement tool | Test |
| item format | Likert Scale/selected response |
| primary language(s) | eng |
| construct name | Performance on a mix of general-knowledge and impossible (unknowable) questions |
These are classifications the IRW records, not the study’s own words. Some were annotated by hand and some written by an automated tagger whose output is published only where it clears a measured accuracy bar; which values came from which is not recorded per table. Coverage and accuracy per column, and what is held back, are on Tag quality.
This table is in 2 collections. Each membership records how it was decided:
longitudinal — by ruleresponse_time — by ruleMembership is the IRW’s judgment and is worth checking before you use a collection as a sample. A hand-curated list is a snapshot: it is not re-checked when tables are added, so a table that would qualify may be missing. See Collections.
This table has item text in the IRW: the wording administered to respondents, not just the response codes.
| Instrument | This dataset measures accuracy on a two-alternative forced-choice general-knowledge task that also contains unanswerable ('impossible') questions, on which the correct response is to opt out. |
|---|---|
| Mean words per item | 10.600 |
| Mean characters per item | 60.967 |
| Mean characters per response | 2 |
| Flesch-Kincaid grade level | 7.068 |
This codebook is reconstructed by the IRW from the IRW data standard and the script that built it, data_impossible_question.py. It is our best reading, not the source’s own codebook, and it may contain mistakes. Where the two disagree, the source data are the authority. If you find an error, please tell us.
| Column | What it is | Source column |
|---|---|---|
| IRW standard columns | ||
id | IRW standard Identifier for the focal unit being measured — typically a person, but sometimes another entity (e.g., a word in a lexical task). | item_col |
item | IRW standard Item identifier. | not traced |
resp | IRW standard Response value. | not traced |
rt | IRW standard Response time. | not traced |
wave | IRW standard Longitudinal wave indicator. | not traced |
| Respondent covariates | ||
cov_session_id | Covariate | not traced |
What item codes and covariate values stand for is in the codebook released with the source data.
Or load it directly in R (no account needed with irw 1.4.0 or later) or Python (needs a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("irw")
library(irw)
df <- irw_fetch("himmelstein-impossible_question-2025")
# Python
pip install irw
import irw
df = irw.fetch("himmelstein-impossible_question-2025")
Or in Stata, which reads the same no-account CSV as the button above:
* Stata
net install irw, from("https://raw.githubusercontent.com/itemresponsewarehouse/Stata-pkg/main/")
irw_fetch himmelstein-impossible_question-2025, clear
@article{Himmelstein_2025, title={The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability}, url={http://dx.doi.org/10.31234/osf.io/a7kdx_v8}, DOI={10.31234/osf.io/a7kdx_v8}, publisher={Center for Open Science}, author={Himmelstein, Mark and Zhu, Sophie Ma and Petrov, Nikolay and Karger, Ezra and Helmer, Jessica and Livnat, Sivan and Bennett, Amory and Hedley, Page and Tetlock, Phil}, year={2025}, month=nov }
| IRW version | v674 |
|---|---|
| Redivis dataset | item_response_warehouse_2 v35.0 |
| Redivis dataset DOI | 10.57761/kpjk-f829 |
| Manifest pin for this IRW version | v35.0 |
| Metadata source | irw_meta v59.0 |