These vignettes show how to work with IRW data for common psychometric tasks. Each row is one vignette: click its title to read it, or click anywhere else on the row for a longer description and the key papers behind it.
Many of the vignettes below — including the dataset search, compute scripts, analysis design, and writing — were produced largely by Claude (Anthropic), working from a task specification and with human review. Individual pages note this explicitly where it applies; if you’re relying on the numbers shown, check the underlying compute scripts directly.
Many are also my attempts to explore new ideas rather than settled results, and they should be read that way: suggestive, sometimes wrong, and a starting point for your own analysis rather than a citation.
Each of these vignettes uses the irw R package to fetch data and walks through a complete analysis. They fall into four rough genres, grouped below.
Introductions to a technique or tool, worked through on one example dataset.
A short worked example computing cross-validated InterModel Vigorish (IMV) for one dataset, with a pointer to an interactive app for building intuition about what IMV values mean.
Key paper: Domingue et al. (2025), Psychometrika
Also see: the IMV Shiny app, for building intuition about what different IMV values mean
Draws from a pooled empirical distribution built from 145+ IRW datasets via irw_simu_diff(), instead of assuming a normal or uniform shape.
Key paper: Zhang et al. (2025), PsyArXiv
Walks through a full lavaan workflow on a Big Five dataset — cleaning, reshaping to wide format, and building a multi-factor model directly from item-naming conventions — that generalizes to most other multi-scale IRW tables.
Broad scans checking whether an assumption standard IRT models rely on actually holds, across hundreds of real IRW tables at once, rather than in a single example dataset.
Compares two exploratory diagnostics — the eigenvalue ratio and parallel analysis — across hundreds of tables and finds they disagree sharply on how often it holds.
Key paper: Horn (1965), Psychometrika (parallel analysis)
Flags locally dependent pairs (testlet effects) via Yen’s Q3 statistic across ~800 tables and checks two known testlet structures as a sanity check.
Key paper: Yen (1984), Applied Psychological Measurement (Q3 statistic)
LSIRM adds a single respondent-item distance term and produces an “interaction map” where items and respondents sharing unexplained structure are pulled close together. Fit to 16 IRW tables, it splits by response format so cleanly it forced a validity check on the estimator itself.
Key paper: Jeon, Jin, Schweinberger & Baugh (2021), Psychometrika (LSIRM)
Runs both DIF families side by side on every IRW dataset with a gender/sex covariate.
Takes a specific claim or theoretical result from a published paper and checks whether it holds up against real IRW data.
Six sections moving from raw manifest probabilities to a Rasch-model diagnostic and a test of Holland’s conservation-of-complexity law.
Key paper: Holland (1990), Psychometrika
Also see: the IMV Shiny app, for the IMV values used in Section 6
Tests four IRW tasks — Hearts & Flowers, an alcohol Stroop, a stop-signal task, and three Self-Regulation Ontology tasks — for the same attenuation.
Key paper: Hedge, Powell & Sumner (2018), Behavior Research Methods
Tests whether a network’s node centrality tracks IRT discrimination across real IRW instruments, and whether disagreement concentrates in tables already flagged as multidimensional or locally dependent elsewhere on this site.
Key paper: Epskamp, Maris, van Borkulo & Borsboom (2018), in The Wiley Handbook of Psychometric Testing
The original paper never fits any of the three to real item response data; this vignette runs a real-data IMV holdout comparison to fill that gap.
Key paper: Shim & Bonifay (2026), British Journal of Mathematical and Statistical Psychology
Compares Noel & Dauvier’s Beta IRT, Samejima’s Continuous Response Model, and Mueller’s Continuous Rating Scale Model against a naive linear baseline, on a full simulation grid and 11 real IRW tables — and finds the naive baseline loses badly on distributional fit but is often competitive on point predictions.
Key paper: Noël & Dauvier (2007), Applied Psychological Measurement (Beta IRT)
Compares three published lenses on guessing — ability-based guessing (San Martín, Del Pino & De Boeck, 2006), person-level purification (Torres Irribarra, Echeverría & Espinoza, 2026), and a two-class “safety valve” mixture (Xiao, Ulitzsch, Zhang, Frank & Domingue, 2026) — plus the standard Rasch/2PL/1PLg/3PL baselines, on 11 real IRW tables (eight ENEM admissions-exam tables and three from a literacy-intervention study) using out-of-sample IMV. Finds that estimating a guessing floor helps on every table, while fixing it at 1/m can cost you.
Key papers: San Martín, Del Pino & De Boeck (2006), Applied Psychological Measurement; Xiao, Ulitzsch, Zhang, Frank & Domingue (2026)
A recent preprint argues that ignoring the floor biases the person-level correlations DSEM is fitted to estimate, and its remedy is a censored model. But censoring is only one theory of what a floor is: a bounded link models the boundedness directly, and a graded model says the floor was never special. Measures how much floor mass IRW’s intensive longitudinal tables actually carry, shows that the composite scores the field models hide almost all of it, and puts the three families against each other on a study that randomised participants to a Likert or a slider version of the same protocol — where carryover survives the choice of floor model and the correlation the preprint cares about does not.
Key papers: Beneath the Floor: Censored DSEM Models for Analyzing ESM Data with Floor Effects (2026), PsyArXiv preprint; Vollbracht, Ottenstein, Ecker & Lischetzke (2026), Behavior Research Methods
Applied methods and novel analyses that use IRW’s breadth to ask a research question that a single dataset couldn’t answer on its own.
Across nearly every table in the competition collection (a century of team sports, Lichess chess, game-playing bots, human judgments and animal dominance), tests whether cycles are more common than a transitive Bradley-Terry model produces by chance, and shows why counting cycles, or a naive bootstrap, misleads.
Also see: the course’s competitions lesson
Fits a 2PL to every eligible table and asks how consequential the Rasch model’s a=1 constraint really is in practice.
Extracts features like length, negation, and complexity from item text and tests how much of item difficulty they explain, separate from substantive content.
Across NBA shots, NFL kicks and passes, soccer shots and adaptive map practice, asks how much adjusting for the attempt (or for a coach’s or practice system’s choice of attempt) changes who looks good, and in which direction.
Also see: the course’s trials lesson, which uses the simulated nbashots_sim
Fits a Bradley-Terry model with a home term to every season of baseball, basketball, hockey, American football and English and Italian football back to the 1870s, and compares the 2020-21 seasons played without crowds.
Also see: the course’s competitions lesson, which fits the same home term to one NBA season
Uses cross-validated IMV to isolate the predictive value response time adds beyond ability alone.
Key paper: Domingue et al. (2022), Journal of Educational and Behavioral Statistics
Also see: the IMV Shiny app, for building intuition about what different IMV values mean
Compares a 2PL with the nominal response model and the nested logit, out of sample, across the IRW’s nominal tables. It uses the IMV for right/wrong and, through Kanopka’s polytomous IMV (ωc), for the option chosen and for which wrong answer.
Key papers: Bock (1972), Psychometrika; Suh & Bolt (2010), Psychometrika; Kanopka (2023), Stanford dissertation; Rivadeneira Quintero et al. (2026), PsyArXiv
Shows that item-level variation in treatment response can bias estimated interactions even when the average treatment effect looks fine.
Key paper: Gilbert et al. (2025), Journal of Policy Analysis and Management
Applies the released code and models to 117 IRW tables: the predictions track the validity criteria about as well as on the original data, but they recommend the same measure for most tables.
Key paper: Sotoudeh & DiMaggio (2021), Sociological Methods & Research
Take one of the experiments yourself, then see the answer across 728 conjoint experiments in IRW: the attributes carry a median held-out IMV of 0.30, lifting a coin flip to two choices in three, while the same calculation on ordinary item response data gives item difficulty about a quarter as much work. Choices between places and products are the most predictable, policies and political messages the least, and the spread within each kind dwarfs the differences between them.
Key paper: Hainmueller, Hopkins & Yamamoto (2014), Political Analysis
Also see: the conjoint data standard
Builds the idea up from perfect cuts to real items on an interactive numeracy test, then counts the bits in more than two hundred dichotomous IRW tables: the median test carries about one bit, reliability undercounts the bits, the sum score loses little under the 2PL and nothing under Rasch, and ordered-category items carry more than one bit but far less than their ceiling.
Also see: Olah, Visual Information Theory, for the message-length reading of entropy used throughout
These vignettes use the irw Python package to fetch data.
IRT Models in Python irwmirtgirth