Overview of the IRW Data Standards
Data standards are an essential mechanism of increasing reusability and interoperability of data amongst researchers. As such, they play a critical role in increasing the extent to which science is FAIR (findable, accessible, interoperable, reusable). The IRW’s response data come in several data families that use distinct data standards. We focus here on describing the families and the differences between their standards (with pointers to complete descriptions of the specific standards).
The data families
Every response table in the IRW belongs to exactly one data family, and each family is kept in its own Redivis dataset. In the irw packages, the source argument selects the family.
| Family | source = |
What defines it |
|---|---|---|
| Core | "core" (the default) |
A person responds to an item; the responses are ordered or continuous |
| Nominal | "nom" |
As in core, but the response categories have no order |
| Competitions | "comp" |
Two entities meet and an outcome is recorded: a game, or a choice between two fixed things such as two essays |
| Simulated and synthetic | "sim" |
Responses that no real person produced |
| Conjoint | "conj" |
Respondents choose between, or rate, profiles whose attributes are assigned at random |
| Rankings | "rank" |
Each respondent puts a fixed set of named options in order: ranked-choice ballots, jury rankings, survey ranking questions |
Five families are defined by the shape of the data. Simsyn is defined by where the data come from: a simulated table can have any shape, and it belongs to simsyn rather than to the family its shape would suggest.
Core, competitions, conjoint and rankings
Set simsyn aside, and fold nominal into core (it differs only in that the response categories are unordered). That leaves four shapes of data. What separates them is what gets compared, and what we would want a parameter for.
- Core. A person responds to an item. Persons and items are categorically different things, and both persist: the same item is answered by many people. The usual model gives each of them a parameter, an ability and a difficulty, say.
- Competitions. There is no item. Two entities meet—teams in a game, chess engines, two essays put in front of a judge, two named options put to a voter—and an outcome is recorded. The entities recur across many comparisons, and the target is a parameter for each of them (an Elo rating, a Bradley–Terry strength). Where there is a judge, the judge is the instrument rather than the object of measurement.
- Conjoint. A person chooses between, or rates, profiles (hypothetical candidates, immigrants, policies) whose attributes are assigned at random. A profile is built for one task and never seen again, so there is nothing to give a parameter to. The target is instead the effect of each attribute on the response, and how that effect varies across people. In IRT terms, this sits closest to an explanatory model such as the LLTM, wherein the items are generated on the fly from their features.
- Rankings. A person puts a fixed set of named options in order: candidates on a ranked-choice ballot, songs before a jury, policies in a survey. The options keep their identity, as items do, but the response is a position, and one ranking compares all the options at once rather than judging each on its own. A Plackett–Luce model gives each option a worth, as a Bradley–Terry model does in competitions. What is peculiar to the data is how incomplete a ranking can be: a voter may stop after two choices, skip a position, or mark the same candidate twice, and the rankings standard keeps all of that as recorded, because the counting rules that resolve it differ by jurisdiction.
A simple test sorts most new datasets. First ask whether the things being responded to keep a stable identity, so that we would want a parameter for each. If they do not, because each one is assembled fresh from randomized attributes and the question is which attributes drive the response, the data are conjoint. If they do, ask how a person meets them: one at a time is core (nominal when the response categories are unordered), two at a time is competitions, and all at once in order is rankings. Two cases that look like exceptions are not. A “conjoint” study that shows everyone the same fixed set of profiles fails the first question: its profiles have identities, so it belongs in core or competitions. A ranking question that asks only for a single first choice is a pick from a list, which is core or nominal, not rankings.
Item text and the metadata are not families. They hold no responses and describe the tables in the families.
The data standards
In all cases, we are interested in cross-classified responses. The differences amongst the standards pertain to the nature of either the cross-classification or the response.