A conjoint experiment asks people to choose between pairs of randomly built profiles: two immigrants, two candidates, two vaccines. The usual summary is the average effect of each attribute. Read as item response data, each choice is a paired comparison, which lets us ask one measurement question across every conjoint experiment in IRW: how much do a profile’s attributes predict the choice of a respondent nobody has seen yet? Start by taking one of the experiments yourself.
A conjoint experiment shows a respondent two profiles side by side and asks which one they prefer. Each profile is a bundle of attributes (an immigrant’s education, profession, country of origin and so on), and every attribute is drawn at random, independently for each profile. The respondent sees a handful of such pairs. Because the attributes are randomized, comparing how often profiles with one level are chosen against profiles with another level estimates that level’s average effect on the choice, the average marginal component effect or AMCE (Hainmueller et al. 2014). The design is popular in political science because it lets one experiment estimate the effects of many attributes at once. One property of that average is worth flagging early: it pools the tasks where two levels actually met with tasks where each of them met some third level, so it returns a transitive ranking of the levels whether or not head-to-head choices support one (Abramson et al. 2022). The Limitations say what that costs here.
The easiest way to see how this works is to take part in one. The card below deals real pairs from four IRW experiments, and the menu switches between them: immigrants applying for admission to the United States (Hainmueller et al. 2014), packages of terms for Colombia’s peace agreement with the FARC (Tellez 2021), Democratic candidates for state office (Laterzo-Tingley and Christiani 2024), and packs of barley flour in Scottish shops (CHEN et al. 2025). Each time, choose the one you would pick. After each choice you will see what the person who actually saw that pair chose, and the chart below the card turns your choices into AMCEs and sets them against the estimates from everybody who took that experiment. Your choices are kept per experiment, so you can switch and come back.
The four are readable as a card – a handful of attributes, short labels – and they differ in how much their attributes turn out to explain, which is what the rest of the page is about. Taking the last two back to back is the quickest way to feel that difference: in the candidate experiment nothing printed on the card moves the choice much, while in the flour experiment the price alone nearly settles it.
Diamonds mark each attribute’s baseline level, against which the others are measured. Two things stand out after ten or twenty choices. Your own estimates are noisy: the original respondents each made about five choices, which is very little information about one person, while the reference estimates pool hundreds or thousands of people. And in three of the four experiments the pooled effects are modest next to that noise. In the immigration experiment none of them moves the chance of admission by more than 0.28. The flour is the exception: moving from the cheapest to the dearest pack costs it 0.83 of its chance of being bought, which you can see in your own estimates after a handful of choices.
The candidate experiment is the one where taking it tells you least, and that is worth a paragraph of its own. Five features of a Democratic candidate for state office – gender, race, a tax position, a crime position, a schools position – and across 7,445 tasks only one of them is distinguishable from zero: a candidate who runs on prevention rather than on being tough on crime gains about 0.02 (SE 0.008). That is why the red dot for this experiment sits at the bottom of the next figure. A model that knows the five features and nothing about the respondent predicts a held-out voter’s choice no better than a coin, and the IMV scale floors there; the companion table from the same study, with student respondents, comes out at zero too.
The reading to avoid is that race or crime policy does not matter to these choices. The averages are small partly because respondents pull against each other: Democrats in this sample lean toward the Black candidate (+0.03, SE 0.01) and independents away from them (-0.03, SE 0.02), which is the kind of difference the study was built to find and the kind this page’s model is deliberately blind to. The question here is a narrow one – what do the printed attributes alone buy you against a stranger’s choice – and for a pair of Democratic candidates who differ on five features, the honest answer is nothing. Taking a few of these pairs and then switching to the flour makes the contrast concrete: here you are guessing, there you can feel yourself deciding on price.
The published estimates plotted here are not borrowed: all 41 of them, and their standard errors, come back out of the IRW table hainmueller_2014_immigrant to at least ten decimal places, that is, exactly. The conjoint standard shows how to read a conjoint table and fit the model that gives them. The simple difference in choice rates the widget uses, applied to all 1,396 respondents, is within 0.009 of every AMCE.
The rest of this page steps back from the average effects and asks a predictive question instead. Read as item response data, each task is a paired comparison: two objects, one chosen. That is the same shape as the comparisons elsewhere in IRW, except that the objects are built at random rather than fixed. So we can ask: how much do the attributes explain? How much better than a coin flip does a model of the attributes predict the choices of a respondent it was not fitted to? We score that with the InterModel Vigorish (IMV, Domingue et al. 2025, 2024), which the IMV vignette introduces, and we ask it of every eligible conjoint experiment in IRW and of the item side of the warehouse for comparison.
The IRW conjoint source holds 1143 tables across its two shards. We use those with a forced choice between exactly two profiles and at least 500 usable tasks. A task is one choice screen: a respondent sees a pair of profiles and picks one, so a study with 1,000 respondents who each saw five pairs has 5,000 tasks. A usable task is one in which exactly one of the two profiles was chosen, which drops opt-outs and unanswered screens. The 500 is a total over all respondents, not a count per respondent:
meta <- read.csv(paste0("https://redivis.com/api/v1/tables/datapages.irw_meta:v44_0.",
"conj_metadata/rows?format=csv"))
candidates <- subset(meta, grepl("(^|;)choice(;|$)", outcomes) & n_profiles == 2 &
n_respondents * n_tasks >= 500)$table728 tables met this screen as of October 2026, all of them with at least 500 usable tasks. Tables with more than 5,000 respondents were subsampled to 5,000 at random.
For each table, the two profiles in a task are coded as attribute dummies and the model works with their difference, so a task becomes one binary outcome: was the first profile chosen? Attributes that were not shown on a profile get a level of their own.
The choice is then regressed on the difference in attribute dummies by L2-penalised logistic regression (penalty 1, intercept unpenalised), fitted on four fifths of the respondents and used to predict the other fifth, rotating over five folds. The baseline is a coin weighted to the training share of first-profile choices, and the IMV compares the two. Whole respondents are held out, so the number says how well a design’s attributes predict the choices of someone the model has never seen.
An IMV of some size is hard to read on its own, so we compute the same quantity on ordinary item response data, where the counterpart of an attribute is an item. We sampled 200 tables at random from the 254 dichotomous core IRW tables that are cross-sectional, at least 90% complete, and have 300 to 20,000 respondents and 5 to 100 items. The design is the one above with items in place of attributes: whole persons are held out, a fifth at a time; the model is each item’s difficulty, the logit of its mean among the training persons; and the baseline is a coin weighted to the training share of correct or endorsed responses. 193 of the 200 gave results (a table whose responses turn out not to be dichotomous is dropped).
The two sides are not two estimates of one quantity. Each says, for one kind of design, how much the features of the thing being judged predict a stranger’s response to it.
Start with one experiment, the immigration study the card opens on (the arithmetic below is for that one whichever you took). Fitting the model above to hainmueller_2014_immigrant – on four fifths of the 1,396 respondents at a time, predicting the fifth left out – gives a held-out probability for each of its 6,980 tasks: the chance that this respondent, whose choices the model never saw, picked the first of the two applicants. The model is often fairly sure. Its predictions run from 11% to 90% at the fifth and ninety-fifth percentiles, and across that range they are about right.
cal <- example_pred |>
mutate(bin = ntile(p_attr, 10)) |>
group_by(bin) |>
summarise(pred = mean(p_attr), obs = mean(y), n = n())
ggplot(cal, aes(pred, obs)) +
geom_abline(linetype = "dashed", colour = irw_grey) +
geom_point(aes(size = n), colour = irw_blue, alpha = 0.8) +
scale_size_area(max_size = 5, guide = "none") +
coord_fixed(xlim = c(0, 1), ylim = c(0, 1)) +
labs(x = "Predicted chance the first applicant is admitted",
y = "Share actually admitted")
hainmueller_2014_immigrant, grouped into ten equal bins: the model’s average predicted probability that the first applicant was admitted against the share of those tasks in which the respondent did admit them. Points on the diagonal are predictions that come true at the rate claimed. Point size is the number of tasks in the bin.
To turn that into one number we compare two predictions of the same held-out choices: the attribute model, and a coin weighted to the share of first-applicant choices in the training respondents. The IMV converts each model’s average log-likelihood into the accuracy of a weighted coin that would earn it, and reports the relative gain.
mean_ll <- function(y, p) mean(y * log(p) + (1 - y) * log(1 - p))
# The accuracy w of a coin whose log-likelihood matches this model's.
w_of <- function(ll) {
if (ll <= log(0.5)) return(0.5)
uniroot(function(w) w * log(w) + (1 - w) * log(1 - w) - ll, c(0.5, 1 - 1e-12))$root
}
ll_base <- mean_ll(example_pred$y, example_pred$p_base) # weighted coin
ll_attr <- mean_ll(example_pred$y, example_pred$p_attr) # attribute model
w_base <- w_of(ll_base)
w_attr <- w_of(ll_attr)
imv <- (w_attr - w_base) / w_base
c(ll_base = ll_base, ll_attr = ll_attr, w_base = w_base, w_attr = w_attr, imv = imv) ll_base ll_attr w_base w_attr imv
-0.6932891 -0.5617500 0.5000000 0.7505535 0.5011070 The applicants’ attributes are randomized, so the coin is right 50% of the time – which is what makes a conjoint a clean test. The attribute model predicts these choices like a coin that comes up right 75.1% of the time, and the IMV is the gain over the baseline on that scale: 0.50. Nine attributes, none of which moves the average chance of admission by more than 0.28, together carry half again as much predictive accuracy as knowing nothing at all.
That is a lot, and this experiment is not typical: 91% of the conjoint experiments in IRW come out below it.
Across 728 experiments the median IMV of the attribute model over a coin flip is 0.30. In accuracy terms, the median attribute model predicts a held-out choice like a coin that comes up right 67% of the time, against 51% for the baseline. That is a substantial amount for a randomly assembled bundle of five or ten features: knowing nothing about a respondent, and only what the two profiles look like, gets you most of the way from a coin flip to two choices in three.
The spread is wide. Eight in ten experiments fall between 0.13 and 0.5, and designs with more attributes are not more predictable (Spearman \(\rho\) = -0.02 between the IMV and the number of attributes). More attributes give the model more to work with but also more ways for a respondent to trade one against another, and the two appear to cancel.
# The four experiments a reader can take, labelled in the words the menu uses.
q1_widget <- q1 |>
filter(table %in% widget_tables) |>
left_join(tibble(table = c("hainmueller_2014_immigrant", "tellez_2021_peace_packages",
"laterzotingley_2024_crime_national", "chen_2025_bere_barley_scot"),
tag = c("immigrants", "peace agreements", "candidates", "barley flour")),
by = "table")
p1 <- ggplot(q1, aes(n_attr, imv_attr,
text = paste0(table, "<br>IMV = ", round(imv_attr, 3),
"<br>attributes: ", n_attr, "<br>usable tasks: ", n_tasks_used))) +
geom_jitter(width = 0.15, height = 0, colour = irw_blue, alpha = 0.7, size = 2) +
geom_hline(yintercept = q1_med, linetype = "dashed", colour = irw_grey) +
geom_point(data = q1_widget, colour = irw_red, size = 3.5) +
scale_x_continuous(expand = expansion(mult = c(0.03, 0.12))) +
labs(x = "Number of attributes", y = "IMV, attributes over a coin flip")
# ggplotly centres geom_text, so the four tags are plotly annotations instead,
# anchored to the right of their point.
ggplotly(p1, tooltip = "text") |>
layout(annotations = lapply(seq_len(nrow(q1_widget)), function(i) list(
x = q1_widget$n_attr[i], y = q1_widget$imv_attr[i], text = q1_widget$tag[i],
xref = "x", yref = "y", xanchor = "left", yanchor = "middle", xshift = 8,
showarrow = FALSE, font = list(color = irw_red, size = 11),
bgcolor = "rgba(255,255,255,0.78)", borderpad = 1)))An IMV of 0.30 is easier to read next to the same quantity computed the same way on ordinary item response data. There, across 193 dichotomous tables, knowing which item a person is answering raises held-out prediction by a median IMV of 0.085, roughly a quarter of what a profile’s attributes do for a choice.
meds <- both |> group_by(family) |> summarise(m = median(imv))
p2 <- ggplot(both, aes(imv, family, text = paste0(table, "<br>IMV = ", round(imv, 3), "<br>", detail))) +
geom_jitter(aes(colour = family), height = 0.2, width = 0, alpha = 0.6, size = 2) +
geom_point(data = meds, aes(m, family), inherit.aes = FALSE, shape = 23, size = 4,
fill = "white", colour = "black", stroke = 1.2) +
geom_vline(xintercept = 0, colour = irw_grey) +
scale_colour_manual(values = c(irw_red, irw_blue), guide = "none") +
labs(x = "IMV over a weighted coin", y = NULL)
ggplotly(p2, tooltip = "text")Two things keep that gap from being a like-for-like contest between designs. The first is the baseline. A forced choice between two randomly built profiles is a coin flip by construction, so the baseline accuracy is almost exactly 51%, and any predictive signal has room to show. Items are not balanced that way: the median dichotomous table here is already right 67% of the time by always guessing the more common response, and relative gains over a baseline that good are necessarily smaller (the item model reaches 77%). The second is what the features are. A conjoint’s attributes are the whole stimulus, varied on purpose across the range the designer chose; an item’s difficulty is one number summarising a question somebody wrote for other reasons. A conjoint is, in effect, an experiment built to make the stimulus matter, and the comparison shows how much that design choice buys.
| What is being chosen | Experiments | Median IMV | Median held-out accuracy |
|---|---|---|---|
| Places | 52 | 0.37 | 70% (from 52%) |
| Products and services | 93 | 0.33 | 71% (from 51%) |
| Scenarios | 38 | 0.31 | 68% (from 52%) |
| Organisations | 25 | 0.31 | 67% (from 51%) |
| People | 272 | 0.30 | 66% (from 51%) |
| Policies | 234 | 0.27 | 66% (from 51%) |
| Messages | 14 | 0.26 | 63% (from 51%) |
The conjoint literature is dominated by choices between people – candidates, immigrants, job applicants – and between policies, but those are not where the attributes do the most work. Choices between things, places to live and products to buy, are the most predictable, and choices between policies and between political messages the least. A plausible reading is that a profile’s attributes come closer to exhausting what there is to know about a product than about a policy or a candidate, where what a respondent brings to the choice matters more.
Even so, the differences between kinds of choice are small next to the variation within them, and that is the more striking fact. At the top, chen_2025_bere_barley_scot reaches an IMV of 0.82, predicting held-out choices 91% of the time from 6 attributes; at the bottom, 10 experiments with designs of much the same size do not beat a weighted coin at all. What the attributes explain is a property of the particular experiment, not of the method.
irw_conjoint v4.2 and irw_conjoint_2 v1.4. Later versions will add experiments; rerunning means changing the version strings at the top of conjoint_compute.R.joo_2026_descriptive_rep meets the criteria but is excluded: in its deposit every respondent’s chosen candidate has the same position in all six tasks, so its outcomes do not line up with its profiles.Source code for this page: conjoint.qmd · conjoint_compute.R
These results were computed against approximately IRW v584 (the corpus as of October 17, 2026).
Results were computed on October 17, 2026 from irw_conjoint v4.2, irw_conjoint_2 v1.4 and irw_meta v44.0. To regenerate:
# 1. Re-run the compute script (from the project root)
source("vignettes/conjoint_compute.R")
# 2. Re-render this page
quarto::quarto_render("vignettes/conjoint.qmd")