Shim & Bonifay (2026) introduce three IRT models with asymmetric item response functions – LPE, RH, and AO – and show, via simplex-sampled synthetic data, that two of them fit about as well as a standard 2PL even when fitting propensity alone is compared, while the third (RH) is more parsimonious. They never apply the models to real data. We do: after a dedicated correctness/recovery validation gate, we fit a 2PL reference plus all three asymmetric models across dozens of binary IRW tables and score every model on genuinely held-out responses (InterModel Vigorish, not in-sample BIC). The two metrics tell very different stories: RH looks like the dominant winner by BIC, but is the weakest of the three – often the worst model in the batch – once judged on real predictive accuracy.
The three asymmetric models below are easier to feel than to read as equations. Pick one, drag \(\kappa\), and watch the fitted curve (blue) pull away from the symmetric baseline it’s compared against everywhere in this vignette (grey dashed) — the standard logistic 2PL for LPE and AO, a normal-ogive model for RH.
Every IRT model used elsewhere on this site — the 2PL, the graded response model, the Dutch Identity’s loglinear framework — has a symmetric item response function: the probability of a correct or endorsing response rises smoothly and symmetrically around the item’s difficulty. But nothing requires that. A guessing-prone multiple-choice item, for instance, has a response curve that rises quickly from a nonzero floor and then flattens gradually near the top — asymmetric around its inflection point.
Shim and Bonifay (2026) study three such asymmetric models, each adding a single shape parameter \(\kappa\) to a standard two-parameter item:
In all three, a symmetric baseline is a special case: \(\kappa = 1\) recovers the ordinary logistic 2PL for LPE and AO, and \(\kappa = 0\) recovers a 2-parameter normal-ogive model for RH. The paper’s own contribution is methodological, not empirical: using synthetic data sampled arbitrarily from the response-probability simplex (not generated from any of these models), they show LPE and AO have fitting propensity statistically indistinguishable from the 2PLM — so any real advantage isn’t just an artifact of the extra parameter — while RH is more parsimonious, fitting arbitrary data somewhat less readily. They never fit any of the three models to real item response data.
That’s the gap this vignette fills: do LPE, RH, and AO actually improve on a standard 2PL across real IRW instruments, and — since Shim & Bonifay frame asymmetry around phenomena like guessing (a leniency effect concentrated at the low end of \(\theta\)) — does the direction of the fitted asymmetry relate to construct_type, e.g. more leniency in cognitive/educational items than in opinion/attitude items?
Implementing three custom, hand-derived item types is the highest-risk part of any vignette on this site, so before touching a single real IRW table we ran a dedicated validation gate — shown next — and only proceeded to the real data once it passed.
Each model type was implemented as a custom mirt item, with the paper’s own \(P\)/\(dP\)/\(d^2P\) math (copied, not re-derived) cross-checked against the authors’ own reference implementation. Three checks had to pass before any real IRW table was fit.
At \(\kappa = 1\) (LPE, AO) or \(\kappa = 0\) (RH), each custom item’s \(P(\theta)\) should exactly reproduce its symmetric reference model.
reduction_check |>
mutate(max_abs_diff = formatC(max_abs_diff, format = "e", digits = 2)) |>
rename(Model = model, Reference = reference, `Max |diff|` = max_abs_diff, Pass = pass) |>
knitr::kable(align = "llrc")| Model | Reference | Max |diff| | Pass |
|---|---|---|---|
| AO | 2PLM (logistic) | 1.11e-16 | TRUE |
| LPE | 2PLM (logistic) | 1.11e-16 | TRUE |
| RH | 2-parameter normal-ogive (probit) | 1.11e-16 | TRUE |
All three differences are at machine precision (1.1e-16 or smaller) — the custom items reduce to their reference models exactly, not just approximately.
Shim and Bonifay (2026) (Section 2.1) prove that LPE curves at different \(\kappa\) never cross: for \(\kappa < 1\) the curve sits at or above the \(\kappa = 1\) curve at every \(\theta\), and for \(\kappa > 1\) it sits at or below it. This is a strong, cheap check directly on the \(P\) function.
lpe_df <- as.data.frame(lpe_curves) |>
mutate(theta = theta_grid) |>
pivot_longer(-theta, names_to = "kappa_label", values_to = "p") |>
mutate(kappa = as.numeric(sub("^k", "", kappa_label)),
kappa_label = factor(paste0("κ = ", kappa),
levels = paste0("κ = ", sort(kappa_vals))))
ggplot(lpe_df, aes(x = theta, y = p, colour = kappa_label)) +
geom_line(linewidth = 0.9) +
scale_colour_viridis_d(name = NULL, option = "plasma", end = 0.9) +
labs(x = expression(theta), y = expression(P(theta)))
Non-crossing holds across every pair of \(\kappa\) values checked (0.25, 0.5, 0.75, 1, 2, 3): TRUE.
Finally, we simulated data from each model at known, clearly asymmetric \(\kappa\) values (\(N = 2{,}000\) persons, 20 items — matching the real-data scale used below) and fit the corresponding custom item to see whether the true shape parameter comes back out.
ggplot(recovery_df, aes(x = shape_true, y = shape_est)) +
geom_abline(slope = 1, intercept = 0, linetype = "dashed", colour = irw_grey) +
geom_point(aes(colour = model), size = 2, alpha = 0.8) +
scale_colour_manual(values = model_colors, guide = "none") +
facet_wrap(~model, scales = "free") +
labs(x = "True shape parameter", y = "Estimated shape parameter")
recovery_summary |>
mutate(across(c(cor_shape, mean_abs_error), \(x) round(x, 3))) |>
rename(Model = model, `Correlation` = cor_shape, `Mean abs. error` = mean_abs_error) |>
knitr::kable(align = "lrr")| Model | Correlation | Mean abs. error |
|---|---|---|
| AO | 0.742 | 0.371 |
| LPE | 0.509 | 0.534 |
| RH | 0.958 | 0.352 |
Recovery is strong for RH (\(r = 0.96\)), moderate for AO (\(r = 0.74\)), and weakest for LPE (\(r = 0.51\)) — though clearly better than noise. This check draws only a single simulated dataset per model (20 items each), so these correlations are themselves noisy estimates of each model’s true recoverability at this sample size — LPE’s correlation in particular shouldn’t be read as a precise number, just as “visibly weaker than RH.” Checks (a) and (b) above involve no simulation or optimization and pass at machine precision on every machine we’ve tried; the out-of-sample calibration check in (d) below passes cleanly; and check (e) below sweeps sample size with multiple replicates per cell, giving a more precise picture of how much noise to expect at this scale than this single-dataset check can. Treat LPE’s per-item \(\kappa\) estimates below with the most caution as a result, RH’s with the least, and AO’s in between. The real-data batch below still uses a higher participant floor (\(N \geq 2,000\)) than other IRW vignettes for the same underlying reason: shape parameters are harder to pin down than location/discrimination.
The real-data comparison below scores every model on held-out responses (response-level holdout, not k-fold — a fixed ~20% of each table’s observed cells are masked before fitting, then compared against predictions), using the InterModel Vigorish [IMV; Domingue et al. (2025)] rather than in-sample BIC. Before trusting that pipeline at batch scale, we ran it on synthetic data where the truth is known: does it behave sensibly when there’s genuinely no asymmetry to find, and does an in-sample BIC preference reliably translate into a held-out predictive edge?
Raw IMV values are hard to eyeball without a reference scale, so throughout the Results section below we anchor them against the benchmark ladder in Domingue et al. (2024), which reports both simulation-based and empirical (89 real dichotomous-item datasets, several drawn from the IRW) IMV values for a sequence of familiar IRT modeling upgrades: item-mean → 1PL is large (IMV ≈ 0.1–0.3), 1PL → 2PL is an order of magnitude smaller (IMV ≈ 0.006–0.01), and 2PL → 3PL — adding one extra item parameter, structurally the closest published comparison to adding a single shape parameter to a 2PL as LPE/RH/AO do here — is smaller still (simulation: 0.0005; empirical average across their 89 datasets: 0.00016, max 0.0046). That last row is the natural yardstick for the numbers below.
imv_sanity |>
mutate(truth = recode(truth, null = "Symmetric truth", asymmetric = "Asymmetric truth"),
across(c(mean_imv, sd_imv), \(x) formatC(x, format = "e", digits = 2))) |>
rename(Model = model, Truth = truth, `Mean IMV` = mean_imv, `SD` = sd_imv) |>
arrange(Truth) |>
knitr::kable(align = "llrr")| Model | Truth | reps | Mean IMV | SD |
|---|---|---|---|---|
| AO | Asymmetric truth | -2.189209e-04, 3.378204e-04, -2.632553e-04, -8.599009e-04, 2.798318e-05 | -1.95e-04 | 4.42e-04 |
| LPE | Asymmetric truth | -0.0000871383, -0.0005474392, 0.0001554460, 0.0001066972, -0.0004854767 | -1.72e-04 | 3.28e-04 |
| RH | Asymmetric truth | -0.0040197054, -0.0022518829, -0.0005100824, 0.0006185674, -0.0057577534 | -2.38e-03 | 2.58e-03 |
| AO | Symmetric truth | -0.0001671149, -0.0002419086, -0.0001800227, -0.0007092747, -0.0002920367 | -3.18e-04 | 2.24e-04 |
| LPE | Symmetric truth | -0.0001363337, -0.0002385595, -0.0002346335, -0.0008014219, -0.0002992128 | -3.42e-04 | 2.63e-04 |
| RH | Symmetric truth | -7.287468e-04, -5.344300e-04, 7.828173e-06, -1.402013e-03, -2.624495e-04 | -5.84e-04 | 5.35e-04 |
Under a genuinely symmetric truth, mean held-out IMV sits close to zero for all three models (confirmed: \(|{\rm mean\ IMV}| < 0.01\)) — the holdout/prediction pipeline itself isn’t biased toward finding an advantage that isn’t there. But under a known, realistic-magnitude asymmetric truth, held-out IMV was not reliably positive — RH in particular showed a small negative mean IMV even when the data-generating process really was asymmetric, despite a large in-sample BIC preference for RH in the same simulations (500 to 1,500+ BIC points, depending on the run). We confirmed this wasn’t a computation bug by manually reconstructing each model’s log-likelihood from the same extract.item()/probtrace() calls used for held-out prediction and checking it matched the model’s own reported in-sample fit direction — it did. The extra shape parameter can add enough person-ability-estimation noise to offset its true signal at realistic sample sizes, even when that signal is real.
That finding is the reason this vignette scores the real-data batch below by held-out IMV rather than BIC. It also means the result shouldn’t be surprising: if a large in-sample BIC advantage for RH doesn’t reliably survive this same check on synthetic data with a known asymmetric truth, there’s no reason to expect it to survive on real IRW tables either.
All four checks passed (July 21, 2026); only then did we move to real IRW data.
Check (c) above confirms recovery is in the right ballpark at a single sample size (\(N = 2{,}000\)) — the same scale used for the real-data batch. It says nothing about whether the estimator actually converges: whether bias and variance keep shrinking as \(N\) grows, the way a well-behaved MLE/MAP estimator should. Added as a follow-up to the original four-check gate (not a prerequisite for it — the real-data batch above was already complete when this was run), this check refits all three models across a sample-size grid from 250 to 16,000 persons, holding 12 items’ true parameters fixed throughout so that “bias at this \(N\)” is well defined, and redrawing 8 replicate person samples at every sample size.
ggplot(convergence_summary, aes(x = n, y = rmse_shape, colour = model)) +
geom_line(linewidth = 0.9) +
geom_point(size = 2) +
scale_x_log10(labels = scales::label_comma()) +
scale_colour_manual(values = model_colors) +
labs(x = "N (persons, log scale)", y = "RMSE of shape parameter", colour = "Model")
convergence_summary |>
mutate(across(c(bias_shape, rmse_shape, bias_a, rmse_a, bias_d, rmse_d), \(x) round(x, 3))) |>
arrange(model, n) |>
rename(Model = model, N = n, `Fits` = n_fits, `Bias (shape)` = bias_shape,
`RMSE (shape)` = rmse_shape, `RMSE (a)` = rmse_a, `RMSE (d)` = rmse_d) |>
select(Model, N, Fits, `Bias (shape)`, `RMSE (shape)`, `RMSE (a)`, `RMSE (d)`) |>
knitr::kable(align = "lrrrrrr")| Model | N | Fits | Bias (shape) | RMSE (shape) | RMSE (a) | RMSE (d) |
|---|---|---|---|---|---|---|
| AO | 250 | 8 | -0.056 | 0.775 | 0.367 | 0.465 |
| AO | 500 | 8 | 0.027 | 0.721 | 0.345 | 0.416 |
| AO | 1000 | 8 | 0.012 | 0.720 | 0.318 | 0.417 |
| AO | 2000 | 8 | 0.026 | 0.655 | 0.321 | 0.427 |
| AO | 4000 | 8 | -0.039 | 0.580 | 0.262 | 0.362 |
| AO | 8000 | 8 | -0.001 | 0.466 | 0.182 | 0.238 |
| AO | 16000 | 8 | 0.002 | 0.350 | 0.142 | 0.217 |
| LPE | 250 | 8 | -0.012 | 0.761 | 0.757 | 1.670 |
| LPE | 500 | 8 | -0.003 | 0.803 | 0.514 | 1.442 |
| LPE | 1000 | 8 | -0.010 | 0.713 | 0.448 | 1.283 |
| LPE | 2000 | 8 | 0.000 | 0.677 | 0.334 | 1.107 |
| LPE | 4000 | 8 | 0.066 | 0.623 | 0.304 | 1.091 |
| LPE | 8000 | 8 | 0.085 | 0.521 | 0.262 | 0.903 |
| LPE | 16000 | 8 | -0.011 | 0.413 | 0.180 | 0.650 |
| RH | 250 | 8 | 0.018 | 1.168 | 0.231 | 0.211 |
| RH | 500 | 8 | -0.013 | 0.919 | 0.167 | 0.122 |
| RH | 1000 | 8 | -0.013 | 0.795 | 0.112 | 0.088 |
| RH | 2000 | 8 | 0.048 | 0.623 | 0.079 | 0.065 |
| RH | 4000 | 8 | -0.003 | 0.494 | 0.057 | 0.055 |
| RH | 8000 | 8 | -0.028 | 0.402 | 0.034 | 0.029 |
| RH | 16000 | 8 | -0.023 | 0.303 | 0.029 | 0.023 |
RMSE of the shape parameter falls with \(N\) for all three models — by 55% for AO ( 0.78 to 0.35), 46% for LPE ( 0.76 to 0.41), and 74% for RH ( 1.17 to 0.3), moving from the smallest to the largest \(N\) tested. The trend isn’t perfectly monotone at every step for every model — with only 8 replicates per cell there’s real sampling noise in each point in Figure 3 — but none of the three plateaus or reverses direction, which is what a consistent estimator should look like. (The raw RMSE values aren’t directly comparable across models: each model’s true shape values were drawn from a different range, matching check (c)’s convention, so a larger RMSE for one model at a given \(N\) can just as easily reflect a wider true-parameter scale as a harder estimation problem. The within-model trend across \(N\) is the finding here, not the between-model ranking.) That supports treating check (c)’s single-\(N\) recovery numbers, and the real-data batch’s per-item \(\kappa\) estimates below, as genuinely improvable with more data rather than stuck at some irreducible floor. It does not make them precise at the sample sizes IRW tables actually offer: even at the largest \(N\) tested here, shape-parameter RMSE is still several times larger than discrimination or difficulty RMSE at the same \(N\) (see Table 4) — consistent with check (c)’s finding that shape parameters are harder to pin down than location/discrimination, and the reason for this vignette’s higher-than-usual participant floor (\(N \geq 2,000\)).
all_candidates <- irw_filter(
n_categories = 2,
n_items = c(5, 40),
n_participants = c(2000, Inf)
)Binary items only — polytomous extensions of LPE/RH/AO are future work, not attempted here. 65 tables met this criterion as of July 2026; 57 of them produced usable results across all four models (the rest were dropped for too few usable items after cleaning, a model that failed to converge, or an irw_long2resp() limitation with a handful of tables that carry a date column).
For each candidate table, fit_asymmetric_irt():
irw_fetch() + irw_long2resp()), capping at 10,000 respondents and canonicalizing item coding to 0/1 (IRW tables aren’t guaranteed to use 0/1 — some use 1/2 — and the held-out scoring below requires literal 0/1 outcomes).tryCatch; non-convergent fits are logged and dropped rather than retried.fscores(..., method = "EAP")), then predicts the held-out cells from that model’s own item parameters (extract.item() + probtrace()). Compares each asymmetric model’s held-out predictions against the 2PL reference’s via imv::imv.binary(), and records the winning model by IMV.fit_asymmetric_irt <- function(table_name) {
resp <- fetch_wide(table_name) # irw_fetch() + irw_long2resp(), cleaned, 0/1-coded
ho <- mask_holdout(resp, frac = 0.2) # per-person cell mask, >=1 response retained per person
fit_2pl <- mirt(ho$train, 1, itemtype = "2PL", technical = list(customTheta = Theta_mat_ref, ...))
fit_lpe <- fit_custom(ho$train, "LPE", make_custom_item("LPE"), ad_start = extract_ad(fit_2pl), ...)
fit_rh <- fit_custom(ho$train, "RH", make_custom_item("RH"), ad_start = convert_ad_logit_to_probit(...), ...)
fit_ao <- fit_custom(ho$train, "AO", make_custom_item("AO"), ad_start = extract_ad(fit_2pl), ...)
# theta from retained responses only, predictions on held-out cells, IMV vs. the 2PL reference
th_2pl <- fscores(fit_2pl, method = "EAP")
p_2pl_heldout <- heldout_preds(fit_2pl, th_2pl, ho$mask_idx)
# ... same for LPE/RH/AO, then imv::imv.binary(ho$true_vals, p_2pl_heldout, p_model_heldout) ...
}summary_df |>
count(winning_model, name = "n_tables") |>
mutate(pct = round(100 * n_tables / sum(n_tables), 1)) |>
arrange(desc(n_tables)) |>
rename(`Winning model` = winning_model, `# Tables` = n_tables, `%` = pct) |>
knitr::kable(align = "lrr")| Winning model | # Tables | % |
|---|---|---|
| 2PLM | 23 | 40.4 |
| LPE | 15 | 26.3 |
| AO | 12 | 21.1 |
| RH | 7 | 12.3 |
The plain 2PLM wins outright in the largest share of tables. Among the asymmetric models, RH — the model that looked dominant by BIC — wins the least often, behind both LPE and AO. That’s a striking reversal from what in-sample BIC would suggest, and it’s exactly the pattern the validation-gate synthetic check above warned to expect: a large in-sample BIC edge for RH doesn’t reliably survive contact with genuinely held-out data.
ranked_df <- summary_df |>
rowwise() |>
mutate(best_imv = max(c(imv_AO, imv_LPE, imv_RH), na.rm = TRUE)) |>
ungroup() |>
arrange(desc(best_imv)) |>
mutate(
rank = row_number(),
tooltip = paste0(
"Table: ", table,
"<br>Winning model: ", winning_model,
"<br>Best IMV vs. 2PL: ", formatC(best_imv, format = "e", digits = 2),
"<br>n_items: ", n_items, ", n: ", n_participants
)
)
p <- ggplot(ranked_df, aes(x = best_imv, y = rank, colour = winning_model, text = tooltip)) +
geom_vline(xintercept = 0, linetype = "dashed", colour = irw_grey) +
geom_point(size = 2) +
scale_colour_manual(values = model_colors, name = "Winning model") +
labs(x = "Best asymmetric-model IMV vs. 2PL (positive favors asymmetry)", y = NULL) +
theme(axis.text.y = element_blank(), axis.ticks.y = element_blank())
ggplotly(p, tooltip = "text") |>
layout(
legend = list(orientation = "h", y = -0.25, x = 0.5, xanchor = "center"),
margin = list(b = 90)
)There’s no long dramatic tail in Figure 4 the way there was for BIC: IMV values cluster close to zero on both sides, and even the largest positive values are modest by IMV standards — well under 0.01, and in the same neighborhood as Domingue et al. (2024)’s empirical IMV(2PL,3PL) benchmark (average 0.00016, max 0.0046 across their 89 datasets), not their IMV(1PL,2PL) benchmark (average ≈ 0.006–0.01). That places the predictive value of these asymmetric shape parameters alongside one of the weakest published IRT modeling upgrades, not a meaningful one — a world away from the BIC deltas running into the hundreds of points for the same tables. Whatever asymmetric structure is real in these data, its predictive value is small.
bucket_construct <- function(x) {
x <- ifelse(is.na(x), "Unknown", x)
case_when(
grepl("Cognitive", x) ~ "Cognitive/educational",
grepl("Opinion", x) ~ "Opinion/attitude",
x == "Unknown" ~ "Unknown",
TRUE ~ "Other"
)
}
crosstab_df <- summary_df |> mutate(bucket = bucket_construct(construct_type))
crosstab_df |>
count(bucket, winning_model) |>
pivot_wider(names_from = winning_model, values_from = n, values_fill = 0) |>
rename(`Construct bucket` = bucket) |>
knitr::kable(align = "lrrrr")| Construct bucket | AO | LPE | RH | 2PLM |
|---|---|---|---|---|
| Cognitive/educational | 5 | 5 | 2 | 0 |
| Opinion/attitude | 6 | 7 | 2 | 18 |
| Other | 1 | 2 | 1 | 1 |
| Unknown | 0 | 1 | 2 | 4 |
Construct type is far more diagnostic here than it was under BIC (where RH won in every bucket regardless). Among Cognitive/educational tables, the plain 2PLM never wins — some asymmetric model always predicts at least marginally better out of sample. Opinion/attitude tables show the opposite pattern: 2PLM wins the clear majority. That split is a real, substantive finding in its own right, separate from the leniency-direction question below.
lambda_bucket_df <- item_level_df |>
inner_join(crosstab_df |> select(table, bucket, winning_model), by = "table") |>
filter(model == winning_model, bucket != "Unknown")
ggplot(lambda_bucket_df, aes(x = bucket, y = lambda_P)) +
geom_hline(yintercept = 0, linetype = "dashed", colour = irw_grey) +
geom_boxplot(fill = irw_blue, alpha = 0.3, outlier.alpha = 0.4, width = 0.5) +
labs(x = NULL, y = "Л_P (item-level, winning model)")
Cognitive/educational items skew toward leniency (mean \(\text{Л}_P\) = 0.069, 67% of items positive) while Opinion/attitude items skew the other way, toward strictness (mean \(\text{Л}_P\) = -0.039, only 51% positive). That’s a full sign flip between construct types, not just a difference in degree — a cleaner separation than the same comparison showed under the old BIC-based winning model, and squarely the direction the guessing hypothesis predicts (guessing on cognitive items should show up as leniency at the low end of \(\theta\)). Both buckets still have items on either side of zero, so treat this as a real, directionally consistent pattern rather than a hard rule at the individual-item level.
colombia_2023_politics_attitudes has the single largest positive out-of-sample IMV in this batch (LPE beats the 2PL by 7.90e-03 on held-out responses). That’s still comfortably under Domingue et al. (2024)’s IMV(1PL,2PL) benchmark ceiling (empirical max 0.068), but it’s roughly on par with their average IMV(1PL,2PL) (0.006) — i.e., this single best case in the whole batch is about as valuable as a typical 1PL-to-2PL upgrade, while the typical table here (per Figure 4) sits down in IMV(2PL,3PL)-negligible territory. Its item item_p5261s7 (LPE: \(a\) = 7.45, \(b\) = 1.35, \(\kappa\) = 0.1, \(\text{Л}_P\) = 0.465) shows a substantial item-level asymmetry even though the table-level predictive gain is small — below, the fitted LPE curve is plotted against the symmetric logistic curve that the same \(a\) and \(b\) would imply under \(\kappa = 1\) (the exact reduction validated above), i.e. what a standard 2PL would have predicted for this item. Several other items in this same table show the opposite sign (strong implied strictness, \(\text{Л}_P \approx -1\)) at implausibly extreme \(\kappa\) — the thin-data outliers flagged in Limitations below, not shown here. That mix of a real, sizeable item-level effect in one direction and unreliable extreme estimates in others is exactly why the table-level aggregate win ends up modest rather than dramatic.
theta_ex <- seq(-4, 4, length.out = 401)
p_lpe_ex <- plogis(a_ex * theta_ex + d_ex)^kappa_ex
p_sym_ex <- plogis(a_ex * theta_ex + d_ex)
ex_df <- tibble(theta = theta_ex, LPE = p_lpe_ex, `Symmetric (kappa=1)` = p_sym_ex) |>
pivot_longer(-theta, names_to = "curve", values_to = "p") |>
mutate(tooltip = paste0(curve, "<br>theta: ", round(theta,2), "<br>P: ", round(p,3)))
bins_ex <- empirical_bins_df |>
mutate(tooltip = paste0(
"Empirical bin<br>theta (bin mean): ", round(theta, 2),
"<br>Observed rate: ", round(obs_prop, 3),
"<br>n: ", n
))
p <- ggplot(ex_df, aes(x = theta, y = p, colour = curve, group = curve, text = tooltip)) +
geom_line(linewidth = 1) +
geom_vline(xintercept = b_ex, linetype = "dotted", colour = irw_grey) +
geom_point(
data = bins_ex, inherit.aes = FALSE,
aes(x = theta, y = obs_prop, size = n, text = tooltip),
colour = "#e6550d", alpha = 0.7
) +
scale_colour_manual(values = c("LPE" = irw_blue, "Symmetric (kappa=1)" = irw_grey), name = NULL) +
scale_size_continuous(range = c(1.5, 5), guide = "none") +
labs(x = "θ", y = "P(θ)")
ggplotly(p, tooltip = "text") |>
layout(legend = list(orientation = "h", y = -0.15))This item’s discrimination is unusually high (\(a\) = 7.5), which is exactly the regime where empirical theta-bins can look noisy bin-to-bin even when the underlying model fit is sound: a tiny shift in who lands in which bin, near a near-step-function response curve, swings the observed rate a lot. Read the smooth fitted curves as the main comparison here, and the empirical points as a rough check rather than a precise one for this particular item.
The LPE curve sits above the symmetric reference through most of the lower and middle trait range and only catches up near the top — exactly the qualitative signature of \(\kappa < 1\) (0.1 here) and positive \(\text{Л}_P\): a person one logit above and one logit below this item’s difficulty are, on average, more likely to endorse it than a symmetric model centered on the same difficulty would predict.
Binary items only. LPE/RH/AO extensions to polytomous items exist in principle but weren’t attempted here.
Fixed regularization. The \(N(0,1)\) MAP prior on each shape parameter is the paper’s own “moderate shrinkage” condition, applied uniformly; no per-table prior-sensitivity sweep was run at IRW scale (the paper does run one, in a different context).
RH is probit-linked; the others are logistic. RH is not exactly nested in the logistic 2PLM the way LPE and AO are. Scoring every model by held-out IMV on the response probability scale sidesteps the specific caution the paper raises about comparing BIC across link functions — IMV only needs comparable predicted probabilities, not a comparable likelihood parameterization — but RH’s probit link is still a real, separate modeling choice from the other two, worth keeping in mind when comparing its \(\kappa\) to LPE/AO’s.
Per-item shape estimates are noisy, and a few are extreme. A handful of items — typically ones that are answered almost unanimously one way — show implausibly large \(|\kappa|\) (up to roughly 26 for AO, 52 for LPE, and -7 to 6 for RH in this batch), consistent with the imperfect recovery documented above (weakest for LPE, moderate for AO, strongest for RH) rather than a broken fit, but not individually verified case by case. Treat extreme per-item \(\kappa\) values as thin-data artifacts unless checked directly — the worked example above shows this concretely: several items in that same table hit \(\text{Л}_P \approx -1\) at extreme \(\kappa\), alongside the one genuinely interesting item featured there.
The response-level holdout is generous but not huge. ~20% of each table’s observed cells, capped so every person keeps at least one retained response — enough to detect the null-truth calibration cleanly in the validation gate, but individual held-out IMV values for smaller tables (some under 2,500 respondents) are based on proportionally fewer held-out cells than the largest tables and should be read with that in mind.
Shape-parameter recovery is imperfect and uneven across models. See the discussion in Validation (c) above — RH recovers well (\(r = 0.96\)), AO moderately (\(r = 0.74\)), and LPE weakest (\(r = 0.51\)). Treat LPE’s per-item \(\kappa\) with the most caution, RH’s with the least.
A shape parameter may be structurally hard to identify at all, not just noisy to estimate. (Holland 1990)’s conservation law — worked out in detail in this site’s Dutch Identity vignette — shows that a unidimensional test of \(J\) items supports at most \(2J\) parameters as \(J \to \infty\), exactly the 2PL’s \((a, d)\) budget per item; the 3PL’s third parameter (guessing) exceeds that budget, and that vignette finds empirically that the 2PL → 3PL predictive gain shrinks toward zero as test length grows. LPE, RH, and AO are in the same structural position: each adds one third per-item parameter (\(\kappa\)) on top of an otherwise-2PL base, mechanically identical to how the 3PL adds guessing. That framing fits two things found here better than “these models just need more data”: the real-data IMV gains above sit at the IMV(2PL,3PL) benchmark rather than the substantially larger IMV(1PL,2PL) one, and the sample-size sweep in check (e) found LPE’s shape parameter doesn’t fail cleanly on its own — it drags its own difficulty (\(d\)) estimate down with it, exactly the kind of budget-overrun a third per-item parameter would produce if a unidimensional test’s real information ceiling is the two the 2PL already spends. This isn’t the same asymptotic Holland’s theorem uses (that result is about test length \(J \to \infty\) with \(N\) effectively unlimited; check (e) is about sample size \(N \to \infty\) with \(J\) fixed at 12 items) — so treat it as a motivating frame for why \(\kappa\) is hard to pin down, not as a confirmation of the conservation law itself.
Results were computed on July 21, 2026. To regenerate:
# 1. Re-run the validation gate (from the project root)
source("vignettes/asymmetric_irt_data/validation_gate.R")
# 2. Re-run the real-data batch
source("vignettes/asymmetric_irt_compute.R")
# 3. Re-fit the single worked example and extract its empirical theta bins
source("vignettes/asymmetric_irt_data/example_itemfit_compute.R")
# 4. Re-run the sample-size convergence sweep (validation gate check (e))
source("vignettes/asymmetric_irt_data/convergence_sim_compute.R")
# 5. Re-render this page
quarto::quarto_render("vignettes/asymmetric_irt.qmd")