Who Takes the Hard Shot? Skill and Selection in Sports Trial Data
Shooters, kickers and quarterbacks choose their attempts, so a raw success rate mixes skill with the difficulty of what was attempted. Using NBA shots, NFL kicks and passes, soccer shots and an online map-practice system, we fit a ladder of explanatory IRT models that add what is known about each attempt and ask how much the picture of who is good changes.
Published
October 23, 2026
This vignette — the choice of data sources, the compute script, the analysis design, and the writing below — was produced largely by Claude (Anthropic), working from a task specification and with human review. Check the compute script (sports_trials_compute.R) directly if you are relying on the numbers here.
Motivation
A free throw is an item in the textbook sense: every shooter faces the same 15 feet, with no defender. A game of HORSE is closer still to a test. One player calls a shot, everyone else has to make that same shot, and so the game builds a set of standardized items that every player attempts. In an actual game, every shot other than a free throw is different. A player chooses where to shoot from, and that choice depends on who the player is (a center lives near the rim), how good they are, and the game. So a shooting percentage mixes two things: how well someone shoots, and how hard the shots they chose were. This is but one example of a more general problem. Whenever respondents choose their own trials, or someone chooses for them, raw accuracy is a blend of skill and selection.
The IRW has a simulated version of this problem, nbashots_sim, which the course’s trials lesson uses. There the true skills are known, so one can check how well an explanatory model recovers them. In that simulation, better shooters take harder shots by design, and adjusting for the shot widens the spread of skill. This vignette asks what the same analysis looks like in real data, where nobody hands us the truth. We use five sources: NBA shots, NFL kicks and passes, soccer shots and, as a non-sport case, slepemapy.cz, an online system for practising geography on maps that picks each learner’s next question. In each one the item is a coarse binning of one feature of the attempt (a court zone, a distance band, a body part), and each attempt also comes with detailed information about where and when it happened.
The punchline is that selection does not run in a single direction. In some settings the harder attempts go to the better performers, and raw rates understate how different people are. In others, raw rates mostly reward people for having taken easy attempts, and they overstate it.
Data and methods
Data
ImportantThe NBA shot data are not in the IRW
The NBA.com shot files used here, as compiled in Samangy’s collection, come with no clear licence. Without one, the IRW cannot redistribute them. We analyse them because they are the real version of the problem in nbashots_sim. This page and its cached results hold only aggregates (SDs, correlations, IMVs, decile averages), with no shots and no players. Anyone who wants to rerun the NBA analysis must download the files from the source. The other four tables are in the IRW.
NBA shots: every field-goal attempt of the 2023-24 regular season, from NBA.com via Samangy’s collection(Samangy 2024). Not in the IRW (see the note above): the analysis reads the files when it runs and keeps only aggregates. Shooters with at least 100 attempts are retained. The item is the court zone; we also have the distance, the kind of shot (action: dunk, layup, pull-up jumper and so on), the game clock and home court.
NFL kicks: field goals and extra points, 2015–2025, from nflverse_kicks(nflverse 2025). 2015 is when the extra point moved back to 33 yards. Kickers with at least 20 attempts are retained. The item is the distance band (or extra point), plus exact distance, roof, wind, score and playoff. Kickers do not choose their kicks. A team usually has one kicker, and the coach decides whether to try a field goal at all, so a strong kicker gets to try kicks from distances where a coach with a weaker kicker would punt or go for it.
NFL passes: pass attempts, 2016–2025, from nflverse_passes(nflverse 2025). Passers with at least 100 throws are retained. The item is depth × direction (e.g., deep_left), plus air yards, down, distance to go, field position, shotgun and score.
Soccer shots: all shots in the 2017-18 seasons of five European leagues, the 2018 World Cup and Euro 2016, from wyscout_shots(Pappalardo et al. 2019). Players with at least 20 shots are retained. The item is foot, head, free kick or penalty, plus distance, the angle the goal mouth subtends, counter-attack, the clock and home side.
Map practice: answers from slepemapy.cz, a Czech website for practising geography (finding and naming places on maps), from geography(Papoušek et al. 2016). We use a random sample of 5,000 users with at least 20 answers. The item is the place asked (1,458 of them); we also have the number of options offered (2–6, or open) and the question type (find the place vs. name it). This is adaptive practice, not an adaptive test. The system updates its estimate of what the learner knows after each answer and picks the next question, including how many options to offer, to keep the learner succeeding most of the time without making the questions trivial. Its goal is learning rather than measurement, but the consequence is the same: learners it thinks are strong get harder questions (fewer options, or none). “I don’t know” answers are missing in the IRW table and are dropped.
Code
tibble(Domain =ifelse(names(domains) =="nba", "NBA shots (not in the IRW)", unname(domains)),Person =c(nba ="shooter", kicks ="kicker", passes ="passer",soccer ="shooter", geo ="learner")[names(domains)],Item =c(nba ="court zone (5)", kicks ="distance band, or extra point (5)",passes ="depth × direction (12)", soccer ="foot, head, free kick, penalty",geo ="place asked (random effect)")[names(domains)],Persons =sapply(names(domains), function(k) num(res[[k]]$desc$n_persons)),Attempts =sapply(names(domains), function(k) num(res[[k]]$desc$n_attempts)),`Success rate`=sapply(names(domains), function(k) f2(res[[k]]$desc$rate))) |> knitr::kable()
Table 1: The five tables as analysed. The NBA shots are not in the IRW (no clear licence); the other four are IRW tables.
Domain
Person
Item
Persons
Attempts
Success rate
NBA shots (not in the IRW)
shooter
court zone (5)
395
212,137
0.48
NFL kicks
kicker
distance band, or extra point (5)
91
25,652
0.90
NFL passes
passer
depth × direction (12)
136
185,214
0.64
Soccer shots
shooter
foot, head, free kick, penalty
779
34,411
0.12
Map practice
learner
place asked (random effect)
5,000
786,718
0.81
Models
For each domain we fit a ladder of Rasch-style logistic mixed models with lme4(Bates et al. 2015), with a random intercept for the person:
M0: person only. The person effect is, roughly, a shrunken success rate.
M1: adds the item, as fixed effects (a random effect for map practice, which has 1,458 places).
M2: adds a continuous description of the attempt’s difficulty: a spline of distance (NBA, kicks), of air yards (passes), of distance and angle (soccer), or the number of options and question type (map practice).
M3: adds the context of the attempt: game situation, conditions and, for the NBA, the kind of shot. Map practice has no M3, since the IRW table records no context.
From these we take four things. The person SD and the rank order of person effects say how the picture of who is good changes. A 5-fold out-of-sample InterModel Vigorish (IMV, Domingue et al. 2024) says how much each step improves prediction. A selection diagnostic says who takes the hard attempts. Split-half reliability says whether the person effect behaves like a trait.
The selection diagnostic works as follows. We fit the M2 terms without a person effect and use the result to give each attempt an expected log-odds of success, its “easiness.” We then average easiness within each person and correlate that average with the person effects from M0 and from the fullest model.
Code
step_bg <-c(M1 ="#deebf7", M2 ="#9ecae1", M3 ="#4292c6")step_fg <-c(M1 ="#08306b", M2 ="#08306b", M3 ="#ffffff")rhs <-function(f) trimws(strsplit(sub("^.*~", "", gsub("\\s+", " ", f)), " \\+ ")[[1]])# show spline bases as the spline they came fromtidy_terms <-function(tt) { lab <-c(d ="ns(distance, %d)", a ="ns(air yards, %d)", g ="ns(angle, %d)")for (b innames(lab)) { hit <-grepl(paste0("^", b, "[0-9]+$"), tt)if (any(hit)) tt <-c(tt[!hit], sprintf(lab[[b]], sum(hit))) } tt}chip <-function(tt, m) if (length(tt)) sprintf('<span style="background:%s;color:%s;padding:1px 5px;border-radius:3px;-webkit-box-decoration-break:clone;box-decoration-break:clone">%s</span>', step_bg[[m]], step_fg[[m]], paste(tt, collapse =" + ")) elseNULLlegend <-paste(sapply(names(step_bg), function(m) chip(m, m)), collapse =" ")rows <-lapply(names(domains), function(k) { fs <- res[[k]]$formulas prev <-rhs(fs[["M0"]]) parts <-character()for (m insetdiff(names(fs), "M0")) { cur <-rhs(fs[[m]]) add <-setdiff(cur, prev)if (m =="M1") add <-setdiff(add, "1") parts <-c(parts, chip(tidy_terms(add), m)) prev <- cur }tibble(Domain = domains[[k]],Model =paste0('<span style="font-family:SFMono-Regular,Menlo,Consolas,monospace;font-size:0.85em;line-height:1.9">resp ~ ',paste(parts, collapse =" + "), " + (1 | id)</span>"))})cat("<p style='margin-bottom:4px;font-size:0.9em'>Terms added at each step: ", legend, "</p>")
Table 2: The fullest model in each domain. The models are nested: M0 is resp ~ (1 | id), and each shaded block is the set of terms that step adds. ns(x, k) is a natural spline with k degrees of freedom.
Terms added at each step: M1M2M3
Domain
Model
NBA shots
resp ~ item + ns(distance, 6) + action + late + home + (1 | id)
Consider first the correlations in Table 3. In the NBA, raw skill (the M0 effect) correlates 0.78 with the easiness of a player’s shots. A shooting percentage is, to a large extent, a record of where someone shoots from. The simulated table does not build this in. There, with the same models, the raw correlation is 0.11, and true skill correlates -0.54 with easiness (last row of Table 3). The 0.78 is a feature of real shot selection, not of the method. Once the shot is accounted for, the relationship reverses: adjusted skill correlates -0.28 with easiness, so the better shooters, net of location, take the harder shots. The two measures also barely agree about who those better shooters are: raw and adjusted skill correlate only 0.32, against 0.84 in the simulation. Soccer looks similar on the raw side (0.66), but adjusted finishing skill is essentially unrelated to shot location (0.05).
Table 3: Correlations across persons. Easiness is the mean expected log-odds of a person’s attempts; raw skill is the M0 person effect; adjusted skill is the fullest model’s. The last row is the simulated nbashots_sim, fitted with the NBA models, where true skill is known.
Domain
Adjusted skill × easiness
Raw skill × easiness
Raw × adjusted skill
True skill × easiness
NBA shots
-0.28
0.78
0.32
NFL kicks
-0.24
-0.02
0.93
NFL passes
0.12
0.38
0.96
Soccer shots
0.05
0.66
0.74
Map practice
-0.39
0.07
0.85
NBA shots, simulated
-0.45
0.11
0.84
-0.54
Kicks and map practice look different. Raw rates are unrelated to easiness (-0.02 and 0.07), but adjusted skill is negatively related (-0.24 and -0.39). Coaches let their kickers try longer field goals when they trust them to make them, and the map-practice system gives the learners it rates as strong harder questions. In these two settings, the selection hides skill rather than inflating it.
Figure 1 shows the same comparisons by decile of adjusted skill. Map practice gives the cleanest picture: easiness falls steadily across all ten deciles, which is the practice system’s question selection doing what it was built to do. The NBA panel falls too, though less evenly. The kicks panel is noisy (there are only 91 kickers, about nine per decile) but tilts downward.
The U in the soccer panel is not about skill. Wyscout’s player file, which is not part of the IRW table, gives each player’s position, and the positions take very different shots (Table 4). Forwards shoot from closer in (16.9 m on average, against 21.2 m for midfielders), take twice the midfielders’ share of penalties, and score on 15% of their shots, against 9%. Their attempts are easier. They also shoot more often: 53 shots per player, against 40. More shots mean less shrinkage, so forwards’ adjusted estimates are more spread out (SD 0.11, against 0.08) and reach into both tails, while midfielders, who shoot less and from range, are pulled toward the middle. Forwards make up 60% of the bottom decile and 63% of the top one, against 27%–39% in deciles 5–7, where midfielders dominate, and the top and bottom deciles average 65 and 48 shots. A decile plot of shrunken estimates sorts people partly by how much data they have. The soccer panel is the clearest case of this, but every panel is subject to it. Passes show little selection either way (0.12).
Table 4: Soccer shots by the shooter’s position (Wyscout’s player file; players with at least 20 shots). Easiness is the mean expected log-odds of the position’s shots; adjusted skill is the M3 person effect.
Figure 1: Average expected log-odds of success of a person’s attempts (from the item and difficulty terms alone, no person effect), by decile of adjusted skill (the person effect from the fullest model). A downward slope means the more skilled take harder attempts. The y-axes differ across panels.
How much does the picture of who is good change?
The two kinds of selection have opposite consequences for the spread of skill (Figure 2). Where the hard attempts go to the better performers (kicks and map practice), adjusting for the attempt widens the person SD: from 0.18 to 0.27 logits for kickers and from 0.71 to 0.92 for learners. This is what happens in nbashots_sim. Where raw rates reward easy attempts (NBA and soccer), adjusting shrinks it: from 0.25 to 0.14 for NBA shooters and from 0.45 to 0.24 for soccer players. The real NBA data do not behave like the simulation: shot location by role dominates the “better shooters take harder shots” pattern.
The orderings move accordingly (Table 5). For kickers and passers the rank correlation with M0 stays above 0.9, so the adjustment changes the scale more than the order. For NBA shooters it falls to 0.39, and none of the raw top ten are still in the top ten under M3. Some care is needed here: M3 includes the kind of shot, and the ability to finish with a dunk is arguably part of shooting skill rather than a nuisance. M2 (zone and distance only) is the more conservative adjustment, and there the rank correlation is 0.56.
Code
p <-ggplot(ladder, aes(model, person_sd, fill = model,text =paste0(domain, ", ", model, "<br>Person SD: ", f2(person_sd),"<br>Spearman with M0: ", f2(spearman_m0)))) +geom_col(width =0.7) +facet_wrap(~factor(domain, unname(domains)), nrow =1) +scale_fill_manual(values = step_cols, guide ="none") +labs(x =NULL, y ="Person SD (logits)")show_plot(p)
Figure 2: Person SD (logits) as each model adds what is known about the attempt.
Code
ladder |>transmute(Domain = domain, Model = model, `Person SD`=f2(person_sd),`Spearman with M0`=f2(spearman_m0), `Top 10 kept`= top10_kept) |> knitr::kable()
Table 5: Person SD, rank correlation of person effects with M0, and how many of M0’s top ten remain in the top ten.
Domain
Model
Person SD
Spearman with M0
Top 10 kept
NBA shots
M0
0.25
1.00
10
NBA shots
M1
0.14
0.63
5
NBA shots
M2
0.14
0.56
3
NBA shots
M3
0.14
0.39
0
NFL kicks
M0
0.18
1.00
10
NFL kicks
M1
0.27
0.94
7
NFL kicks
M2
0.29
0.93
6
NFL kicks
M3
0.27
0.94
5
NFL passes
M0
0.15
1.00
10
NFL passes
M1
0.14
0.96
8
NFL passes
M2
0.14
0.95
7
NFL passes
M3
0.15
0.95
7
Soccer shots
M0
0.45
1.00
10
Soccer shots
M1
0.42
0.97
8
Soccer shots
M2
0.24
0.72
4
Soccer shots
M3
0.24
0.72
4
Map practice
M0
0.71
1.00
10
Map practice
M1
0.88
0.88
8
Map practice
M2
0.92
0.84
6
What does the attempt add to prediction?
The item does a lot of predictive work for the NBA (an IMV of 0.133 over shooters alone) and for passes (0.084), and much less for kicks (0.018), where most attempts are near-certain extra points. Continuous difficulty adds most where the item is coarsest relative to the variation it hides. For soccer, foot vs. head says little, and distance and angle add 0.024. For kicks, the bands already capture most of what distance knows (0.002). Context matters for NBA shots (0.037, mostly the kind of shot) and passes (0.008), and very little for kicks and soccer.
Passes illustrate a point that merits emphasis: an attempt feature can matter a great deal for prediction while barely changing who looks good. Quarterbacks all throw a broadly similar mix of short and deep passes, so the zone predicts completions well but does not reorder passers.
Code
p <-ggplot(imv_tab, aes(model, imv_vs_prev, fill = model,text =paste0(domain, ", ", model, "<br>IMV over previous step: ", f3(imv_vs_prev),"<br>IMV over M0: ", f3(imv_vs_m0)))) +geom_col(width =0.7) +geom_hline(yintercept =0, colour = irw_grey) +facet_wrap(~factor(domain, unname(domains)), nrow =1) +scale_fill_manual(values = step_cols, guide ="none") +labs(x =NULL, y ="IMV over previous step")show_plot(p)
Figure 3: Out-of-sample IMV (5-fold over attempts) of each step over the step before it.
Code
imv_tab |>transmute(Domain = domain, Model = model, `IMV vs M0`=f3(imv_vs_m0),`IMV vs previous`=f3(imv_vs_prev)) |> knitr::kable()
Table 6: Out-of-sample IMV of each model over M0 and over the step before it.
Domain
Model
IMV vs M0
IMV vs previous
NBA shots
M1
0.133
0.133
NBA shots
M2
0.152
0.017
NBA shots
M3
0.194
0.037
NFL kicks
M1
0.018
0.018
NFL kicks
M2
0.020
0.002
NFL kicks
M3
0.021
0.000
NFL passes
M1
0.084
0.084
NFL passes
M2
0.092
0.007
NFL passes
M3
0.101
0.008
Soccer shots
M1
0.009
0.009
Soccer shots
M2
0.033
0.024
Soccer shots
M3
0.034
0.000
Map practice
M1
0.018
0.018
Map practice
M2
0.020
0.002
Is skill a stable trait once selection is accounted for?
If adjusting for selection isolates something like a skill, person effects estimated from one half of a person’s attempts should predict those from the other half at least as well as the raw rates do. This is what happens where selection hides skill: split-half correlations rise from 0.25 to 0.33 for kickers and from 0.77 to 0.82 for learners. Where raw rates reward easy attempts, they fall: from 0.76 to 0.55 for NBA shooters and from 0.34 to 0.13 for soccer players. Much of what is stable about a raw shooting percentage is the player’s shot diet, which is stable because it is tied to role. Net of shot location, soccer finishing skill is barely reproducible from one half of a player’s shots to the other. This agrees with work on expected goals, which finds that finishing skill is hard to separate from noise at the shot volumes most players have (Davis and Robberechts 2024).
Figure 4: Correlation of person effects estimated separately on alternating attempts (odd vs even in time; a random half for map practice, whose IRW table has no timestamp). The fullest model is M3, except for map practice, where it is M2.
Discussion
Where does this leave us? In all five settings, the attempt is not an item in the usual sense, and the analyst has to decide how much of what is known about it to adjust for. The adjustment can matter a great deal, and its direction depends on who did the selecting and on what basis.
When someone else decides which attempts a performer gets, and gives the harder ones to the stronger performers (a coach deciding whether to try a long field goal, a practice system choosing the next question), raw success rates understate differences. Adjusting widens the spread and makes the person effect more reliable. This is the same logic as adaptive testing, and the explanatory model behaves as one would hope.
When performers settle into attempts that suit their role (bigs at the rim, strikers in the six-yard box), raw rates are largely a record of that role. Adjusting removes most of it, and what remains is smaller and noisier, sometimes much noisier.
This is not to say that the adjusted estimates are the “true” skill; the limitations below say why. Scoring the attempts is fixable; selecting them is not.
Limitations
The attempts not taken are never observed. Every model here conditions only on attempts that were made and recorded. A kick the coach declined, or a shot a player passed up, leaves no row. Selection models in economics (Heckman 1979) are one place to start, but we do not know of a satisfying answer for data like these.
Some attempt features are partly skill. The NBA M3 includes the kind of shot (dunk, layup, pull-up jumper). Being able to finish at the rim with a dunk is arguably part of shooting skill, so M3 may over-adjust; M2 (zone and distance only) is the more conservative comparison. Shot location raises the same question in a milder form: getting open close to the goal is a skill too.
The selection diagnostic is descriptive. Easiness comes from a model with no person effect, and both raw and adjusted skill are estimated from the same attempts, so the correlations in Table 3 are not clean estimates of how selection works. The decile plots also sort people partly by how many attempts they have, since estimates from fewer attempts are shrunk more (the soccer panel shows this most clearly).
Choices of window and sample. One NBA season (2023-24); NFL kicks from 2015 (the extra-point change) and passes from 2016; minimum attempt counts that differ by domain (100 for NBA shooters and passers, 20 elsewhere); and a random sample of 5,000 map-practice users. The IRW’s geography table has no timestamp, so its split-half check uses a random half rather than alternating attempts, and “I don’t know” answers (missing in the table) are dropped.
Two checks are not in the main results. The nbashots_sim row of Table 3 and the soccer players’ positions in Table 4 (from Wyscout’s player file, which is not part of the IRW table) come from sports_trials_checks.R, which saves them alongside the main cache.
These results were computed against approximately IRW v506 (the corpus as of October 1, 2026).
NoteHow this version was determined
The version was inferred from when the results were computed, not recorded during the run, so it may be off by a version or more. 19 IRW versions were released that day, so the exact one is ambiguous. To pin data to an exact version, use irw_use_version().
Results computed October 01, 2026. The four IRW tables are fetched with irw::irw_fetch(). The NBA shots are not in the IRW, because the files have no clear licence. The compute script downloads the 2023-24 file from Samangy’s collection when it runs (set NBA_DIR to use a local copy), and nothing from them is stored here except the aggregates shown on this page. To regenerate (about 40 minutes):
Rscript vignettes/sports_trials_compute.RRscript vignettes/sports_trials_checks.R # the simulation row of Table 3 and the soccer positionsquarto::quarto_render("vignettes/sports_trials.qmd")
References
Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using lme4.”Journal of Statistical Software 67 (1): 1–48. https://doi.org/10.18637/jss.v067.i01.
Domingue, Benjamin W., Klint Kanopka, Radhika Kapoor, et al. 2024. “The InterModel Vigorish as a Lens for Understanding (and Quantifying) the Value of Item Response Models for Dichotomously Coded Items.”Psychometrika 89 (3): 1034–54. https://doi.org/10.1007/s11336-024-09977-2.
Heckman, James J. 1979. “Sample Selection Bias as a Specification Error.”Econometrica 47 (1): 153–61. https://doi.org/10.2307/1912352.
Papoušek, Jan, Radek Pelánek, and Vı́t Stanislav. 2016. “Adaptive Geography Practice Data Set.”Journal of Learning Analytics 3 (2): 317–21. https://doi.org/10.18608/jla.2016.32.17.
Pappalardo, Luca, Paolo Cintia, Alessio Rossi, et al. 2019. “A Public Data Set of Spatio-Temporal Match Events in Soccer Competitions.”Scientific Data 6: 236. https://doi.org/10.1038/s41597-019-0247-7.