When sample sizes are large, or when the question of interest is whether an effect is practically meaningful, power against a point null is often the wrong target. pilotr implements a precision-based design analysis against a region of practical equivalence (ROPE). This is a fast, frequentist analogue of the Bayesian approach that compares a highest-density interval with a ROPE.
As in the power vignette, the fixed-N analysis below uses a small
n_simsso that the vignette builds quickly. The sample-size sweep needs far more model fits, so its result was precomputed with exactly the code shown and included in the package. For planning a study, we recommendn_sims >= 200, the value the sweep uses.
Over Monte Carlo replicates, and for each focal fixed effect, pilotr records the 95% confidence interval and the expected interval width. It also records whether the interval falls entirely outside the ROPE, indicating a practically meaningful effect, or entirely inside it, indicating practical equivalence to zero. The interval is a Wald approximation, the estimate plus or minus 1.96 standard errors, chosen for speed and for comparability across replicates. Sweeping sample size then locates the minimum N at which a focal effect reaches a determinate decision with a target probability.
We reuse the crossed priming design, which represents a small priming
effect on log(RT − 200), the logarithm of the reaction time less its
shift. Every subject sees every item in both conditions. A
counterbalanced study, in which each subject sees each item in one
condition only, has half the observations and wider intervals, so it
needs more subjects than the sweep below finds. Plan such a study with
counterbalance = TRUE in build_spec(), which
needs specification version 0.4, or with the two-list encoding among the
worked
encodings of the specification, which earlier releases read too.
spec_c <- build_spec(list(
name = "priming", seed = 1, design_kind = "within", include_items = TRUE,
n_subject = 24, n_item = 18,
factor_name = "condition", lev1 = "related", lev2 = "unrelated",
intercept = 6, effect = 0.05,
subj_int_sd = 0.12, subj_slope_sd = 0.04, subj_corr = 0.2,
item_int_sd = 0.08, item_slope_sd = 0.02, item_corr = -0.1,
family = "shifted_lognormal", resp_name = "RT", sigma = 0.3, shift = 200))So that the analysis can run from the specification alone, pilotr auto-derives the analysis model, including the contrast columns, the response transform and the mixed-model formula.
.y ~ effect + (1 + effect | subject) + (1 + effect | item)
The companion model_data() adds the other two
derivations to a simulated data set: the numeric effect
contrast column the formula’s terms refer to, and the .y
response, here log(RT − 200) because the family is
shifted_lognormal with a shift of 200.
subject item condition RT .y effect
1 1 1 related 688.7993 6.191952 -0.5
2 1 1 unrelated 778.3731 6.360219 0.5
3 1 2 related 884.1392 6.528161 -0.5
4 1 2 unrelated 788.8995 6.378256 0.5
5 1 3 related 651.0253 6.111523 -0.5
6 1 3 unrelated 518.3355 5.763106 0.5
We declare the focal effect by its coefficient name and true value, and set a ROPE half-width. Here the effect is on the log scale, and we treat anything smaller than 0.02 as practically equivalent to zero.
param true mean_ci_width p_meaningful p_meaningful_mcse p_meaningful_lo
1 effect 0.05 0.09809118 0.28 0.08979978 0.1428385
p_meaningful_hi p_equivalent p_equivalent_mcse p_equivalent_lo
1 0.4757661 0 0 0
p_equivalent_hi n_attempted n_returned n_converged n_singular n_warning
1 0.1331923 25 25 8 17 0
fitter
1 lme4::lmer, Wald z
p_meaningful is the probability that the 95% confidence
interval falls entirely outside the ROPE, a determinate ‘meaningful’
decision. p_equivalent is the probability that it falls
entirely inside, a determinate ‘equivalent’ decision.
mean_ci_width is the expected precision.
The sweep has to be given the same ROPE. Leaving rope at
its default of 0.05 would put the upper bound of the region at the true
effect, and p_meaningful would then stay near 0.025 at
every N.
prc <- precision_curve(spec_c, focal = c(effect = 0.05),
subject_ns = c(15, 30, 60, 100, 140, 180, 220, 260),
rope = 0.02, n_sims = 200)
print(prc[, c("n_subject", "p_meaningful", "p_equivalent", "mean_ci_width",
"n_returned", "n_singular")], digits = 3) n_subject p_meaningful p_equivalent mean_ci_width n_returned n_singular
1 15 0.140 0 0.1215 200 156
2 30 0.300 0 0.0834 200 141
3 60 0.550 0 0.0605 200 115
4 100 0.680 0 0.0477 200 98
5 140 0.870 0 0.0407 200 81
6 180 0.885 0 0.0370 200 73
7 220 0.945 0 0.0339 200 45
8 260 0.945 0 0.0322 200 53
As N grows, the interval narrows and
p_meaningful rises. The probability of a determinate
decision is a more informative criterion than power against a point
null, because it asks whether the study can distinguish the effect from
a negligible one. Power asks only whether the effect can be
distinguished from zero.
p_equivalent stays at 0 throughout. The true effect of
0.05 lies outside the ROPE by construction, so no interval should ever
fall entirely inside it. The column is useful in designs where practical
equivalence is the hypothesis of interest.
Each probability is a proportion over the replicates that returned an
estimate. Their number, n_returned, is 200 at each size
here, so the Monte Carlo standard error of each is at most 0.035. The
figure below draws the Wilson interval of p_meaningful as a
band.
n_singular counts the fits whose variance estimates sat
on a boundary, 156 at 15 subjects and 53 at 260.
n_converged, which the table above omits, counts those with
neither a singular fit nor a warning. Singular fits are kept in the
proportions, since their fixed-effect estimates remain interpretable and
discarding them would bias the result. A large n_singular
means that the model is richer than the design can support at that
size.
Scanning the table for the smallest simulated N that meets a
target of ‘p_meaningful ≥ 0.90’ gives 220 subjects. That
answer uses only the two points either side of the target. It is a
property of where the grid points happen to fall, and it comes without
an interval. solve_curve() fits the decision probability
against the sample size and inverts the fit, so all eight points inform
the answer.
value lo hi dispersion
181.127671 161.378958 202.016032 2.287547
The fitted crossing is 181 subjects. With 200 replicates per point, this design needs somewhere between 161 and 203 subjects to reach a determinate ROPE decision nine times in ten. The 220 read by eye lies above that interval. Only the interval states the uncertainty of the answer, and a preregistration quoting a single figure would claim more than the simulation supports.
The fit is a binomial probit regression weighted by the replicates
behind each point, and the interval is the delta-method interval that
MASS::dose.p() computes for a fitted glm. The
dispersion is Pearson’s chi-square over its degrees of
freedom, the heterogeneity factor of classical probit analysis, floored
at 1. Here it is 2.29. A value above 1 means that the points scatter
about the fitted curve more than their replicate counts alone would
explain, and the interval is widened to match.
solve_curve() does not extrapolate. A curve that never
reaches the target within the sizes swept gives an error, and the
message reports the range the sweep did cover.
library(ggplot2)
# Each probability is a proportion over the replicates that returned an
# estimate, and the sweep carries its 95% Wilson interval, which the band draws.
# The two panels are on different scales, hence the free y axis.
panels <- c("P(CI outside the ROPE)", "Mean 95% CI width")
long <- rbind(
data.frame(n_subject = prc$n_subject, panel = panels[1], y = prc$p_meaningful,
lo = prc$p_meaningful_lo, hi = prc$p_meaningful_hi),
data.frame(n_subject = prc$n_subject, panel = panels[2],
y = prc$mean_ci_width,
lo = NA, hi = NA))
long$panel <- factor(long$panel, levels = panels)
# 0.90 is the target decision probability. 0.06 is the width at which a CI
# centred on the true effect just clears the ROPE, that is 2 * (0.05 - 0.02).
refs <- data.frame(panel = factor(panels, levels = panels), ref = c(0.90, 0.06))
# The solved sample size and its interval, drawn only on the decision panel,
# since the width panel is on a different scale and answers a different question.
solve_band <- data.frame(panel = factor(panels[1], levels = panels),
lo = solved$lo, hi = solved$hi, at = solved$value)
ggplot(long, aes(n_subject, y)) +
geom_hline(data = refs, aes(yintercept = ref), linetype = 2,
colour = "grey60") +
geom_rect(data = solve_band, inherit.aes = FALSE,
aes(xmin = lo, xmax = hi, ymin = -Inf, ymax = Inf),
fill = "grey60", alpha = .15) +
geom_vline(data = solve_band, aes(xintercept = at), linetype = 2,
colour = "grey60") +
geom_ribbon(aes(ymin = lo, ymax = hi), alpha = .15, fill = "#2C6FB0",
na.rm = TRUE) +
geom_line(colour = "#2C6FB0", linewidth = 0.8) +
geom_point(colour = "#2C6FB0", size = 2.2) +
facet_wrap(~ panel, scales = "free_y") +
labs(x = expression(italic(N) ~ "subjects"), y = NULL) +
theme_minimal(base_size = 12) +
# theme_minimal still paints a white plot.background over the transparent
# canvas, so both surfaces have to be cleared for the page colour to reach the
# figure.
theme(plot.background = element_rect(fill = NA, colour = NA),
panel.background = element_rect(fill = NA, colour = NA),
panel.grid = element_line(colour = "grey80"),
strip.background = element_rect(fill = NA, colour = NA))The two panels show the same sweep, one through the decision probability and the other through the interval width. On the left, the horizontal dashed line marks the 0.90 target, the vertical dashed line the sample size solved against it and the vertical band its interval. On the right, the dashed line marks a width of 0.06, at which a confidence interval centred on the true effect just clears a ROPE of 0.02. The interval has to keep its lower limit above the ROPE, so it must be narrower than 2 × (0.05 − 0.02). Width falls below that line well before the decision probability reaches 0.90, because an interval centred exactly on the true effect is the best case and sampling variation moves the centre about.
power_report() writes the sample-size justification that
a preregistration needs. It takes a precision_design()
result at the chosen size, the target_n() result for this
curve, which rounds the solved size up to whole subjects, and the ROPE
as rope. The justification gives each effect’s mean
interval width and the probabilities of the two decisions, each with its
Monte Carlo error, beside the solved size and its interval. The power vignette shows one for a power
analysis.
The power vignette covers simulation-based power and Type S/M errors. The getting-started vignette covers the core simulate-and-inspect loop.