The R and Python ecosystems use different default random-number generators (R’s Mersenne-Twister with inversion, NumPy’s PCG64), so a direct port produces different data from the same seed. That would undermine the guarantee that a design authored once can be reproduced anywhere.
On one platform, the same specification and seed always give the same data.
spec <- build_spec(list(
name = "demo", seed = 2024, design_kind = "between", n_subject = 200,
factor_name = "group", lev1 = "a", lev2 = "b",
intercept = 100, effect = 5, family = "gaussian",
resp_name = "score", sigma = 10))
identical(simulate_design(spec), simulate_design(spec))[1] TRUE
Across operating systems, the same holds on the terms set out above,
since R takes exp(), log() and
pow() from the platform’s maths library, as Python
does.
Changing the seed changes the draws, while the structure and the ground-truth parameters stay fixed.
spec2 <- spec; spec2$seed <- spec$seed + 1L
identical(simulate_design(spec)$score, simulate_design(spec2)$score)[1] FALSE
The Python package reads the same specification. The code below, which is not evaluated here, shows the Python you would run. Its output matches the R data to the last bit wherever the two share a maths library.
from pilotr import simulate, load_spec
# the same design.json the R package (and the app) use
data = simulate("design.json")The repository includes a parity check that compares the CSV files the R and Python demo scripts write for six of the worked example designs, all but the ordinal and Poisson ones. It reports the maximum absolute difference per design, reads values at the precision a written file carries and has a tolerance of 1e-6. The table below summarises it.
| design | rows | columns | max_abs_diff | categorical_mismatches |
|---|---|---|---|---|
| between | 64 | 3 | 0 | 0 |
| crossed | 1440 | 4 | 0 | 0 |
| continuous | 4000 | 7 | 4.885e-15 | 0 |
| nested | 4800 | 5 | 0 | 0 |
| beta | 200 | 3 | 0 | 0 |
| partial | 1440 | 4 | 0 | 0 |
The difference is exactly zero for five of the six designs. The
sixth, continuous (reading_time_continuous),
has a rounded response that agrees exactly. Its continuous predictors
are unrounded, and R’s CSV writer prints them to 15 significant digits.
Those predictors differ by about 5e-15. At that precision, the residual
cannot be told apart from the writer’s own rounding, and the tolerance
of 1e-6 is there to absorb it. The parity check can be rerun on any new
platform, so a design verified on a laptop can be confirmed on an HPC
node before a large run.
The bit-identical claim rests on a stricter check, the harness under
tools/parity/. It dumps every value at 17 significant
digits, which shows whether the values themselves differ. It accounts
for each differing cell in units in the last place against a recorded
allowance and must pass in continuous integration. On Windows, it finds
a handful of cells one unit in the last place apart, from
exp(), and no categorical mismatches. On Linux, where it
runs in continuous integration, every cell agrees. The uniform stream is
exact by construction. The harness checks the Gaussian cases against
recorded hashes in tools/parity/golden.json, relying on the
observed agreement of AS 241’s log() across platforms. The
cases are small enough that none has met one of the rare normal draws
that differ. The cases whose values pass through exp(),
log() or pow() on the way to the response have
no recorded hashes, because the maths library determines those last
bits. They must still pass the cross-language comparison on every
run.
solve_curve() fits a model to a curve that has already
been simulated, and it is held to a different standard from the
generator. Its fit calls exp() and the normal distribution
function at every iteration. A recorded hash would therefore be valid
only for the maths library that produced it, and the two engines cannot
be expected to agree to the last bit. They must instead agree far more
closely than any reading of the result could depend on. The script
tools/parity/solve_cross.py puts the same fixed curves
through both engines and compares the solved value, its interval, the
fitted coefficients and the text of a selection of error messages. Each
of those error messages is identical character for character. The
largest relative difference measured over those curves is about 5e-15,
against an allowance of 1e-9. On a solved sample size of 220 subjects,
that allowance comes to two ten-millionths of a subject.
The mixed models of the two power_mixed() functions come
from model_formula() and model_data(), which
the Python package ports from this one. The script
tools/parity/formula_cross.py gives the Python ports the
data R simulated for every worked example and parity case. Both packages
must write the same formula character for character and the same
contrast and product columns bit for bit. The fits that follow can
differ. lme4 and statsmodels are different
programs, this package estimates the random-effect correlations that the
Python one leaves out, and the two test with different statistics. The
Python power guide sets out where their results diverge.
power_report() writes a sample-size justification from a
result’s counts of significant replicates and of the replicates behind
each rate. It computes each Monte Carlo error with the functions both
packages use to compute these errors, which return the same doubles for
the same counts, and formats the errors in the same way. The script
tools/parity/report_cross.py renders a set of stored
records in both engines and requires the same text, byte for byte.
Because every implementation must follow the specification and the generator is shared, a design authored once in the app, in R or in Python can be reproduced everywhere. It can be preregistered as a small JSON file that anyone can run, scaled from a laptop to a cluster and handed between collaborators who work in different languages. Its data agree to the last bit wherever the machines share a maths library, outside the Beta case under R 4.6.0 or later.