Reproducibility, parity and the materials datasheet

A stimulus set is usually described without being shared. The methods section says the words were matched on length and frequency, and the set itself, the criteria that produced it and the code that applied them are not recoverable from the paper. lexsync treats the set as an artefact in its own right. The selection and the trial order are deterministic, the R and Python engines produce the same artefacts from the same input, and each run emits a provenance record with the checksums needed to verify it.

This vignette covers the guarantee and its limits, the mechanisms that achieve it, the continuous-integration test that checks it and the two records a run writes.

library(lexsync)
schema <- yaml::read_yaml(
  system.file("extdata", "schema.yaml", package = "lexsync")
)
lex <- load_lexicon(
  system.file("extdata", "en_example.csv", package = "lexsync"),
  schema, language = "english"
)
pool <- build_pool(lex, list(length = c(3, 7), frequency = c(3.8, 7.0)))

design <- list(
  name = "vignette_repro", language = "english", n_per_condition = 15,
  conditions = list(
    list(name = "high", define_by = list(frequency = c(5.2, 7.0))),
    list(name = "low",  define_by = list(frequency = c(3.8, 4.4)))
  ),
  match_on = list("length", "n_density", "old20"),
  counterbalance = list(lists = 1)
)

What is guaranteed

Given the same lexicon and the same design, the R and Python engines produce byte-identical stimuli under the standardised_euclidean and joint matching methods, under continuous selection and under both pseudoword generators. The guarantee covers which items are chosen, how they are paired and the order they are presented in, so the generated experiments are byte-identical in full. For every worked design, the PsychoPy script, the OpenSesame file, the jsPsych page and the three CSV tables that the PsychoPy and OpenSesame experiments read are byte-identical across the engines. The only exceptions are the mahalanobis and optimal matching methods.

‘Byte-identical’ is meant literally, and the artefacts are compared as bytes. A comparison of parsed values would treat 1 and 1.0, FALSE and False, or 9e-4 and 0.0009 as equal although the files differ.

Means need the same care, since two engines that sum in different orders, one pairwise and one not, can report means that differ in the last decimal. Every sum in the package therefore goes through one compensated-summation algorithm written out in both engines, and the mean, variance and standard deviation are built on it. The two implementations of that algorithm agree because IEEE-754 requires addition and subtraction to be correctly rounded. The median sorts and takes the middle value, or the plain-double average of the two middle values when the count is even. It avoids stats::median(), which uses a long-double accumulator by way of mean().

Two files are outside the guarantee because each records the engine that produced it. The materials datasheet names its engine and that engine’s package versions, and the run log adds wall-clock timestamps. Their contents are compared field by field instead.

Within one engine, selection is a pure function of its input.

a <- match_stimuli(pool, design, schema)
b <- match_stimuli(pool, design, schema)
identical(a, b)
[1] TRUE

Repeated calls agree because nothing in the selection path consults a random number generator and because every operation that could depend on the ambient environment has been made independent of it. The Python engine agrees as well, although the two implementations share no code.

How byte-identical output is achieved

Cross-engine identity rests on a set of mechanisms, each of which removes one way in which the engines could otherwise differ.

There is no random number generator anywhere in the package. The obvious way to choose 15 items from 700 is to sample them, and lexsync never does. The anchor condition is chosen by an even spread across its sorted subpool, taking positions along the whole range, and every later item is determined by a distance. A seeded generator would not help, because R’s Mersenne-Twister stream and NumPy’s stream differ. The same seed gives different draws, so any sampling step would break parity. Trial order is also decided without a generator, by the shuffle described below.

anchor_pool <- pool[pool$frequency >= 5.2 & pool$frequency <= 7.0, ]
anchor_pool <- anchor_pool[
  order(anchor_pool$frequency, anchor_pool$word, method = "radix"),
]
idx <- unique(round(seq(1, nrow(anchor_pool), length.out = 15)))
idx
 [1]  1  7 13 19 25 31 37 44 50 56 62 68 74 80 86

The spread is seq() over the row positions, rounded to integers. Both engines compute the same positions from the same row count with the same expression, unique(round(seq(...))), and the rounding rule is written out in both. The unique() matters when the pool is smaller than the requested n and the rounding produces a repeated index.

Sorting is by byte order everywhere. R’s default string sort is locale-collated. A French locale, a C locale and a Windows locale can order the same words differently, so a locale-collated order cannot be relied on to match Python’s, which sorts by code point. Every sort on the selection path therefore passes method = "radix", which sorts by bytes.

words <- c("zebra", "Apple", "apple", "Zebra")
sort(words, method = "radix")
[1] "Apple" "Zebra" "apple" "zebra"

Uppercase sorts before lowercase because that is the byte order, and Python’s sorted() gives the same result. A locale-collated sort would typically interleave them.

Case folding is made independent of the locale for the same reason. load_lexicon() lower-cases through ICU at the root locale. It avoids tolower(), whose behaviour depends on the C library and the locale and differs from Python’s on Greek final sigma and the dotted Turkish capital I. The locale is kept out of two more places. Design, schema and registry files are read as UTF-8 whatever the session’s character type, and a file in another encoding is rejected with an error. The text of every hash key is made UTF-8 before it is digested, whatever encoding mark it carries. These keys are used for the trial order, list balancing, jitter and the tie-break between candidates, and a condition label such as café therefore ranks by the same digest in R under the C locale as in Python.

Frequencies of bigrams and of subsyllabic constituents are accumulated as integers, so the counts cannot depend on summation order.

Distances are rounded to nine decimal places before the tie-break. Two candidate words can be at exactly equal distance from an anchor, and the last bits of a square root can differ between two linear-algebra implementations. If the engines compared raw doubles, a difference in the fifteenth decimal would make one engine call a tie and the other not, and they would pick different words. Rounding before ranking puts both engines on the same side of every comparison that matters.

Rounding creates ties, and both engines break them by the same total order. Under tie_break: bytes, the rule of lexsync 0.1, candidates are ordered by distance, then by the word’s bytes, then by its integer id. The matcher’s comparison is reproduced here on three imaginary candidates.

distance <- c(0.5, 0.5, 0.2)
word     <- c("beta", "alpha", "gamma")
id       <- c(1L, 2L, 3L)
word[order(distance, word, id, method = "radix")]
[1] "gamma" "alpha" "beta" 

gamma wins on distance. alpha and beta are tied on distance to the ninth decimal, and the word’s bytes settle the tie in the same way in both engines. Since word is unique within a lexicon, the order is total and there is always exactly one winner.

Under tie_break: hash, the default, the word’s tie key takes the place of its bytes. The tie key is the hexadecimal SHA-256 of lexsync-tie-v1| followed by the word. Both engines compute the key from the same UTF-8 bytes and sort it the same way. The code below orders four candidates tied on distance by each rule.

tied <- c("bat", "cat", "dog", "eel")
key <- vapply(paste0("lexsync-tie-v1|", tied), digest::digest, character(1),
              algo = "sha256", serialize = FALSE, USE.NAMES = FALSE)
data.frame(bytes = tied[order(tied, method = "radix")],
           hash = tied[order(key, method = "radix")])
  bytes hash
1   bat  eel
2   cat  dog
3   dog  cat
4   eel  bat

Either rule is a total order that the two engines compute identically, so either keeps the guarantee. The setting matching.tie_break chooses between them. bytes reproduces the selections of lexsync 0.1. hash stops a condition decided by ties from filling with early-alphabet words, which the matching vignette measures. The salt is fixed and independent of the seed, which keys the trial order. A tie key built on the seed would change the items whenever a cohort’s order was re-randomised, and would let a researcher choose among equally matched sets by re-seeding.

Trial order: a seeded shuffle without a generator

Trial order cannot come from a seeded generator either, since R’s Mersenne-Twister and NumPy’s PCG64 give different permutations from one seed. Both engines therefore order the trials of a list by a keyed hash. Each row is ranked by the SHA-256 digest of seed|replicate|list|set|condition, a tuple that identifies the trial uniquely under either counterbalancing recipe. Distinct inputs to SHA-256 behave as independent uniform draws, so sorting the rows by their digests realises a seeded random permutation. Because a digest is a pure function of its input, the permutation is a pure function of the design. Both engines give the same bytes on any platform, there is no generator state to save or restore, and a different seed gives a different order. The implementation is .shuffle_deterministic() in R/counterbalancing.R, mirrored in the Python engine’s counterbalancing.py.

stim <- counterbalance(match_stimuli(pool, design, schema), design, schema)
head(stim[, c("trial", "condition", "word")], 4)
  trial condition    word
1     1       low arrange
2     2      high    with
3     3      high  coming
4     4      high   major
same <- counterbalance(match_stimuli(pool, design, schema), design, schema)
identical(stim$trial, same$trial)
[1] TRUE

The order is random in the sense a study needs, with no systematic relation between an item’s properties and its position. Counterbalancing does not touch the random number generator, since nothing in the package consults one. Calling it from a longer script therefore cannot disturb the stream your own code is drawing from.

The CSV input contract

Both engines read a lexicon, a norm table, a supplied pool or an item table by one rule. Each engine reads every cell as text and converts it itself, because readr and pandas guess types by different heuristics. Left to them, a padded category or a column of T and F would be read differently by the two engines.

Header names and cells are trimmed of spaces, tabs, carriage returns and line feeds. After trimming, an empty cell and NA, quoted or not, are missing, and every other word is a value. A column is numeric only when every value is a plain decimal number, with an optional sign, point and exponent and no thousands separators, so 1,000 is text. It is logical only when every value is TRUE or FALSE in upper, lower or title case, so T and F stay text. Any other column is text, a column of dates included. Every row counts towards the type. The columns a paradigm presents, and an item table’s item and condition, are always text. A header that names a column twice is rejected with an error, and an empty header cell is named by its position (...3).

The numbers themselves are parsed to the same double in both engines. Python’s float() returns the double nearest the decimal. readr’s parser and R’s as.numeric() can each miss it by one unit in the last place, readr’s on 50.768212762897100 and as.numeric() on 0.586135918132, for instance. This engine therefore computes every number of up to 15 significant digits, with a power of ten within 10^22, as one exact integer multiplied or divided by one exact power of ten. IEEE-754 rounds that single operation correctly, so the result is Python’s double. Every value in the bundled lexica is of that kind. Any other number is rounded in exact integer arithmetic, which gives Python’s double on every platform. as.numeric() is not used there, because it accumulates the digits in long double. Where that type is no wider than a double, as on Apple Silicon, an emulation of R’s parser missed on about one in ten of the values Python’s repr() writes for computed doubles. A negative zero is read as zero, since the two engines would otherwise write it as -0 and 0.

The YAML input contract

A design and the schema are read as YAML 1.1, by yaml::read_yaml() here and by PyYAML in Python, and the two readers resolve a few unquoted values differently. YAML 1.1 lists y, Y, n and N among its booleans, and PyYAML keeps them as text. lexsync reads them as text in both engines, so keys: [y, n] gives the response keys y and n. Booleans are written true and false. An unquoted yes, no, on or off is a boolean to both readers as well. A response key that YAML reads as a boolean is therefore rejected with an error that suggests quoting it. A norms entry names its join key join_on, since a bare on is read as true. counterbalance.optimise must be true or false, and any other value, such as n or 1, is rejected with an error.

Some known differences remain. PyYAML reads 1:30 as the base-60 number 90, 1_000 as 1000, 0b101 as 5 and 2026-10-13 as a date, and this engine reads all four as text. No worked design or schema in the repository holds such a value. Quote one, or write the number plainly, where a design must read alike in both engines. This engine also reads a sequence of one value, such as [true], as that value, while PyYAML keeps a list. optimise: [true] or name: [demo] therefore passes here and is rejected in Python, so write a single value without brackets.

The Unicode input contract

Word matching, deduplication and the length dimension operate on Unicode code points, and every text key is normalised to Unicode Normalization Form C (NFC). The words of a lexicon or a supplied pool and both sides of a norm-table or pair-member join are first trimmed and then lower-cased through ICU at the root locale. The item id, condition and presented fields of an item table, practice and filler tables included, and a supplied reference list are normalised as read. A precomposed character such as é and its decomposed counterpart, the base letter followed by a combining accent, are therefore one character, and a lexicon that mixes the two conventions holds each word once, in the first row the file gives it. Unnormalised, the two spellings would be two words of different lengths, which the matcher could place in opposite conditions of one matched set. When normalisation rewrites or merges anything, the run log says how many forms it rewrote and how many duplicates it removed, and the datasheet records the two counts as materials_source$input_normalisation. Input already in NFC, the form most corpora distribute, produces neither the log entry nor the datasheet field.

Unicode’s stability policy fixes the NFC form of every assigned character, so the two engines can disagree only on a character that one engine’s Unicode version assigns and the other’s lacks. R’s stringi uses the ICU it bundles or, on some Linux builds, the system’s ICU, which may implement another version, and Python’s unicodedata follows the interpreter (Unicode 15.1 in Python 3.13).

The syllable estimator and the pseudoword generators carry a narrower contract still. Both are orthographic approximations. The estimator is defined for Latin orthographies, and the generators for alphabetic scripts. count_syllables() decomposes a word (NFD), sets its combining marks aside and counts the maximal runs of the letters a, e, i, o, u, y, ae (U+00E6), o with stroke (U+00F8), dotless i (U+0131) and oe (U+0153), so a vowel counts whatever its accent, in Polish, Czech, Hungarian, Romanian, Turkish and Vietnamese as in English or Spanish. Hiatus (idea counts 2) and the Czech syllabic r and l (vlk, krk) stay undercounted. A word holding a character outside the code points U+0001-U+024F, U+02B0-U+036F, U+1E00-U+1EFF and U+2000-U+206F gets no estimate. Its n_syllables is NA in R and NaN in Python, and run_pipeline() stops with an error when a design refers to n_syllables while its pool holds such a word. The test is on code-point ranges, which keeps the two engines independent of their Unicode tables, and the same stability policy fixes the decompositions. The pseudoword generators substitute among the letters the reference words attest, in UTF-8 byte order, so with a Spanish lexicon they use its accented vowels and ñ. They stop with an error on a word in a logographic script such as Chinese. The worked Chinese design matches on length, n_density and old20 for these reasons.

File names

Every file a run writes is named after a stem built from the design’s name and language, which design_stem() computes in both engines. The stem keeps ASCII letters and digits only, so names that differ only outside ASCII would share it. Two designs with Chinese names, both in chinese, would write the same files, the second overwriting the first. Whenever the name or the language holds a non-ASCII character, the stem therefore ends in an underscore and the first eight hexadecimal digits of the SHA-256 of the NFC-normalised text name|language, such as chinese_14aadfa7. A name and language with no ASCII letter or digit get those eight digits alone. Normalising to NFC here keeps a precomposed and a decomposed spelling of one name on one stem. Names in plain ASCII are unaffected. run_all() also computes every design’s stem before it runs any, and stops with an error if two designs would write the same files, such as en-lexdec and en_lexdec.

The documented exceptions

Under mahalanobis and optimal, the two engines compute equivalent quantities but need not select the same items.

mahalanobis needs the inverse of the pool’s correlation matrix. A matrix inverse is computed by LAPACK, and R’s LAPACK is not necessarily the one NumPy is built against. The two may differ in blocking, in vectorisation and in the order they accumulate sums. The results agree to well within any tolerance you would care about but need not agree in their last bits. The rounding that keeps the Euclidean path identical does not help here, because the distance is a function of a matrix whose entries already differ below the rounding threshold, so the rounding can go either way.

optimal solves a linear assignment problem. Where several assignments have the same total cost, which one a solver returns is an implementation detail of the solver, and R’s clue::solve_LSAP() and SciPy’s implementation resolve those ties differently. Both return an optimal assignment. They need not return the same one.

Neither difference is a bug. Guaranteeing these methods would mean writing a bit-exact matrix inverse and a tie-canonical assignment solver in both languages, a large amount of numerical code to maintain for a modest gain. The chosen trade-off is to keep a byte-identical method as the default, document the exception and record it per run. The same trade-off shapes the roadmap. A covariance-aware distance might well be the better default, and it stays optional because it cannot carry the guarantee.

maha <- design
maha$matching <- list(method = "mahalanobis")
maha_stim <- match_stimuli(pool, maha, schema)
maha_ds <- build_datasheet(
  maha, schema, NULL, maha_stim,
  system.file("extdata", "en_example.csv", package = "lexsync"),
  list(stimuli = NA_character_), schema$seed
)
maha_ds$selection$cross_engine
[1] "approximate (platform linear algebra)"

Every datasheet carries this field, so a reader of the record does not have to know the rule. It reads byte-identical for standardised_euclidean and joint, and approximate (platform linear algebra) for mahalanobis and optimal. The generated methods paragraph adds a sentence stating the caveat in prose when it applies.

Almost everything on the path to a compared artefact is built from operations that IEEE-754 either mandates to be correctly rounded or makes exact, so the engines agree by construction rather than by measurement. That argument does not cover the Student t quantile and distribution behind the confidence interval and the TOST p-value. They come from stats::qt and stats::pt in R and from SciPy’s implementation in Python, two independent implementations that were measured to disagree by up to about 1e-12 over the range these reports use. The published values are rounded to three and four decimal places, which leaves margins of 5e-4 and 5e-5. The comparisons files therefore still come out byte-identical, and these values are not among the documented exceptions. The guarantee there rests on a margin of some seven orders of magnitude rather than on construction, and the reported precision should not be widened without checking that margin first.

normalise_lexicon() has the same kind of step. Zipf = log10(per million) + 3 calls each engine’s own log10(), which IEEE-754 does not require to be correctly rounded. A value whose scaled Zipf lies within one unit in the last place of a rounding boundary could therefore round to different three-place values in the two engines. The function counts the values that lie within 1e-9 of a boundary, and the count is expected to be 0. A run over a lexicon carrying freq_permillion records it as materials_source$zipf_derivation in its datasheet and in its run log. A registry entry’s derived file, a lexicon normalised once and checksummed, lets both engines read the same bytes and involves no such step.

The parity test

The parity gate in continuous integration runs the R pipeline over the worked designs, regenerates each with the Python engine and compares the two. The designs cover both engines’ matching methods, continuous selection, both pseudoword generators, resampling, priming and self-paced reading, three languages including a logographic script, and three reproductions of published designs.

The comparison is on identity columns. For a design that selects its own items, they are the word or target, the condition and the set index, and for one that reads an item table, they are the list, the set index, the condition and the item’s presented fields. A case adds the columns it exists to check: list for balanced lists, replicate for resampled sets, trial for practice and filler blocks, the realised prime_ms and iti_ms for jittered timing, the joined member and pair columns for the pair-keyed design, the looked-up dimensions for a supplied pool, prime_type for the masked-priming reproduction and answer for categorisation. set is included because, without it, the engines could select the same items, pair them differently and still pass. The test then checks every other shared column, so the trial position and the dimensions each engine computes at run time (n_density, old20, n_syllables, bigram_freq) must also agree.

Three details keep the gate from passing vacuously. The test skips when the R reference output is absent, since the Python package can be installed on its own. CI sets LEXSYNC_REQUIRE_PARITY=1 to turn those skips into failures, so a job meant to run both engines cannot pass by quietly skipping. The R references are regenerated and then checked with git diff --exit-code, so an R-engine regression fails the build. Without that step, the comparison would run against the committed snapshot, which an R-engine regression leaves unchanged. The R suite is also run once from the source tree with a check that nothing skipped, because the repository-coupled tests skip silently when run from an installed copy.

The worked designs have clean inputs: every cell is in NFC, none is padded and the norms cover every word. The gate therefore never runs the code that handles padded fields, missing values or unusual YAML, and a defect that both engines share would pass it. The twin-edge cases test that code. Each is a folder under parity/twin_edge/ at the repository root, holding a small design.yaml, its inputs, an optional schema.yaml and an expect.yaml that states the expected outcome. Input paths are relative to the folder. An artefacts case keeps the files the design must produce, byte for byte, in expected/, and may assert values in the datasheet. A refusal case gives a substring of the error the design must raise.

The R suite’s test-twin-edge.R and the Python suite’s test_twin_edge.py each check their own engine against every case. CI also runs tools/compare_engines.py --config parity/twin_edge, which compares the two engines on every artefacts case. Run without arguments, python tools/compare_engines.py repeats the whole-tree comparison on any machine. To add a case, create its folder with design.yaml and expect.yaml and write expected/ with python tools/twin_edge.py --update <case>. Read the files it writes to confirm they are right, then run the R suite, which checks that the R engine writes the same bytes. A fix for an input defect adds its case in the same commit, so that no commit leaves the gate failing.

The materials datasheet

Each run emits a datasheet in two forms, a JSON record for machines and a Markdown rendering for people. It exists because the reproducibility literature in linguistics keeps finding the same gap. Materials are described but rarely deposited in a form that permits reuse or verification (Bochynska et al., 2023), and undisclosed researcher degrees of freedom in data analysis inflate false positives (Roettger, 2019). A datasheet supplies the machine-readable metadata, licence and provenance that the FAIR principles ask to accompany data (Wilkinson et al., 2016).

out <- file.path(tempdir(), "lexsync_repro")
dir.create(out, showWarnings = FALSE)
report <- match_report(
  stim, c("frequency", "length", "n_density", "old20"), schema
)
stim_path <- file.path(out, "stimuli.csv")
write.csv(stim, stim_path, row.names = FALSE)

ds <- build_datasheet(
  design, schema, report, stim,
  source_path = system.file("extdata", "en_example.csv", package = "lexsync"),
  artifacts = list(stimuli = stim_path),
  seed = schema$seed,
  candidate_pool = lapply(design$conditions, function(cnd)
    list(
      condition = cnd$name,
      n_candidates = nrow(build_pool(pool, cnd$define_by))
    ))
)
names(ds)
 [1] "lexsync_datasheet_version" "design"                   
 [3] "materials_source"          "dimensions"               
 [5] "selection"                 "relational"               
 [7] "analysis"                  "equivalence"              
 [9] "realised_control"          "counterbalancing"         
[11] "presentation"              "resampling"               
[13] "items"                     "reproducibility"          
[15] "artifacts"                

The record answers the questions a sceptical reader would ask. Where the words came from, and whether this is the same file, is settled by materials_source, which carries the path and its SHA-256. What was applied is recorded in selection, covering the method, the matched dimensions and the tolerance windows as resolved by merging the design’s overrides onto the schema’s defaults. The rule that broke exact ties is there too, as tie_break, with its salt under hash.

What the selection achieved is in realised_control, as the effect sizes, intervals, variance ratios and TOST verdicts for every pair of conditions, each row naming its pair in condition and reference, giving the raw difference as mean_diff and the dimension’s role. What it could have chosen instead is in candidate_pool, which records how many items satisfied each condition before matching. A candidate missing a matched dimension satisfies its condition but cannot be matched. Such candidates are counted in incomplete_dropped, present only when a condition lost any, and the methods paragraph leaves them out of its count of eligible candidates.

How the items were assigned to lists is in counterbalancing. It gives the recipe that ran, the lists the stimuli table holds and the trials each list gives each condition, and it says whether a Latin-square rotation is complete. How many items there are is in items, whose n_per_condition_realised counts the items each condition holds, per replicate for resampled sets. The request the selector applied is design$n_per_condition, the selector’s default when the design gives none.

The record keeps each statistic as the comparisons report rounded it, to three places and the TOST p to four. The Markdown rendering and the methods paragraph show them at two or three places and the TOST p at four. The rounding is done from that record by lexsync’s own formatter, half away from zero, so both engines print the same digits. A d of 0.1649 is stored as 0.165 and is therefore shown as 0.17.

ds$materials_source$sha256
[1] "34a3c81cb559a42918864f5ecd2bcea93647cc4b0058dd03132ffe8723f4c310"
str(ds$selection$tolerance_k)
List of 4
 $ length   : num 2
 $ frequency: num 1
 $ n_density: num 2
 $ old20    : num 2
do.call(rbind, lapply(ds$selection$candidate_pool, as.data.frame))
  condition n_candidates
1      high           86
2       low          465

The candidate-pool counts show that the high condition here was drawn from fewer than a hundred eligible words and the low condition from several hundred. Choosing 15 items from a large candidate set invites the concern that a selection could be steered towards almost any result, whereas a condition with barely more candidates than items had little freedom to be steered. Recording the counts, alongside the fact that the selection is deterministic and blind to any outcome measure, lets a reader judge that risk.

The resolved windows include a frequency entry, although this design matches on the three orthographic dimensions only. The record reports the windows as resolved from the schema and the design, and a window is applied only to a dimension named in match_on.

str(ds$reproducibility)
List of 2
 $ seed    : int 2026
 $ versions:List of 10
  ..$ engine    : chr "R"
  ..$ lexsync   : chr "0.2.0"
  ..$ R         : chr "4.6.2"
  ..$ readr     : chr "2.2.0"
  ..$ stringdist: chr "0.9.17"
  ..$ jsonlite  : chr "2.0.0"
  ..$ digest    : chr "0.6.39"
  ..$ yaml      : chr "2.3.12"
  ..$ stringi   : chr "1.8.9"
  ..$ os        : chr "Linux x86_64"

The reproducibility block records the seed and the versions of the engine and its dependencies, since a bug fix in a dependency can change a selection and the record should say which versions produced this one.

The methods paragraph and the pre-registration template

methods_paragraph() turns the datasheet into prose. Generating the text removes a common source of error, a methods section transcribed by hand from numbers on a screen.

cat(strwrap(methods_paragraph(ds), width = 76), sep = "\n")
15 items per condition were selected from the English lexicon (wordfreq
(Speer, 2022) screened against SUBTLEX-US (Brysbaert & New, 2009), data CC
BY-SA 4.0; forms absent from SUBTLEX-US or beginning with a lower-case
letter in under 10% of their uses are left out, and N and OLD20 count
neighbours only among the forms the screen keeps; full corpus licence and
citation at
https://github.com/pablobernabeu/lexsync/blob/main/corpora/ATTRIBUTION.md)
and matched item by item on length, n_density, old20 using lexsync's
standardised_euclidean matcher. The largest difference on a matched
dimension was -0.133 neighbours in n_density (low against high; d = -0.03,
90% CI [-0.66, 0.59]). Two one-sided tests at a bound of 0.5 SD (alpha
0.05) supported equivalence on 0 of 3 matched comparisons; with 15 items
per condition no dimension that varies can pass this test, however well
matched. The smallest condition was selected from 86 eligible candidates,
and the selection was deterministic and blind to any outcome measure.
Materials were counterbalanced into 1 list(s) (a factorial split) and
generated for PsychoPy, OpenSesame and jsPsych. Responses were collected
from target onset until 2800 ms after it, and a response ended the trial.
The selection is deterministic and reproducible (lexsync 0.2.0), and the
seed (2026) fixes the trial order.

The paragraph is generated from the record, so it cannot disagree with the stimuli. It states the number of items each condition holds and the candidate-pool size, and it adds the cross-engine caveat when the design’s method warrants it. It describes the realised control before it tests it, since tests on a selected set concern a closed population (Sassenhagen & Alday, 2016). It gives the largest difference on any matched dimension, in that dimension’s unit, with d and its interval, and only then the two one-sided tests, with the number of matched comparisons they supported. It then names, largest first, up to five comparisons of a dimension the design did not control whose |d| exceeds the equivalence bound. The report here covers only the matched dimensions and frequency, which defines the conditions, so there is none to name.

This run shows why the order matters, and why the paragraph should be read and adapted before use. The demonstration matches only 15 items per condition. At a bound of 0.5 SD and an alpha of .05, no dimension that varies can pass the tests with fewer than 23 items per condition, however well it is matched. The paragraph says so, and a failed test is then not mistaken for a failed match. The matching vignette warns against overlooking this dependence on set size. A study, unlike this demonstration, would either match enough items or report the control by the differences the paragraph describes.

write_datasheet() writes both forms, and the Markdown rendering carries a pre-registration skeleton with its materials section already filled in.

paths <- write_datasheet(
  ds, file.path(out, "datasheet.json"), file.path(out, "datasheet.md")
)
md <- readLines(file.path(out, "datasheet.md"))
grep("^#", md, value = TRUE)
 [1] "# Materials datasheet -- vignette_repro (english)"
 [2] "## Provenance"                                    
 [3] "## Counterbalanced lists"                         
 [4] "## Selection transparency"                        
 [5] "## Realised control"                              
 [6] "## Suggested analysis"                            
 [7] "## Methods paragraph"                             
 [8] "## Pre-registration template"                     
 [9] "### Study information"                            
[10] "### Hypotheses"                                   
[11] "### Design"                                       
[12] "### Materials (from the lexsync datasheet)"       
[13] "### Sampling plan"                                
[14] "### Analysis plan"                                

The sections left blank are the hypotheses, the sampling plan and the inference criteria, which no tool can fill. The analysis plan is the exception and comes with a suggested model.

ds$analysis$suggested_model
[1] "response ~ condition + scale(length, scale = FALSE) + scale(n_density, scale = FALSE) + scale(old20, scale = FALSE) + (1 + condition | subject) + (1 | item)"

The model crosses random effects for subjects and items because items are a sample of the language rather than the population of interest. An analysis treating them as fixed generalises only to the words that were used (Clark, 1973; Yarkoni, 2022). The matched dimensions enter as centred covariates, computed from the stimuli file’s own columns, because matching leaves residual differences and Sassenhagen and Alday (2016) recommend adjusting for potential confounds. A structure is maximal only when every effect that varies within subjects or items has a random slope there (Barr et al., 2013). These covariates have none, so the accompanying note describes the slopes the model has. The note also advises simplifying a model that is singular or fails to converge (Matuschek et al., 2017). The suggested model is a starting point for a design of this shape.

The run log

Alongside the datasheet, which records what a run produced, the run log records what the run did.

log <- new_run_log(
  "vignette_repro",
  meta = list(seed = schema$seed, language = "english")
)
log <- log_step(
  log,
  sprintf("lexicon loaded: %d words", nrow(lex)),
  list(words = nrow(lex))
)
[lexsync] lexicon loaded: 3000 words
log <- log_step(log, sprintf("pool after filters: %d words", nrow(pool)))
[lexsync] pool after filters: 859 words
log <- log_artefact(log, stim_path, rows = nrow(stim))
[lexsync] wrote 'stimuli.csv'
log_md <- write_run_log(log, file.path(out, "run_log.md"),
                        file.path(out, "run_log.jsonl"))
cat(readLines(log_md), sep = "\n")
# lexsync run log: vignette_repro

- Engine: R 4.6.2
- Started: 2026-11-21 19:59:25.103683
- Finished: 2026-11-21 19:59:25.10586

## Run metadata

- seed: 2026
- language: english

## Steps

- **2026-11-21 19:59:25.104481**: lexicon loaded: 3000 words
    - words: 3000
- **2026-11-21 19:59:25.104884**: pool after filters: 859 words
- **2026-11-21 19:59:25.105294**: wrote 'stimuli.csv'
    - path: /tmp/RtmpPssDGo/lexsync_repro/stimuli.csv
    - rows: 30
    - md5: 06832f9e8e1d901adde134e464579401

The pipeline builds this log as it goes, recording each stage with its parameters, the equivalence verdict on every dimension, any balance warning and an entry per file written with its MD5. The experiments’ loop and text tables are among them, and the stimuli and report tables are logged with their row counts too. A design or schema key that lexsync does not read is noted there as well, since it changed nothing about the run.

A Markdown log and a JSON Lines log are emitted. The Markdown log alone carries the run metadata, such as the seed and the schema path, and the JSON Lines log holds the steps, one object per line, so that a run can be checked by a script. The MD5 here is a content fingerprint for detecting changes between runs, whereas the datasheet’s SHA-256 is the stronger one that belongs in a permanent record.

Reproducing a run

The pipeline does all of the above in one call. run_pipeline() on a design configuration is the reproducible unit, and it writes the stimuli, the reports, the three experiments, the datasheet and the log. Sharing the design file and the corpus is then enough for someone else to regenerate the materials and verify the checksums in either language.

Two habits keep a run reproducible. Pin the corpus, since a selection is only reproducible against a fixed lexicon. For this reason, the bundled corpora are a checksummed snapshot, and the wordfreq connector draws on a package frozen in 2024, whose data is a stable snapshot of usage through roughly 2021. Where cross-engine reproducibility matters, prefer standardised_euclidean, the default, or joint, since they are the methods that carry the guarantee.

A reproducible lexicon is not a screened one. The repository’s corpora/derived/en.csv, es.csv and zh.csv hold the most frequent forms wordfreq returns, so they include proper names, acronyms and other forms that are not words of the language, and N and OLD20 are computed over them. Their share differs between frequency bands, which can confound a frequency contrast. corpora/derived/en_v2.csv marks such forms in the English lexicon and counts N and OLD20 among the others only. The package’s English example holds only forms that en_v2.csv leaves unmarked, while its Spanish and Chinese examples are slices of es.csv and zh.csv and are not screened. What the derived lexica contain says what to check in a selection.

References

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

Bochynska, A., Keeble, L., Halfacre, C., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023). Reproducible research practices and transparency across linguistics. Glossa Psycholinguistics, 2(1). https://doi.org/10.5070/G6011239

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. https://doi.org/10.1016/S0022-5371(73)80014-3

Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001

Roettger, T. B. (2019). Researcher degrees of freedom in phonetic research. Laboratory Phonology, 10(1), Article 1. https://doi.org/10.5334/labphon.147

Sassenhagen, J., & Alday, P. M. (2016). A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests. Brain and Language, 162, 42–45. https://doi.org/10.1016/j.bandl.2016.08.001

Wilkinson, M. D., Dumontier, M., Aalbersberg, IJ. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18

Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017/S0140525X20001685