A stimulus set is usually described without being shared. The methods
section says the words were matched on length and frequency, and the set
itself, the criteria that produced it and the code that applied them are
not recoverable from the paper. lexsync treats the set as
an artefact in its own right. The selection and the trial order are
deterministic, the R and Python engines produce the same artefacts from
the same input, and each run emits a provenance record with the
checksums needed to verify it.
This vignette covers the guarantee and its limits, the mechanisms that achieve it, the continuous-integration test that checks it and the two records a run writes.
library(lexsync)
schema <- yaml::read_yaml(
system.file("extdata", "schema.yaml", package = "lexsync")
)
lex <- load_lexicon(
system.file("extdata", "en_example.csv", package = "lexsync"),
schema, language = "english"
)
pool <- build_pool(lex, list(length = c(3, 7), frequency = c(3.8, 7.0)))
design <- list(
name = "vignette_repro", language = "english", n_per_condition = 15,
conditions = list(
list(name = "high", define_by = list(frequency = c(5.2, 7.0))),
list(name = "low", define_by = list(frequency = c(3.8, 4.4)))
),
match_on = list("length", "n_density", "old20"),
counterbalance = list(lists = 1)
)Given the same lexicon and the same design, the R and Python engines
produce byte-identical stimuli under the
standardised_euclidean and joint matching
methods, under continuous selection and under both pseudoword
generators. The guarantee covers which items are chosen, how they are
paired and the order they are presented in, so the generated experiments
are byte-identical in full. For every worked design, the PsychoPy
script, the OpenSesame file, the jsPsych page and the three CSV tables
that the PsychoPy and OpenSesame experiments read are byte-identical
across the engines. The only exceptions are the mahalanobis
and optimal matching methods.
‘Byte-identical’ is meant literally, and the artefacts are compared
as bytes. A comparison of parsed values would treat 1 and
1.0, FALSE and False, or
9e-4 and 0.0009 as equal although the files
differ.
Means need the same care, since two engines that sum in different
orders, one pairwise and one not, can report means that differ in the
last decimal. Every sum in the package therefore goes through one
compensated-summation algorithm written out in both engines, and the
mean, variance and standard deviation are built on it. The two
implementations of that algorithm agree because IEEE-754 requires
addition and subtraction to be correctly rounded. The median sorts and
takes the middle value, or the plain-double average of the two middle
values when the count is even. It avoids stats::median(),
which uses a long-double accumulator by way of mean().
Two files are outside the guarantee because each records the engine that produced it. The materials datasheet names its engine and that engine’s package versions, and the run log adds wall-clock timestamps. Their contents are compared field by field instead.
Within one engine, selection is a pure function of its input.
[1] TRUE
Repeated calls agree because nothing in the selection path consults a random number generator and because every operation that could depend on the ambient environment has been made independent of it. The Python engine agrees as well, although the two implementations share no code.
Cross-engine identity rests on a set of mechanisms, each of which removes one way in which the engines could otherwise differ.
There is no random number generator anywhere in the package. The
obvious way to choose 15 items from 700 is to sample them, and
lexsync never does. The anchor condition is chosen by an
even spread across its sorted subpool, taking positions along the whole
range, and every later item is determined by a distance. A seeded
generator would not help, because R’s Mersenne-Twister stream and
NumPy’s stream differ. The same seed gives different draws, so any
sampling step would break parity. Trial order is also decided without a
generator, by the shuffle described below.
anchor_pool <- pool[pool$frequency >= 5.2 & pool$frequency <= 7.0, ]
anchor_pool <- anchor_pool[
order(anchor_pool$frequency, anchor_pool$word, method = "radix"),
]
idx <- unique(round(seq(1, nrow(anchor_pool), length.out = 15)))
idx [1] 1 7 13 19 25 31 37 44 50 56 62 68 74 80 86
The spread is seq() over the row positions, rounded to
integers. Both engines compute the same positions from the same row
count with the same expression, unique(round(seq(...))),
and the rounding rule is written out in both. The unique()
matters when the pool is smaller than the requested n and the rounding
produces a repeated index.
Sorting is by byte order everywhere. R’s default string sort is
locale-collated. A French locale, a C locale and a Windows locale can
order the same words differently, so a locale-collated order cannot be
relied on to match Python’s, which sorts by code point. Every sort on
the selection path therefore passes method = "radix", which
sorts by bytes.
[1] "Apple" "Zebra" "apple" "zebra"
Uppercase sorts before lowercase because that is the byte order, and
Python’s sorted() gives the same result. A locale-collated
sort would typically interleave them.
Case folding is made independent of the locale for the same reason.
load_lexicon() lower-cases through ICU at the root locale.
It avoids tolower(), whose behaviour depends on the C
library and the locale and differs from Python’s on Greek final sigma
and the dotted Turkish capital I. The locale is kept out of two more
places. Design, schema and registry files are read as UTF-8 whatever the
session’s character type, and a file in another encoding is rejected
with an error. The text of every hash key is made UTF-8 before it is
digested, whatever encoding mark it carries. These keys are used for the
trial order, list balancing, jitter and the tie-break between
candidates, and a condition label such as café therefore
ranks by the same digest in R under the C locale as in Python.
Frequencies of bigrams and of subsyllabic constituents are accumulated as integers, so the counts cannot depend on summation order.
Distances are rounded to nine decimal places before the tie-break. Two candidate words can be at exactly equal distance from an anchor, and the last bits of a square root can differ between two linear-algebra implementations. If the engines compared raw doubles, a difference in the fifteenth decimal would make one engine call a tie and the other not, and they would pick different words. Rounding before ranking puts both engines on the same side of every comparison that matters.
Rounding creates ties, and both engines break them by the same total
order. Under tie_break: bytes, the rule of lexsync 0.1,
candidates are ordered by distance, then by the word’s bytes, then by
its integer id. The matcher’s comparison is reproduced here
on three imaginary candidates.
distance <- c(0.5, 0.5, 0.2)
word <- c("beta", "alpha", "gamma")
id <- c(1L, 2L, 3L)
word[order(distance, word, id, method = "radix")][1] "gamma" "alpha" "beta"
gamma wins on distance. alpha and
beta are tied on distance to the ninth decimal, and the
word’s bytes settle the tie in the same way in both engines. Since
word is unique within a lexicon, the order is total and
there is always exactly one winner.
Under tie_break: hash, the default, the word’s tie key
takes the place of its bytes. The tie key is the hexadecimal SHA-256 of
lexsync-tie-v1| followed by the word. Both engines compute
the key from the same UTF-8 bytes and sort it the same way. The code
below orders four candidates tied on distance by each rule.
tied <- c("bat", "cat", "dog", "eel")
key <- vapply(paste0("lexsync-tie-v1|", tied), digest::digest, character(1),
algo = "sha256", serialize = FALSE, USE.NAMES = FALSE)
data.frame(bytes = tied[order(tied, method = "radix")],
hash = tied[order(key, method = "radix")]) bytes hash
1 bat eel
2 cat dog
3 dog cat
4 eel bat
Either rule is a total order that the two engines compute
identically, so either keeps the guarantee. The setting
matching.tie_break chooses between them. bytes
reproduces the selections of lexsync 0.1. hash stops a
condition decided by ties from filling with early-alphabet words, which
the matching vignette measures. The salt is fixed and independent of the
seed, which keys the trial order. A tie key built on the seed would
change the items whenever a cohort’s order was re-randomised, and would
let a researcher choose among equally matched sets by re-seeding.
Trial order cannot come from a seeded generator either, since R’s
Mersenne-Twister and NumPy’s PCG64 give different permutations from one
seed. Both engines therefore order the trials of a list by a keyed hash.
Each row is ranked by the SHA-256 digest of
seed|replicate|list|set|condition, a tuple that identifies
the trial uniquely under either counterbalancing recipe. Distinct inputs
to SHA-256 behave as independent uniform draws, so sorting the rows by
their digests realises a seeded random permutation. Because a digest is
a pure function of its input, the permutation is a pure function of the
design. Both engines give the same bytes on any platform, there is no
generator state to save or restore, and a different seed gives a
different order. The implementation is
.shuffle_deterministic() in
R/counterbalancing.R, mirrored in the Python engine’s
counterbalancing.py.
stim <- counterbalance(match_stimuli(pool, design, schema), design, schema)
head(stim[, c("trial", "condition", "word")], 4) trial condition word
1 1 low arrange
2 2 high with
3 3 high coming
4 4 high major
same <- counterbalance(match_stimuli(pool, design, schema), design, schema)
identical(stim$trial, same$trial)[1] TRUE
The order is random in the sense a study needs, with no systematic relation between an item’s properties and its position. Counterbalancing does not touch the random number generator, since nothing in the package consults one. Calling it from a longer script therefore cannot disturb the stream your own code is drawing from.
Both engines read a lexicon, a norm table, a supplied pool or an item
table by one rule. Each engine reads every cell as text and converts it
itself, because readr and pandas guess types by different heuristics.
Left to them, a padded category or a column of T and
F would be read differently by the two engines.
Header names and cells are trimmed of spaces, tabs, carriage returns
and line feeds. After trimming, an empty cell and NA,
quoted or not, are missing, and every other word is a value. A column is
numeric only when every value is a plain decimal number, with an
optional sign, point and exponent and no thousands separators, so
1,000 is text. It is logical only when every value is TRUE
or FALSE in upper, lower or title case, so T and
F stay text. Any other column is text, a column of dates
included. Every row counts towards the type. The columns a paradigm
presents, and an item table’s item and
condition, are always text. A header that names a column
twice is rejected with an error, and an empty header cell is named by
its position (...3).
The numbers themselves are parsed to the same double in both engines.
Python’s float() returns the double nearest the decimal.
readr’s parser and R’s as.numeric() can each miss it by one
unit in the last place, readr’s on 50.768212762897100 and
as.numeric() on 0.586135918132, for instance.
This engine therefore computes every number of up to 15 significant
digits, with a power of ten within 10^22, as one exact integer
multiplied or divided by one exact power of ten. IEEE-754 rounds that
single operation correctly, so the result is Python’s double. Every
value in the bundled lexica is of that kind. Any other number is rounded
in exact integer arithmetic, which gives Python’s double on every
platform. as.numeric() is not used there, because it
accumulates the digits in long double. Where that type is
no wider than a double, as on Apple Silicon, an emulation of R’s parser
missed on about one in ten of the values Python’s repr()
writes for computed doubles. A negative zero is read as zero, since the
two engines would otherwise write it as -0 and
0.
A design and the schema are read as YAML 1.1, by
yaml::read_yaml() here and by PyYAML in Python, and the two
readers resolve a few unquoted values differently. YAML 1.1 lists
y, Y, n and N among
its booleans, and PyYAML keeps them as text. lexsync reads them as text
in both engines, so keys: [y, n] gives the response keys y
and n. Booleans are written true and false. An
unquoted yes, no, on or
off is a boolean to both readers as well. A response key
that YAML reads as a boolean is therefore rejected with an error that
suggests quoting it. A norms entry names its join key
join_on, since a bare on is read as true.
counterbalance.optimise must be true or
false, and any other value, such as n or
1, is rejected with an error.
Some known differences remain. PyYAML reads 1:30 as the
base-60 number 90, 1_000 as 1000, 0b101 as 5
and 2026-10-13 as a date, and this engine reads all four as
text. No worked design or schema in the repository holds such a value.
Quote one, or write the number plainly, where a design must read alike
in both engines. This engine also reads a sequence of one value, such as
[true], as that value, while PyYAML keeps a list.
optimise: [true] or name: [demo] therefore
passes here and is rejected in Python, so write a single value without
brackets.
Word matching, deduplication and the length dimension
operate on Unicode code points, and every text key is normalised to
Unicode Normalization Form C (NFC). The words of a lexicon or a supplied
pool and both sides of a norm-table or pair-member join are first
trimmed and then lower-cased through ICU at the root locale. The item
id, condition and presented fields of an item table, practice and filler
tables included, and a supplied reference list are normalised as read. A
precomposed character such as é and its decomposed
counterpart, the base letter followed by a combining accent, are
therefore one character, and a lexicon that mixes the two conventions
holds each word once, in the first row the file gives it. Unnormalised,
the two spellings would be two words of different lengths, which the
matcher could place in opposite conditions of one matched set. When
normalisation rewrites or merges anything, the run log says how many
forms it rewrote and how many duplicates it removed, and the datasheet
records the two counts as
materials_source$input_normalisation. Input already in NFC,
the form most corpora distribute, produces neither the log entry nor the
datasheet field.
Unicode’s stability policy fixes the NFC form of every assigned
character, so the two engines can disagree only on a character that one
engine’s Unicode version assigns and the other’s lacks. R’s stringi uses
the ICU it bundles or, on some Linux builds, the system’s ICU, which may
implement another version, and Python’s unicodedata follows
the interpreter (Unicode 15.1 in Python 3.13).
The syllable estimator and the pseudoword generators carry a narrower
contract still. Both are orthographic approximations. The estimator is
defined for Latin orthographies, and the generators for alphabetic
scripts. count_syllables() decomposes a word (NFD), sets
its combining marks aside and counts the maximal runs of the letters a,
e, i, o, u, y, ae (U+00E6), o with stroke (U+00F8), dotless i (U+0131)
and oe (U+0153), so a vowel counts whatever its accent, in Polish,
Czech, Hungarian, Romanian, Turkish and Vietnamese as in English or
Spanish. Hiatus (idea counts 2) and the Czech syllabic r
and l (vlk, krk) stay undercounted. A word
holding a character outside the code points U+0001-U+024F,
U+02B0-U+036F, U+1E00-U+1EFF and U+2000-U+206F gets no estimate. Its
n_syllables is NA in R and NaN in Python, and
run_pipeline() stops with an error when a design refers to
n_syllables while its pool holds such a word. The test is
on code-point ranges, which keeps the two engines independent of their
Unicode tables, and the same stability policy fixes the decompositions.
The pseudoword generators substitute among the letters the reference
words attest, in UTF-8 byte order, so with a Spanish lexicon they use
its accented vowels and ñ. They stop with an error on a
word in a logographic script such as Chinese. The worked Chinese design
matches on length, n_density and
old20 for these reasons.
Every file a run writes is named after a stem built from the design’s
name and language, which
design_stem() computes in both engines. The stem keeps
ASCII letters and digits only, so names that differ only outside ASCII
would share it. Two designs with Chinese names, both in
chinese, would write the same files, the second overwriting
the first. Whenever the name or the language holds a non-ASCII
character, the stem therefore ends in an underscore and the first eight
hexadecimal digits of the SHA-256 of the NFC-normalised text
name|language, such as chinese_14aadfa7. A
name and language with no ASCII letter or digit get those eight digits
alone. Normalising to NFC here keeps a precomposed and a decomposed
spelling of one name on one stem. Names in plain ASCII are unaffected.
run_all() also computes every design’s stem before it runs
any, and stops with an error if two designs would write the same files,
such as en-lexdec and en_lexdec.
Under mahalanobis and optimal, the two
engines compute equivalent quantities but need not select the same
items.
mahalanobis needs the inverse of the pool’s correlation
matrix. A matrix inverse is computed by LAPACK, and R’s LAPACK is not
necessarily the one NumPy is built against. The two may differ in
blocking, in vectorisation and in the order they accumulate sums. The
results agree to well within any tolerance you would care about but need
not agree in their last bits. The rounding that keeps the Euclidean path
identical does not help here, because the distance is a function of a
matrix whose entries already differ below the rounding threshold, so the
rounding can go either way.
optimal solves a linear assignment problem. Where
several assignments have the same total cost, which one a solver returns
is an implementation detail of the solver, and R’s
clue::solve_LSAP() and SciPy’s implementation resolve those
ties differently. Both return an optimal assignment. They need not
return the same one.
Neither difference is a bug. Guaranteeing these methods would mean writing a bit-exact matrix inverse and a tie-canonical assignment solver in both languages, a large amount of numerical code to maintain for a modest gain. The chosen trade-off is to keep a byte-identical method as the default, document the exception and record it per run. The same trade-off shapes the roadmap. A covariance-aware distance might well be the better default, and it stays optional because it cannot carry the guarantee.
maha <- design
maha$matching <- list(method = "mahalanobis")
maha_stim <- match_stimuli(pool, maha, schema)
maha_ds <- build_datasheet(
maha, schema, NULL, maha_stim,
system.file("extdata", "en_example.csv", package = "lexsync"),
list(stimuli = NA_character_), schema$seed
)
maha_ds$selection$cross_engine[1] "approximate (platform linear algebra)"
Every datasheet carries this field, so a reader of the record does
not have to know the rule. It reads byte-identical for
standardised_euclidean and joint, and
approximate (platform linear algebra) for
mahalanobis and optimal. The generated methods
paragraph adds a sentence stating the caveat in prose when it
applies.
Almost everything on the path to a compared artefact is built from
operations that IEEE-754 either mandates to be correctly rounded or
makes exact, so the engines agree by construction rather than by
measurement. That argument does not cover the Student t quantile and
distribution behind the confidence interval and the TOST
p-value. They come from stats::qt and
stats::pt in R and from SciPy’s implementation in Python,
two independent implementations that were measured to disagree by up to
about 1e-12 over the range these reports use. The published values are
rounded to three and four decimal places, which leaves margins of 5e-4
and 5e-5. The comparisons files therefore still come out byte-identical,
and these values are not among the documented exceptions. The guarantee
there rests on a margin of some seven orders of magnitude rather than on
construction, and the reported precision should not be widened without
checking that margin first.
normalise_lexicon() has the same kind of step. Zipf =
log10(per million) + 3 calls each engine’s own log10(),
which IEEE-754 does not require to be correctly rounded. A value whose
scaled Zipf lies within one unit in the last place of a rounding
boundary could therefore round to different three-place values in the
two engines. The function counts the values that lie within 1e-9 of a
boundary, and the count is expected to be 0. A run over a lexicon
carrying freq_permillion records it as
materials_source$zipf_derivation in its datasheet and in
its run log. A registry entry’s derived file, a lexicon
normalised once and checksummed, lets both engines read the same bytes
and involves no such step.
The parity gate in continuous integration runs the R pipeline over the worked designs, regenerates each with the Python engine and compares the two. The designs cover both engines’ matching methods, continuous selection, both pseudoword generators, resampling, priming and self-paced reading, three languages including a logographic script, and three reproductions of published designs.
The comparison is on identity columns. For a design that selects its
own items, they are the word or target, the condition and the
set index, and for one that reads an item table, they are
the list, the set index, the condition and the item’s
presented fields. A case adds the columns it exists to check:
list for balanced lists, replicate for
resampled sets, trial for practice and filler blocks, the
realised prime_ms and iti_ms for jittered
timing, the joined member and pair columns for the pair-keyed design,
the looked-up dimensions for a supplied pool, prime_type
for the masked-priming reproduction and answer for
categorisation. set is included because, without it, the
engines could select the same items, pair them differently and still
pass. The test then checks every other shared column, so the
trial position and the dimensions each engine computes at
run time (n_density, old20,
n_syllables, bigram_freq) must also agree.
Three details keep the gate from passing vacuously. The test skips
when the R reference output is absent, since the Python package can be
installed on its own. CI sets LEXSYNC_REQUIRE_PARITY=1 to
turn those skips into failures, so a job meant to run both engines
cannot pass by quietly skipping. The R references are regenerated and
then checked with git diff --exit-code, so an R-engine
regression fails the build. Without that step, the comparison would run
against the committed snapshot, which an R-engine regression leaves
unchanged. The R suite is also run once from the source tree with a
check that nothing skipped, because the repository-coupled tests skip
silently when run from an installed copy.
The worked designs have clean inputs: every cell is in NFC, none is
padded and the norms cover every word. The gate therefore never runs the
code that handles padded fields, missing values or unusual YAML, and a
defect that both engines share would pass it. The twin-edge cases test
that code. Each is a folder under parity/twin_edge/ at the
repository root, holding a small design.yaml, its inputs,
an optional schema.yaml and an expect.yaml
that states the expected outcome. Input paths are relative to the
folder. An artefacts case keeps the files the design must produce, byte
for byte, in expected/, and may assert values in the
datasheet. A refusal case gives a substring of the error the design must
raise.
The R suite’s test-twin-edge.R and the Python suite’s
test_twin_edge.py each check their own engine against every
case. CI also runs
tools/compare_engines.py --config parity/twin_edge, which
compares the two engines on every artefacts case. Run without arguments,
python tools/compare_engines.py repeats the whole-tree
comparison on any machine. To add a case, create its folder with
design.yaml and expect.yaml and write
expected/ with
python tools/twin_edge.py --update <case>. Read the
files it writes to confirm they are right, then run the R suite, which
checks that the R engine writes the same bytes. A fix for an input
defect adds its case in the same commit, so that no commit leaves the
gate failing.
Each run emits a datasheet in two forms, a JSON record for machines and a Markdown rendering for people. It exists because the reproducibility literature in linguistics keeps finding the same gap. Materials are described but rarely deposited in a form that permits reuse or verification (Bochynska et al., 2023), and undisclosed researcher degrees of freedom in data analysis inflate false positives (Roettger, 2019). A datasheet supplies the machine-readable metadata, licence and provenance that the FAIR principles ask to accompany data (Wilkinson et al., 2016).
out <- file.path(tempdir(), "lexsync_repro")
dir.create(out, showWarnings = FALSE)
report <- match_report(
stim, c("frequency", "length", "n_density", "old20"), schema
)
stim_path <- file.path(out, "stimuli.csv")
write.csv(stim, stim_path, row.names = FALSE)
ds <- build_datasheet(
design, schema, report, stim,
source_path = system.file("extdata", "en_example.csv", package = "lexsync"),
artifacts = list(stimuli = stim_path),
seed = schema$seed,
candidate_pool = lapply(design$conditions, function(cnd)
list(
condition = cnd$name,
n_candidates = nrow(build_pool(pool, cnd$define_by))
))
)
names(ds) [1] "lexsync_datasheet_version" "design"
[3] "materials_source" "dimensions"
[5] "selection" "relational"
[7] "analysis" "equivalence"
[9] "realised_control" "counterbalancing"
[11] "presentation" "resampling"
[13] "items" "reproducibility"
[15] "artifacts"
The record answers the questions a sceptical reader would ask. Where
the words came from, and whether this is the same file, is settled by
materials_source, which carries the path and its SHA-256.
What was applied is recorded in selection, covering the
method, the matched dimensions and the tolerance windows as resolved by
merging the design’s overrides onto the schema’s defaults. The rule that
broke exact ties is there too, as tie_break, with its salt
under hash.
What the selection achieved is in realised_control, as
the effect sizes, intervals, variance ratios and TOST verdicts for every
pair of conditions, each row naming its pair in condition
and reference, giving the raw difference as
mean_diff and the dimension’s role. What it could have
chosen instead is in candidate_pool, which records how many
items satisfied each condition before matching. A candidate missing a
matched dimension satisfies its condition but cannot be matched. Such
candidates are counted in incomplete_dropped, present only
when a condition lost any, and the methods paragraph leaves them out of
its count of eligible candidates.
How the items were assigned to lists is in
counterbalancing. It gives the recipe that ran, the lists
the stimuli table holds and the trials each list gives each condition,
and it says whether a Latin-square rotation is complete. How many items
there are is in items, whose
n_per_condition_realised counts the items each condition
holds, per replicate for resampled sets. The request the selector
applied is design$n_per_condition, the selector’s default
when the design gives none.
The record keeps each statistic as the comparisons report rounded it, to three places and the TOST p to four. The Markdown rendering and the methods paragraph show them at two or three places and the TOST p at four. The rounding is done from that record by lexsync’s own formatter, half away from zero, so both engines print the same digits. A d of 0.1649 is stored as 0.165 and is therefore shown as 0.17.
[1] "34a3c81cb559a42918864f5ecd2bcea93647cc4b0058dd03132ffe8723f4c310"
List of 4
$ length : num 2
$ frequency: num 1
$ n_density: num 2
$ old20 : num 2
condition n_candidates
1 high 86
2 low 465
The candidate-pool counts show that the high condition here was drawn from fewer than a hundred eligible words and the low condition from several hundred. Choosing 15 items from a large candidate set invites the concern that a selection could be steered towards almost any result, whereas a condition with barely more candidates than items had little freedom to be steered. Recording the counts, alongside the fact that the selection is deterministic and blind to any outcome measure, lets a reader judge that risk.
The resolved windows include a frequency entry, although
this design matches on the three orthographic dimensions only. The
record reports the windows as resolved from the schema and the design,
and a window is applied only to a dimension named in
match_on.
List of 2
$ seed : int 2026
$ versions:List of 10
..$ engine : chr "R"
..$ lexsync : chr "0.2.0"
..$ R : chr "4.6.2"
..$ readr : chr "2.2.0"
..$ stringdist: chr "0.9.17"
..$ jsonlite : chr "2.0.0"
..$ digest : chr "0.6.39"
..$ yaml : chr "2.3.12"
..$ stringi : chr "1.8.9"
..$ os : chr "Linux x86_64"
The reproducibility block records the seed and the versions of the engine and its dependencies, since a bug fix in a dependency can change a selection and the record should say which versions produced this one.
methods_paragraph() turns the datasheet into prose.
Generating the text removes a common source of error, a methods section
transcribed by hand from numbers on a screen.
15 items per condition were selected from the English lexicon (wordfreq
(Speer, 2022) screened against SUBTLEX-US (Brysbaert & New, 2009), data CC
BY-SA 4.0; forms absent from SUBTLEX-US or beginning with a lower-case
letter in under 10% of their uses are left out, and N and OLD20 count
neighbours only among the forms the screen keeps; full corpus licence and
citation at
https://github.com/pablobernabeu/lexsync/blob/main/corpora/ATTRIBUTION.md)
and matched item by item on length, n_density, old20 using lexsync's
standardised_euclidean matcher. The largest difference on a matched
dimension was -0.133 neighbours in n_density (low against high; d = -0.03,
90% CI [-0.66, 0.59]). Two one-sided tests at a bound of 0.5 SD (alpha
0.05) supported equivalence on 0 of 3 matched comparisons; with 15 items
per condition no dimension that varies can pass this test, however well
matched. The smallest condition was selected from 86 eligible candidates,
and the selection was deterministic and blind to any outcome measure.
Materials were counterbalanced into 1 list(s) (a factorial split) and
generated for PsychoPy, OpenSesame and jsPsych. Responses were collected
from target onset until 2800 ms after it, and a response ended the trial.
The selection is deterministic and reproducible (lexsync 0.2.0), and the
seed (2026) fixes the trial order.
The paragraph is generated from the record, so it cannot disagree with the stimuli. It states the number of items each condition holds and the candidate-pool size, and it adds the cross-engine caveat when the design’s method warrants it. It describes the realised control before it tests it, since tests on a selected set concern a closed population (Sassenhagen & Alday, 2016). It gives the largest difference on any matched dimension, in that dimension’s unit, with d and its interval, and only then the two one-sided tests, with the number of matched comparisons they supported. It then names, largest first, up to five comparisons of a dimension the design did not control whose |d| exceeds the equivalence bound. The report here covers only the matched dimensions and frequency, which defines the conditions, so there is none to name.
This run shows why the order matters, and why the paragraph should be read and adapted before use. The demonstration matches only 15 items per condition. At a bound of 0.5 SD and an alpha of .05, no dimension that varies can pass the tests with fewer than 23 items per condition, however well it is matched. The paragraph says so, and a failed test is then not mistaken for a failed match. The matching vignette warns against overlooking this dependence on set size. A study, unlike this demonstration, would either match enough items or report the control by the differences the paragraph describes.
write_datasheet() writes both forms, and the Markdown
rendering carries a pre-registration skeleton with its materials section
already filled in.
paths <- write_datasheet(
ds, file.path(out, "datasheet.json"), file.path(out, "datasheet.md")
)
md <- readLines(file.path(out, "datasheet.md"))
grep("^#", md, value = TRUE) [1] "# Materials datasheet -- vignette_repro (english)"
[2] "## Provenance"
[3] "## Counterbalanced lists"
[4] "## Selection transparency"
[5] "## Realised control"
[6] "## Suggested analysis"
[7] "## Methods paragraph"
[8] "## Pre-registration template"
[9] "### Study information"
[10] "### Hypotheses"
[11] "### Design"
[12] "### Materials (from the lexsync datasheet)"
[13] "### Sampling plan"
[14] "### Analysis plan"
The sections left blank are the hypotheses, the sampling plan and the inference criteria, which no tool can fill. The analysis plan is the exception and comes with a suggested model.
[1] "response ~ condition + scale(length, scale = FALSE) + scale(n_density, scale = FALSE) + scale(old20, scale = FALSE) + (1 + condition | subject) + (1 | item)"
The model crosses random effects for subjects and items because items are a sample of the language rather than the population of interest. An analysis treating them as fixed generalises only to the words that were used (Clark, 1973; Yarkoni, 2022). The matched dimensions enter as centred covariates, computed from the stimuli file’s own columns, because matching leaves residual differences and Sassenhagen and Alday (2016) recommend adjusting for potential confounds. A structure is maximal only when every effect that varies within subjects or items has a random slope there (Barr et al., 2013). These covariates have none, so the accompanying note describes the slopes the model has. The note also advises simplifying a model that is singular or fails to converge (Matuschek et al., 2017). The suggested model is a starting point for a design of this shape.
Alongside the datasheet, which records what a run produced, the run log records what the run did.
log <- new_run_log(
"vignette_repro",
meta = list(seed = schema$seed, language = "english")
)
log <- log_step(
log,
sprintf("lexicon loaded: %d words", nrow(lex)),
list(words = nrow(lex))
)[lexsync] lexicon loaded: 3000 words
[lexsync] pool after filters: 859 words
[lexsync] wrote 'stimuli.csv'
log_md <- write_run_log(log, file.path(out, "run_log.md"),
file.path(out, "run_log.jsonl"))
cat(readLines(log_md), sep = "\n")# lexsync run log: vignette_repro
- Engine: R 4.6.2
- Started: 2026-11-21 19:59:25.103683
- Finished: 2026-11-21 19:59:25.10586
## Run metadata
- seed: 2026
- language: english
## Steps
- **2026-11-21 19:59:25.104481**: lexicon loaded: 3000 words
- words: 3000
- **2026-11-21 19:59:25.104884**: pool after filters: 859 words
- **2026-11-21 19:59:25.105294**: wrote 'stimuli.csv'
- path: /tmp/RtmpPssDGo/lexsync_repro/stimuli.csv
- rows: 30
- md5: 06832f9e8e1d901adde134e464579401
The pipeline builds this log as it goes, recording each stage with its parameters, the equivalence verdict on every dimension, any balance warning and an entry per file written with its MD5. The experiments’ loop and text tables are among them, and the stimuli and report tables are logged with their row counts too. A design or schema key that lexsync does not read is noted there as well, since it changed nothing about the run.
A Markdown log and a JSON Lines log are emitted. The Markdown log alone carries the run metadata, such as the seed and the schema path, and the JSON Lines log holds the steps, one object per line, so that a run can be checked by a script. The MD5 here is a content fingerprint for detecting changes between runs, whereas the datasheet’s SHA-256 is the stronger one that belongs in a permanent record.
The pipeline does all of the above in one call.
run_pipeline() on a design configuration is the
reproducible unit, and it writes the stimuli, the reports, the three
experiments, the datasheet and the log. Sharing the design file and the
corpus is then enough for someone else to regenerate the materials and
verify the checksums in either language.
Two habits keep a run reproducible. Pin the corpus, since a selection
is only reproducible against a fixed lexicon. For this reason, the
bundled corpora are a checksummed snapshot, and the
wordfreq connector draws on a package frozen in 2024, whose
data is a stable snapshot of usage through roughly 2021. Where
cross-engine reproducibility matters, prefer
standardised_euclidean, the default, or joint,
since they are the methods that carry the guarantee.
A reproducible lexicon is not a screened one. The repository’s
corpora/derived/en.csv, es.csv and
zh.csv hold the most frequent forms wordfreq
returns, so they include proper names, acronyms and other forms that are
not words of the language, and N and OLD20 are computed over them. Their
share differs between frequency bands, which can confound a frequency
contrast. corpora/derived/en_v2.csv marks such forms in the
English lexicon and counts N and OLD20 among the others only. The
package’s English example holds only forms that en_v2.csv
leaves unmarked, while its Spanish and Chinese examples are slices of
es.csv and zh.csv and are not screened. What
the derived lexica contain says what to check in a selection.
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001
Bochynska, A., Keeble, L., Halfacre, C., Casillas, J. V., Champagne, I.-A., Chen, K., Röthlisberger, M., Buchanan, E. M., & Roettger, T. B. (2023). Reproducible research practices and transparency across linguistics. Glossa Psycholinguistics, 2(1). https://doi.org/10.5070/G6011239
Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. https://doi.org/10.1016/S0022-5371(73)80014-3
Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001
Roettger, T. B. (2019). Researcher degrees of freedom in phonetic research. Laboratory Phonology, 10(1), Article 1. https://doi.org/10.5334/labphon.147
Sassenhagen, J., & Alday, P. M. (2016). A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests. Brain and Language, 162, 42–45. https://doi.org/10.1016/j.bandl.2016.08.001
Wilkinson, M. D., Dumontier, M., Aalbersberg, IJ. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017/S0140525X20001685