The repository carries a Shiny front-end over this package for
building a design without writing code. It runs the workflow in a
browser tab, from choosing a paradigm to downloading the stimuli, the
realised-control report and the materials datasheet. Every number it
shows comes from the installed lexsync package, and the
design it assembles is an ordinary YAML file that you can save, share
and run from a script afterwards.
Because the app needs a live server, this article describes it without running it, and the code is shown but not evaluated.
The app is not part of the package. It is kept in the repository,
under apps/r_shiny, beside the corpora and the example item
tables that it reads, so launch it from a clone. The interface needs
shiny, bslib and DT. The downloads use the zip package or, where it is
not installed, a zip tool on the path or named by R_ZIPCMD.
All four packages are listed in the package’s Suggests.
Run that from the root of the clone. The app opens at
http://127.0.0.1:8502. It looks for
corpora/derived/ in the working directory and then in the
two directories above it, and it finds the example item tables under
items/ the same way. Launched from somewhere else, it has
no lexicon to offer.
The repository also carries a launcher for the terminal, which takes an optional port and is what a container or a headless session should use.
The launcher fixes the host and the port and never opens a browser itself. A server started with an auto-launched browser or an unbound port can exit the moment it is backgrounded. This is the usual reason an app appears not to stay up in an automated environment.
The main panel changes with the paradigm.
‘Lexicon’ lists the derived corpora the clone holds under
corpora/derived/, which are English, English screened
against SUBTLEX-US (en_v2, described in
vignette("matching-and-designs")), Spanish, a Spanish
lexicon carrying grammatical gender, and Mandarin Chinese. Two sliders
then narrow the lexicon to the candidate pool the design will consider,
one for length in letters or characters and one for frequency in Zipf.
They are the design’s pool_filters, applied as a separate
step that matching never re-reads.
The app opens on en_v2, the lexicon the repository’s
English designs use. For a lexicon with a nonstandard
column, such as en_v2, the app also shows a ‘Screened forms
only’ checkbox, ticked from the start. It adds
nonstandard: [0, 0] to the pool filters and so keeps only
the forms the screen leaves unmarked. Clearing it lets the marked forms,
such as names and acronyms, back into the pool.
The conditions are an editable table. Each row names a condition and
defines it as a subset of the pool, either by a numeric window through
the dimension, lower and upper
columns or, for a categorical column such as the gender of the Spanish
lexicon, by a set of comma-separated values through
categories. The optional dimension2,
lower2 and upper2 columns add a second factor
to the same row, so one row can define a cell of a 2 × 2. Three presets
fill the table in one click, a high versus low frequency contrast, a
dense versus sparse neighbourhood contrast and a 2 × 2 of frequency by
neighbourhood. A preset also sets the matched dimensions and the
matching method that suit it. The neighbourhood preset sets matching on
length and frequency by the joint method, and the 2 × 2
preset sets matching on length alone. Every control stays editable
afterwards.
Below the table are the choices that govern the match. ‘Items per
condition’ sets how many stimuli each condition should hold. ‘Matching
method’ offers the four methods the engine implements,
standardised_euclidean, joint,
mahalanobis and optimal, and the note beneath
it says that the last two agree only closely between the engines.
‘Resampled disjoint sets’ draws several matched sets that share no item,
which lets items be treated as a random factor. Zero or one switches it
off. ‘Match on’ picks the dimensions to equate across conditions, from
length, frequency, neighbourhood N, OLD20, syllable count and bigram
frequency. A warning appears beneath it when it names a dimension whose
windows differ between conditions, because matching on a dimension the
conditions manipulate works against the contrast they define. A window
that every condition shares only narrows the pool and draws no warning.
With nothing chosen, the app says so and keeps ‘Run design’ disabled.
‘Counterbalancing lists’ sets how many lists the items are rotated
across.
The advanced panel underneath holds one tolerance window per matched dimension, the half-width in standard deviations of the band a candidate must fall inside. It is collapsed by default because the schema already carries a default for every dimension. A zero there leaves that default in place, although you might expect it to narrow the window to zero width. Reproducing the exact window of a published study is the usual reason to open the panel. The panel gives the worked example that Matching, dimensions and designs discusses, a frequency k of 0.1111111111111111 for a band of the mean plus or minus a ninth of a standard deviation.
For a lexical-decision design, the app uses the same lexicon and pool
filters and then asks only for the number of items per condition, with
words and pseudowords equal in number, and for the generator.
letter_substitution, the engine default, changes as few
single letters as it can while keeping every bigram attested.
subsyllabic swaps whole onset, nucleus and coda
constituents in the manner of Wuggy (Keuleers & Brysbaert, 2010).
Both are deterministic, so the pseudowords are the same ones the package
would generate from a script.
These two paradigms read prepared items from a table and involve no
corpus and no matching. The app uses the bundled example table for the
paradigm, items/priming_pairs_en.csv or
items/spr_sentences_en.csv, and asks only how many
counterbalancing lists to build.
‘Run design’ assembles the design and lays out a temporary folder the
way the download will be laid out. The design goes in under the name the
‘Reproducible code’ tab gives it, with the package’s schema at
config/schema.yaml and the lexicon or item table at the
path the design names. The app then makes the exported call there,
run_pipeline("<name>.yaml", schema_path = "config/schema.yaml", outdir = "output").
The run therefore reads only relative paths, and its datasheet checksums
the files the folder holds. The working directory moves to the folder
for that call alone. Because R serves every session on one thread, no
other session sees the moved directory.
Before the run, the app checks that a lexicon has been chosen where the paradigm needs one and that a factorial design defines at least two conditions. An error raised by the pipeline is shown with the pipeline’s own message. A successful run reports how many rows were selected and opens the results.
The results appear in six tabs, which between them cover what was selected, whether it is usable, what it will present and how to reproduce it.
‘Stimuli’ is the selected set as a table, the same one the run wrote to disk.
‘Realised control’ shows whether the set is usable. It covers every pair of conditions on every dimension the report includes. Each row gives the dimension’s role (manipulated, controlled or uncontrolled), the raw difference, Cohen’s d with its confidence interval, the variance ratio and the verdict of the two one-sided tests procedure. The interval matches the tests at the schema’s alpha and covers 100(1 - 2 alpha) per cent. Its heading takes that level from the run’s datasheet, so it reads 90% at the default alpha of .05.
The tab then draws the absolute standardised mean difference per dimension and pair as a bar chart with a dashed line at 0.5 standard deviations. Manipulated dimensions stand well above that line and matched dimensions sit near zero, so the chart shows whether the matching worked. A high bar for an uncontrolled dimension shows a difference that the design did not hold constant. The per-condition descriptive statistics follow underneath. A design drawn from an item table produces no corpus-matching report, and the tab says so in place of an empty table.
‘Datasheet’ renders the materials datasheet for the run, with the provenance of the corpus, the checksums, the realised control and the pre-registration skeleton.
‘Experiment scripts’ offers one download per presentation target,
each compiled from the same trial description. The PsychoPy download is
a zip archive holding the script and the conditions file it loads from
its own folder. The OpenSesame download is a zip archive holding the
.osexp, one per counterbalancing version when the design
has several, with the loop and text tables each one reads. Unzip either
and run the experiment from the folder it makes. The jsPsych page
carries its trials and runs in a browser as it stands.
‘Reproducible code’ shows the design configuration as YAML, with
repository-relative paths such as corpora/derived/en.csv,
the same paths the run read. Below the YAML are the one-line R, Python
and command-line calls that reproduce the same operation. Saving the
YAML under the name the panel gives it and running any of those three
lines repeats the run outside the app. The command lines quote the
design file’s name, and their quoting holds in bash,
cmd.exe and PowerShell. The sentence under them is taken
from the datasheet’s cross-engine record. Under the deterministic
methods, it says that the two engines select byte-identical stimuli and
trial order, and under mahalanobis and optimal
that they select equivalent but not byte-identical stimuli.
‘Download’ archives the folder the run ran in, under the design’s
name. It holds the design, the schema at
config/schema.yaml, the lexicon or item table at the path
the design names and every output under output/. The
outputs are the stimuli, the reports, the datasheet, the run logs and
the three experiments. The datasheet inside records the checksum of the
design file beside it and no path outside the folder. Running the
exported R call, or its Rscript form, in the unzipped
folder repeats the run. The repeated run rewrites every output with the
same bytes, apart from the run logs, which record when they were
written. The Python lines write the Python engine’s files beside
them.
A session in the app ends with the same artefacts a scripted run produces. The design YAML is portable across both engines, so a set assembled here can be reproduced by a collaborator working in Python, byte for byte under the deterministic matching methods. Reproducibility, parity and the materials datasheet covers the scope and limits of that guarantee, and Matching, dimensions and designs explains the realised-control report column by column.
Keuleers, E., & Brysbaert, M. (2010). Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3), 627–633. https://doi.org/10.3758/BRM.42.3.627