Licensing note for the lexsync package
======================================

The MIT licence declared in DESCRIPTION, whose copyright holder and year are in
the LICENSE file, covers the source code of this package only. Three of the data
files bundled under extdata (inst/extdata in the sources) are third-party
derivatives carrying their own terms, which are set out here.


Bundled corpus derivatives
--------------------------

    extdata/en_example.csv
    extdata/es_example.csv
    extdata/zh_example.csv

These small example lexica are derived from the wordfreq package (Speer, 2022,
https://doi.org/10.5281/zenodo.7199437). wordfreq distributes its code and its
data under different terms. The code is under the Apache License 2.0, and the
data files, which are what these three derivatives come from, are
redistributable under

    Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
    https://creativecommons.org/licenses/by-sa/4.0/

The three files above are therefore distributed under CC BY-SA 4.0 rather than
under the MIT licence. Anyone redistributing them, or an adaptation of them,
must give appropriate credit to the corpus authors, indicate that changes were
made, and place any adaptation under the same licence. Everything else in the
package, including the other files under extdata, is MIT.

One of the three has a second direct source. en_example.csv holds 3,000 of the
forms of the repository's screened English lexicon, corpora/derived/en_v2.csv.
That lexicon holds every form of corpora/derived/en.csv and marks as nonstandard
each form SUBTLEX-US (Brysbaert & New, 2009) lacks or whose uses there begin with
a lower-case letter less than a tenth of the time, and en_example.csv holds only
forms it leaves unmarked. SUBTLEX-US, which openlexicon distributes under the same
licence, CC BY-SA 4.0, thus chose the file's rows, and the file's N and OLD20 count
neighbours among the forms en_v2.csv leaves unmarked.
SUBTLEX-US is therefore a direct source of en_example.csv as well as one of the
lists wordfreq blends.


What lexsync changed
--------------------

The frequencies are wordfreq's own Zipf values, carried through unaltered. What
lexsync added is a standardised column layout, shared by every corpus the
package reads, and two orthographic-neighbourhood variables computed here from
the word list itself: Coltheart's N, the count of same-length neighbours
reachable by a single letter substitution, and OLD20, the mean Levenshtein
distance to the twenty nearest words. Each row names its origin in the 'source'
column. For en_example.csv, both variables count neighbours among the forms the
SUBTLEX-US screen keeps, as described above.


Credit for SUBTLEX
------------------

wordfreq's frequencies blend up to eight domains of text, among them Wikipedia,
OPUS OpenSubtitles, NewsCrawl, Google Books Ngrams, OSCAR, Twitter and Reddit.
The subtitles domain draws in part on the SUBTLEX word lists, which wordfreq
carries by permission on two conditions that pass through to anything derived
from it: work derived from wordfreq must credit the SUBTLEX authors, and it must
remain clear that SUBTLEX is freely available data. Both are honoured here. The
lists are freely available, and their authors are credited as follows.

  Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical
  evaluation of current word frequency norms and the introduction of a new and
  improved word frequency measure for American English. Behavior Research
  Methods, 41(4), 977-990. https://doi.org/10.3758/BRM.41.4.977

  van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014).
  SUBTLEX-UK: A new and improved word frequency database for British English.
  Quarterly Journal of Experimental Psychology, 67(6), 1176-1190.
  https://doi.org/10.1080/17470218.2013.850521

  Cai, Q., & Brysbaert, M. (2010). SUBTLEX-CH: Chinese word and character
  frequencies based on film subtitles. PLoS ONE, 5(6), e10729.
  https://doi.org/10.1371/journal.pone.0010729

  Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A. M., Bölte, J., & Böhl, A.
  (2011). The word frequency effect: A review of recent developments and
  implications for the choice of frequency estimates in German. Experimental
  Psychology, 58(5), 412-424. https://doi.org/10.1027/1618-3169/a000123

  Keuleers, E., Brysbaert, M., & New, B. (2010). SUBTLEX-NL: A new measure for
  Dutch word frequency based on film subtitles. Behavior Research Methods,
  42(3), 643-650. https://doi.org/10.3758/BRM.42.3.643

SUBTLEX contributes to the blend rather than constituting it. No file bundled
here is a copy of a SUBTLEX list. As described above, SUBTLEX-US screened the rows
of en_example.csv. Otherwise, the SUBTLEX resources the corpus registry offers are
separate corpora a user may fetch directly, not the source of anything shipped in
this package.


Corpora fetched at run time
---------------------------

Corpora downloaded on demand through the registry are not redistributed with
this package. Each carries its own licence and citation, recorded in its entry
in extdata/registry.yaml. list_corpora() reports the citation, and fetch_corpus()
gives it in a message after a download.


The fuller account
------------------

The repository keeps the complete record, including the provenance caveat that
travels with the Chinese corpus and the retrieval date and checksum of every
derived file:

    https://github.com/pablobernabeu/lexsync/blob/main/LICENSE-DATA
    https://github.com/pablobernabeu/lexsync/blob/main/corpora/ATTRIBUTION.md
