A release about the gap between what the package could do and what it said it could do. 0.1.0 shipped with no bundled word segmenter, no romanisation and no script conversion, on the reasoning that each needed a dictionary the package could not carry. That was half right. The dictionaries were already installed: ICU carries them, stringi carries ICU, and stringi has been a hard dependency since the first commit. Most of what follows is that discovery being spent.
Four vignettes are new, and there were none before.
engine = "icu" is a real word
segmenter.
cjk_segment("我今天很開心", engine = "icu") returns
我 / 今天 / 很 /
開心, words rather than characters, using ICU’s
dictionary-based break iterators for Chinese and Japanese. It costs no
new dependency.... reaches the engine, locale
included, though see “How the new verbs behave” below, because
locale does not do what its name suggests here.engine still has no default, for the reason it always
had: "character" answers a different question from the one
a caller asking for words is asking, and would otherwise be the answer
they got by accident.cjk_tokens() gains
output, naming the token column. The default is
still "token", and an existing column of that name is still
replaced, but a caller whose data already has a token
column worth keeping now has somewhere else to put the tokens, the
escape hatch tidytext’s
unnest_tokens() provides. It follows ..., so
it has to be given by its full name.as.character() deparses a
list rather than coercing it, so an engine returning
list(c(1, 2)) used to yield the single token
"c(1, 2)". An atomic element is still coerced, so integers
and factors work as before.cjk_romanize() romanises Han, kana and
Hangul to the Latin alphabet. ascii = TRUE drops tone
marks, which is what you want for a sort key.cjk_simplify() and
cjk_traditionalize() convert between simplified
and traditional Han.to_hiragana() and
to_katakana() convert between the kana
syllabaries. Halfwidth input is handled and composed on the way through:
to_katakana("ｶﾞ") is the single character
ガ.cjk_jamo() and
cjk_compose_jamo() decompose Hangul syllables into
jamo and put them back, by the syllable arithmetic in the Unicode
standard rather than by normalisation. The round trip is exact, and text
that is not Hangul passes through untouched: an accented letter keeps
its accent, and a compatibility ideograph, which NFD and NFC would both
replace with its unified form, stays as it is.0.1.0’s “Not in this release” section withheld traditional/simplified
conversion because character-level conversion “is wrong often enough to
matter”. That is true of a character table, and ICU’s transform is not
one. The limits that remain are on the help pages and in
vignette("transliteration"):
头发 gives 頭髮 and 发送 gives
發送, 后天 gives 後天 while
皇后 stays 皇后, and all twenty such words
checked for this release came out right. What it does not do is
substitute regional vocabulary: 软件 becomes
軟件, not Taiwan’s 軟體. OpenCC and its regional
configurations are the answer there.银行 (“bank”) comes back
yín xíng rather than yín háng; and Japanese
kanji come back in pinyin, 日本語 giving
rì běn yǔ rather than nihongo. Use it for a
sort or search key, and a morphological analyser such as gibasa for Japanese
readings.cjk_sentences() splits text into
sentences with ICU’s sentence break iterator, which knows U+3002, U+FF01
and U+FF1F. A regular expression on [.!?] finds no boundary
at all in Chinese or Japanese, which is the reason this exists. The
spans are returned unmodified, so the pieces concatenate back to the
input exactly; whitespace between two sentences stays attached to the
first rather than being trimmed away.cjk_ngrams() returns every run of
n consecutive characters. Character n-grams are the
standard dictionary-free baseline for Chinese retrieval and
classification: most words are one or two characters, so bigrams capture
the majority of them with no model, and they degrade gracefully on the
names and coinages where a segmenter is least reliable. No gram is
formed across whitespace.cjk_strip_punct() removes punctuation by
Unicode category, not by a POSIX class. The usual spelling is
unreliable on CJK twice over. gsub("[[:punct:]]", "", x) is
resolved through the C library, so it removes nothing under
LC_ALL=C and everything under a UTF-8 locale: the same
script, two answers, no warning. perl = TRUE looks safer
and is worse, PCRE’s POSIX classes being ASCII-only unless
(*UCP) is set, so it silently removes no CJK punctuation in
any locale at all.replacement defaults to a space rather than
"". Deleting a full stop closes the gap, and the
characters that flanked it become adjacent, so cjk_ngrams()
then reports a bigram spanning a sentence boundary, a word that was
never written. A space keeps the boundary, and every verb here that
walks a string already declines to cross whitespace.symbols = TRUE additionally removes General_Category
S: the fullwidth tilde, currency signs and mathematical
operators. It is off by default because a currency sign is often
content.cjk_sort() and
cjk_order() sort CJK text by an ICU collation.
sort() on Chinese gives code point order, which is a real
order and the wrong one: the unified ideographs are laid out in KangXi
radical-stroke order, so sorting by code point sorts by radical within
the base block. It is not phonetic, which is what sorting a column of
names calls for, and it is not even radical order across blocks, since
every Extension A character sorts ahead of every base-block one.
locale = "zh" sorts Han by pinyin;
"zh-u-co-stroke" by stroke count.NA
is kept and sorts last, where stri_sort() drops it and
silently shortens a column. And a locale whose language ICU has no
collation for is an error, where stringi warns and falls back to the
root collation, a plausible-looking wrong order. An unrecognised
region is not an error: "zh-CH" resolves to
"zh" and still sorts by pinyin.cjk_normalize() applies the Unicode
normalisation forms ("nfc", "nfd",
"nfkc", "nfkd" and
"nfkc_casefold") so that strings which look the same
compare the same. The package had been telling callers to “normalise
first” without giving them a way to do it."nfc" rewrites them exactly as "nfkc" does:
460 of the 472 characters in the CJK Compatibility Ideographs block are
folded away, along with all 542 of the Supplement block. The 12
survivors are listed. If that distinction carries meaning in your data,
as it can in Korean and Japanese name records, keep the original column,
because no form preserves it.drop_variation_selectors = TRUE removes them
first. Before, not after: a variation selector has combining
class zero and blocks canonical composition across itself, so stripping
one after normalising can leave text that is no longer in the
form just requested. A U+FE00 U+0300 normalises to itself
under NFC and to U+00C0 once the selector is gone."nfkc_casefold" deletes every
Default_Ignorable code point, which the other four
forms keep: the variation selectors, the zero-width joiner and
non-joiner, the zero-width space, the soft hyphen, the tag characters
and U+FEFF. So a byte-order mark survives the other four and not this
one. It is documented rather than worked around, because it is what the
form is defined to do; every other verb in the package preserves a
leading U+FEFF.to_halfwidth() and to_fullwidth() are
unchanged and remain the narrow alternative: they move text along the
width axis and touch nothing else.cjk_wrap() wraps text to a width in
terminal columns, completing the set with cjk_pad() and
cjk_truncate(). Break positions come from ICU’s
implementation of the Unicode line breaking algorithm (Annex #14), so a
line never begins with 。 or ） and an opening
bracket never ends one, none of which splitting every n columns
would respect. Lines are filled greedily and come back joined by
\n, so the result is the same length as x.
indent and exdent are supported.cjk_sentences() and
cjk_sort(), a bare “invalid multibyte string” from
cjk_wrap(), and the bad byte silently dropped by the
"icu" engine. Every verb now stops with “x
must be valid UTF-8” before doing any work, naming the argument
(col in the tidy verbs, and pad,
ellipsis and replacement for the arguments
that carry text of their own). A string marked "bytes" is
refused the same way.match.arg() names a variable the caller never wrote: a
value that is not character becomes “‘arg’ must be NULL or a character
vector”, and a string matching no choice “‘arg’ should be one of”, both
with match.arg()’s own call attached.
cjk_pad(side = ) and cjk_normalize(form = )
now say `side` must be one of "right", "left", "both",
listing the choices. Every value match.arg() accepted
before is still accepted, including partial matching and
NULL meaning the first choice.locale is an error whatever the
input. The check used to run inside the call to stringi, so
cjk_sort(character(0), locale = "xx") and
cjk_wrap(NA, 5, locale = "xx") passed. It now runs first.
The language is matched without regard to case, as BCP 47 specifies, so
"ZH" and "Zh-Hant" are accepted;
"root" and "und" ask for ICU’s root locale by
name; and "" means the session default, as it does to
stringi.cjk_wrap() refuses an indent or
exdent past the integer range by name, rather than handing
stringi an NA it reports in its own terms.cjk_pad(width = ) no longer warns before its
error on a closure or an environment. The check has to call
is.na() so that an all-NA logical width can
pass and propagate, and is.na() on a function warns;
testing is.atomic() first keeps both properties.Error: `n` must be a single positive whole number. rather
than repeating the whole call back at the reader. A test walks the
installed namespace to keep it that way, and checks that the three
handlers which re-raise someone else’s condition leave its call and
message untouched.?tidycjk is brought up to date. Its
contract section lists every verb rather than the 0.1.0 ones. Its “Input
encoding” section gives working ways to read a legacy file,
read.csv(f, fileEncoding = "GBK") and
readLines(file(f, encoding = "Shift_JIS")); the
encoding argument of those functions only labels strings,
and read.csv(f, encoding = "GBK") fails outright in a UTF-8
session. And its related packages are ones you can install: 0.1.0
pointed at pinyin, tmcn, zipangu and Nippon, all four of which are
archived from CRAN.?cjk_truncate
contradicted its own output. “six columns is three ideographs”
sat above a call returning one ideograph, because the ellipsis is
counted against the budget and costs three of the six columns. It now
says what the output shows, and a third call with
ellipsis = "" makes the original point properly.\uXXXX escapes so that
no CJK glyph reaches the LaTeX manual, and ?cjk_ratio,
?cjk_summary, ?cjk_char_counts and
?cjk_tokens had no comment saying what those escapes
spell.Seven things in this release are fixes in the sense a 0.1.0 user cares about. The rest of what changed during development concerned code that had never shipped, and is described under the features it belongs to rather than dressed up as a fix.
x, which is what
lets a numeric or a factor column through. But
as.character() does not coerce a list, it deparses
it, so cjk_width(list(c("a", "b"))) answered
11, the display width of the eleven-character string
c("a", "b"), with no error and no warning. Every exported
verb had this hole, and the list a caller has in hand is usually another
verb’s output. All of them now raise, naming the two ways out, and the
three tidy verbs check a pulled list-column the same way.
NULL is still accepted, and still gives
character(0): is.atomic(NULL) was
TRUE until R 4.4.0 and is FALSE after it, so
the guard admits it explicitly rather than inheriting a contract that
changed under it.cjk_detect_language() takes
any kana as proof of Japanese, and U+30FB sits in the Katakana block,
but it is punctuation that Chinese and Korean text use too, to separate
the parts of a transliterated name. So "서울・부산" (“Seoul
/ Busan”) came back "japanese". The dot, and its halfwidth
form U+FF65, no longer count as kana.?cjk_blocks said every phonetic script was covered in full,
extension blocks included, and four kana blocks were missing: Kana
Supplement, Kana Extended-A, Kana Extended-B and Small Kana Extension,
which hold the hentaigana, the archaic and small kana and the Minnan
tone letters. Extension J, the 4,298 ideographs Unicode 17.0 added, was
missing too. has_cjk() answered FALSE for all
of them. The table is now current to Unicode 18.0, and the kana blocks
are split where Unicode’s Script property changes, so each letter is
labelled hiragana or katakana as it should be.cjk_detect_language(han_only = ) no longer
turns two malformed values into languages.
is.na(NaN) is TRUE, so NaN was
returned as the language "NaN"; and
is.na(list(NA)) is TRUE while
as.character(list(NA)) is the string
"NA", not a missing value, so a downstream
is.na() would have called it a real answer. Both are now
errors.cjk_segment() and cjk_tokens() no
longer delete a byte-order mark.
stri_split_charclass() reads a leading U+FEFF in its own
input as a byte-order mark and drops it, and
omit_empty = TRUE then discards the emptied piece, so a
mark beginning a non-CJK run disappeared from the
"character" engine’s output. U+FEFF was the only zero-width
character this happened to. Both engines now count the leading run off
and restore it, as the other verbs already did.cjk_segment(character(0), engine = "nope") and a zero-row
cjk_tokens() returned quietly. An engine you register
yourself is still not called on an empty vector.tidycjk. The repository had been created as
tidyckj, with the last two letters transposed, and the typo
reached the URL and BugReports fields, the
pkgdown site, the R-CMD-check badge and the pak::pak()
install line. GitHub redirects the repository addresses, so the ones in
the 0.1.0 tarball still resolve; it does not redirect a project Pages
site, so the old documentation URL 404s and this release is what
corrects it on CRAN.Not fixes, since these verbs are new, but the details most likely to surprise you, gathered in one place.
stri_sort(), stri_split_boundaries() and
stri_wrap() all read a leading U+FEFF as a byte-order mark
and drop it. cjk_sort(), cjk_sentences() and
cjk_wrap() hide the whole leading run from stringi and
restore it, and the jamo verbs work on code points, so a mark from an
Excel-written CSV is not silently deleted. A U+FEFF elsewhere
in a string may still be lost to cjk_wrap(), because
re-flowing can put it at the start of a segment; ?cjk_wrap
says so.cjk_sort() is
x[cjk_order(x)], so it can only reorder its input.
A sort must not rewrite the values it was handed, and building it on
stri_sort(), which returns round-tripped values, would let
it.cjk_wrap() fills lines greedily, as a terminal
does. stringi’s own default is an optimal fit that evens out
line lengths, and it is not used: its cost grows far faster than the
text (40,000 characters of CJK exhaust 1.5 GB of memory), and appending
text can re-flow every line above it, which it did in 424 of 2,000
random trials where the greedy fill never does.cjk_wrap() re-flows rather than
measures. Existing newlines are whitespace to the algorithm and
are replaced by the new breaks, so "a\nb" wrapped wide
comes back as "a b". A string of nothing but whitespace
re-flows to "", where cjk_pad() would have
kept it.cjk_wrap() returns NFC, which no other layout
verb does. ICU’s line-breaking works on normalised text, so
stri_wrap() normalises and this verb inherits it:
cjk_wrap("豈", 2) is U+8C48. A sweep of every code point in
Unicode puts the affected set at 1,120: 460 of the 472 assigned CJK
Compatibility Ideographs, all 542 of the supplement, 34 Hebrew
presentation forms, 13 musical symbols, and 71 scattered singletons and
composition exclusions such as the Kelvin and Ohm signs, Greek letters
with an oxia and Devanagari letters with a nukta. That is exactly the
set NFC changes, no more and no fewer: each is a duplicate of a
character or sequence Unicode prefers, kept for compatibility. Everyday
text, precomposed Latin and Hangul syllables included, is already NFC
and passes through untouched. normalize = FALSE would stop
it, but that argument also turns off whitespace collapsing and makes
stri_wrap() raise on any string containing a newline, so it
is documented and pinned by a test rather than changed. Call
cjk_normalize() first if you want the normalisation to be
explicit.to_katakana(to_hiragana(x))."icu" drops punctuation and symbols along
with whitespace; "character" keeps CJK punctuation as
tokens, because those code points are in blocks
cjk_blocks() lists. The same sentence therefore yields
different token counts, and the gap is punctuation rather than a
disagreement about where words end. An emoji goes the same way.locale, and the "icu" engine, shares one
guard, which checks the language subtag against
stringi::stri_locale_list() rather than watching for a
warning that is not dependable. An unrecognised region still
resolves to its language, so "zh-CH" and
"zh-u-co-stroke" are valid.locale does not change CJK word or sentence
boundaries. It is accepted and forwarded, but the CJK
dictionary is chosen by the script of the text: seven Chinese, Japanese
and Korean strings segment and split identically under
"zh", "ja", "ko",
"en" and the session default. It does select the
Annex #14 line-breaking style in cjk_wrap(), which is why
that verb takes one.test-invariants.R checks, over every
code point in the block table, that each normal form is a fixed point of
itself, that nfc(nfd(x)) is nfc(x) and
likewise for NFKC, that the width and kana conversions are idempotent,
that fullwidth text survives a trip to halfwidth and back, that the jamo
round trip is the identity, that the layout verbs leave text that
already fits alone, and that every transform returns one valid UTF-8
string per input. All 11,172 Hangul syllables decompose into 2 or 3 of
67 conjoining jamo and recompose exactly. The exhaustive pass is
skip_on_cran(); a sampled pass and every block boundary run
everywhere.has_cjk(), cjk_ratio() and
cjk_script() agree with the block table in both directions,
no character wrongly called CJK and none wrongly left out, and that
cjk_width() is never missing and never outside 0:2.RoxygenNote replaces
Config/roxygen2/version: man/ is generated by
roxygen2 7.3.2, and the file now records what actually built it.data-raw/unihan.R downloads and parses it, but nothing is
bundled yet.stringi::stri_enc_detect() exists and is unreliable on
short CJK samples, which is exactly when it would be used. It is not
wrapped rather than wrapped badly.cjk_compose_jamo() taking the list
cjk_jamo() returns. It would make the round trip
compose directly instead of needing the vapply() step, and
the case for it is real: that boilerplate sits in the middle of this
package’s own worked example. Declined because it would reopen, for one
function, exactly the hole closed everywhere else this release:
cjk_sentences() and cjk_segment() also return
lists, and cjk_compose_jamo(cjk_sentences(x)) would then
quietly paste sentences together instead of saying no. One function that
guesses is worse than one line of boilerplate, and the error now names
the line to write.First release.
tidycjk is a tidy toolkit for Chinese, Japanese and
Korean text: script and language classification, display width, width
normalisation, and a pluggable word-segmentation engine that the caller
names, as verbs that return tibbles. The package itself has no compiled
code, bundles no data, and makes no network requests.
cjk_summary(data, col) reports n_docs,
n_with_cjk, prop_with_cjk and
mean_ratio for a text column.cjk_char_counts(data, col) returns one row per distinct
CJK character with its code point, script, Unicode block and count.cjk_segment(x, engine) splits CJK text into words, and
cjk_tokens(data, col, engine) is the tidy version,
returning one row per token.cjk_segmenters() lists what is available and
register_cjk_segmenter() adds an engine: any function of
(x, ...) returning a list of character vectors.engine has no
default. jiebaR
was the obvious candidate and was archived from CRAN on 2025-05-01, so
it cannot be a dependency of a CRAN package.
?cjk_segmenters shows the four lines that register it once
you have installed it from source."character": one token per
CJK character, with runs of non-CJK text split on whitespace. It is
character tokenisation rather than word segmentation and says so on its
help page. Making it the silent default would have handed character
tokens to callers asking for words, which is the mistake this package
exists to avoid, so the choice is explicit instead.has_cjk(), cjk_script() and
cjk_ratio() classify and measure.cjk_detect_language() infers the language from the
scripts present.cjk_width(), cjk_pad() and
cjk_truncate() work in terminal columns rather than
characters.to_halfwidth() and to_fullwidth()
normalise width variants.cjk_blocks() exports the Unicode block table the
package is built on, covering every unified ideograph block through
Extension I as well as the phonetic scripts, CJK punctuation and the
width variants.Language detection returns NA rather than
guessing. Japanese written without kana is not distinguishable
from Chinese by script alone, so a Han-only string gets NA.
cjk_detect_language(x, han_only = "chinese") opts into the
guess and keeps the assumption visible in the calling code.
Width is delegated to stringi.
cjk_width() and cjk_pad() wrap
stringi::stri_width() and stringi::stri_pad(),
which read the live Unicode tables in ICU, the Unicode Consortium’s C
library. A hand-maintained range table would go stale at every Unicode
release, and the version commonly copied around is already wrong for
tens of thousands of assigned code points: it stops below the
supplementary planes, so every ideograph in Extensions B through I comes
out one column instead of two. cjk_truncate() has no
stringi equivalent and is implemented here.
Normalisation is surgical.
to_halfwidth() maps fullwidth ASCII, the ideographic space
and halfwidth katakana, and touches nothing else. NFKC
additionally rewrites ligatures, superscripts, Roman numerals, circled
numbers, the no-break space and the CJK compatibility ideographs, which
is almost never wanted.
Voiced marks are composed by default. Halfwidth
katakana writes a voiced syllable as two code points;
compose = TRUE folds them into the single precomposed
character, so "ｶﾞ" becomes "ガ". Every pair in
the composition table agrees with Unicode NFC, including the five that
break the base-plus-one rule.
Ties never depend on the locale.
cjk_script() and cjk_char_counts() break ties
by first appearance rather than by collation order.
data-raw/unihan.R downloads and parses it,
but no character data is hand-written or bundled yet. pinyin and hanyupinyin
are on CRAN today.