No function was gained or lost. This release fixes an append that could leave a table unreadable, filters that returned the wrong rows without saying so, including on any table whose schema another engine has changed, and a hang in forked workers. It stops handing credentials to things that should not have them, and it installs where 0.1.0 could not: on Alpine Linux, and with the older Rust toolchain on CRAN’s oldest macOS builder.
A filter on a column that older data files lack returned
rows R would not. After another engine adds a column, the files
written before it do not have one, and iceberg-rust 0.10
answers a predicate on such a file with a fixed value per operator, some
of them TRUE: b < 15 returned every row of every older
file, each with b NA, and so did
b <= 15, !(b > 15),
s < "y" and !startsWith(s, "x"). Each
comparison now also requires a value to be present, which is what R and
Iceberg’s own null rules both say. Of 100 random filters over such a
table, 15 disagreed with R before and none do now.
A read made after another engine changed the schema
without writing checked names against the wrong schema. An
ALTER TABLE ... RENAME COLUMN makes a new schema current
while the current snapshot still names the old one, and
iceberg-rust reads with the snapshot’s. filter
and select were checked against the table’s current schema
instead, so the renamed column could be selected by neither name, an
added one passed the check and failed in planning, and a read that found
no rows reported columns one that found rows did not. Names now resolve
against the current snapshot’s schema, and selecting a column only the
newer schema has says why it is refused.
case_sensitive = FALSE broke filters on a
table with both id and ID. Each name
was resolved to the right column here, then bound again by
iceberg-rust, whose case-insensitive index keeps only one
of the two, so a filter on the other failed with “Can’t convert datum”,
or, between columns of the same type, read the wrong one. An empty
result could also report the wrong one of the pair.
A decimal filter value below about 1e-5, or of 1e16 or
more, was refused as “exponent notation” although R had written
it out in full, so a decimal(18, 8) column could not be
compared with 0.00000002 at all.
An append to a table registered from a metadata file
outside the table’s metadata directory wrote its
Parquet and then failed to commit, leaving the files behind. It is now
refused before anything is written, as a misnamed file already was. Both
checks are now skipped for a REST catalog, whose server chooses metadata
locations itself and may name them as it likes.
icebergr_create_table(location =) recorded
the path as typed. "~/tables/events" created a
directory literally called ~ in the working directory, a
relative path left the table readable only from there, and
"" tried to create /metadata at the filesystem
root. A local path is now expanded and made absolute, and an empty one
is refused.
An infinite Date or POSIXct in a filter
is refused as infinite, rather than reaching iceberg-rust
as a date to parse spelled “Inf”.
icebergr_append(properties =) could make a
table unreadable. A property named operation was
written as a second "operation" key in the table’s
metadata, after which no engine could load, register or append to the
table again. Iceberg’s own summary metrics were open to it too: a
deleted-records property was subtracted into
total-records. Every key Iceberg writes into a snapshot
summary itself is now refused, and so is a name given twice.
A timestamp filter was a microsecond out.
Fractional seconds were truncated rather than rounded, and the double R
holds for a microsecond timestamp is often just below it, so
ts == x, with x read back from the same table,
found no row for most values, and >= or
< moved the boundary.
A POSIXlt filter value or as_of
was read in the wrong zone. Its wall-clock fields were taken as
UTC, so a time from strptime(..., tz = "America/New_York")
filtered or travelled four or five hours away from the instant it
named.
A filter now keeps the rows R would, where
NA and NaN are concerned. It was
handed to Iceberg with Iceberg’s semantics, which differ from R’s in
four ways. x > 1 matched NaN, and only in
files the scan read, since file statistics leave NaN out:
of two identical NaN rows, one could come back and the
other not. is.na(x) missed NaN.
x == 0 missed a stored -0. And
!(x %in% c(1, 2)) dropped the rows where x is
NA, which R keeps because %in% is never
NA. Checked against R’s own evaluation of 3,200 random
filters over columns holding all of these. String ordering still follows
Iceberg’s byte order, which is R’s only in the C locale.
A column on the value side of a filter was read as a
local variable. filter = a > b, with
b a column, compared a against whatever local
b was in scope, silently. An Iceberg predicate cannot
compare two columns, so this is now an error that says so.
A forked worker hung forever. After any icebergr
call in the parent, the first call in a
parallel::mclapply() worker waited on runtime threads that
do not survive fork(), with no error, and neither Ctrl-C
nor the timeout could end it. It now fails at once and points at
parallel::makeCluster().
A data frame naming a column twice lost the second one on append. It is now refused.
icebergr_register_table() accepts an object-storage
location such as s3://..., which it used to refuse as a
missing file, and no longer refuses every registration on a REST catalog
whose warehouse is a name rather than a directory.
A catalog property given as a number or a logical reaches
iceberg-rust in the form it parses: 1e5 as
"100000" rather than "1e+05", and
TRUE as "true". A property passed twice
through ... is refused rather than silently keeping the
last.
An S3:// warehouse is object storage whatever the
case of its scheme, as it already was for forwarding credentials; it
used to be put on the local disk.
The cleartext check recognises http://LOCALHOST and
the rest of 127.0.0.0/8 as loopback, and counts a
header.Authorization, header.Cookie or
header.X-Api-Key property as a credential.
batch_size = 0 is refused rather than read as the
default.
A large append no longer runs under a single timeout. Rows reach the Parquet writer in slices, each its own call, so the five-minute ceiling bounds one slice rather than the whole upload. The files written are byte-identical.
icebergr_snapshots() orders a v1 table’s snapshots
the same way on every call, and its summary JSON lists keys
in sorted rather than hash order.
An interrupt or a timeout while reading a batch is reported as such, rather than as “the Iceberg scan panicked”.
Closing the review findings tracked in #2.
http:// uri, or OAuth2
endpoint, is now an error when any credential property is populated,
rather than putting Authorization: Bearer ... on the wire
in the clear. A loopback address is exempt; set
ICEBERGR_ALLOW_INSECURE_CREDENTIALS=true to override.AWS_* credentials are forwarded only to
object storage. They were previously added to every catalog
regardless of type, so a third-party REST catalog, which controls each
table’s location and can return its own
s3.endpoint, could have the client sign requests to a host
it chose using the user’s keys. They now require
storage = "s3", type = "glue", or an
s3:// warehouse. The explicit ICEBERGR_S3_*
variables are unaffected.icebergr_register_table() gains
confine, defaulting to TRUE. A
metadata file names absolute paths for its location,
manifests and data files, so registering one from an untrusted source
read whatever it nominated, and with the s3 feature
compiled in it could make an outbound request from a nominally offline
memory catalog. The file must now sit inside the catalog’s
warehouse; pass confine = FALSE for the previous
behaviour.user:password@ in a URL, and the value
of a secret query or form parameter such as client_secret
or X-Amz-Signature, are redacted, whether it is written
key=value, key = value or as a JSON field,
"client_secret": "...". The module already promised this
and only delivered it for the key list.print() on a catalog redacts
user:password@ in its uri.block_on parks R’s thread
and Ctrl-C is only checked between evaluations.
ICEBERGR_TIMEOUT_SECONDS changes it; 0
disables it. The ceiling is per request, per record batch or per slice
of an append, so a long scan or a large write is not truncated.ICEBERGR_WORKER_THREADS is clamped to 64. It had no
upper bound, so a typo spawned threads until the allocator gave up.icebergr_scan_plan() drains the plan into its five
output columns as tasks arrive, rather than collecting every
FileScanTask and then walking the collection five times.
One row per task is inherent, but a task carries its schema, its
predicate and its delete-file list, none of which this function returns,
so holding all of them alongside the columns was avoidable.block_on parks R’s own thread and R only
tests its interrupt flag between evaluations, so an interrupt used to do
nothing at all until the call finished. The future is now polled in 200
ms slices, and between slices, back on R’s thread and outside the
runtime, R’s flag is read and turned into an ordinary error. The timeout
above remains the backstop for where Ctrl-C cannot reach, such as a
non-interactive session. Closes #12.unwrap() on
NULL, still prints in full.vendor.tar.xz was compressed with a 128 MiB dictionary, and
BusyBox’s tar, which those systems use, refuses anything
over 64 MiB, reporting only “corrupted data”. CRAN’s musl check recorded
exactly that for 0.1.0. It is now 64 MiB, and
tools/vendor.R checks the archive it writes.rustc 1.88 or newer, down from 1.92,
measured by building the vendored tree with 1.88.0 and running the test
suite on it. CRAN’s r-oldrel-macos-arm64 builder carries 1.91.1, so
0.1.0 could not be installed there. uuid is held below
1.27, the first release to need 1.89.iceberg-rust is 0.10.1, whose Rust code is identical to
0.10.0’s. rustls 0.23.45 fixes RUSTSEC-2026-0285,
tokio 1.53.2 fixes several runtime and timer bugs, and for
the optional s3 and glue features
h2 0.4.19 fixes RUSTSEC-2026-0258. Those two features still
carry quick-xml 0.39.4, affected by RUSTSEC-2026-0194 and
RUSTSEC-2026-0195, because opendal 0.57, the version
iceberg-rust 0.10 requires, depends on it.icebergr_spec_support() now reads the
iceberg-rust and arrow versions it reports out
of Cargo.lock when the package is built, rather than from
strings typed into the source that a dependency bump could leave stale,
and arrow_version carries the patch number.?icebergr_catalog now says that a REST or Glue handle
does not contact the server: iceberg-rust connects on the
first operation that needs it, so a mistyped uri, an
unreachable host or an unset credential all return a handle and fail
later, at icebergr_list_namespaces() or
icebergr_table(). The troubleshooting section of
vignette("catalog-configuration") gained the same note,
since the symptom reads as a fault in those functions.vignette("catalog-configuration") gave
glue.region as the Glue region property, where the Glue
client reads region_name, and showed SigV4 settings for a
REST catalog that iceberg-rust 0.10.0 does not support and
silently ignores. vignette("time-travel") said a rollback
removes a snapshot-log entry (it adds one) and that
iceberg-rust lacks snapshot expiry (it lacks rollback). The
description no longer says that going through DuckDB rules out writes,
which DuckDB has supported for REST catalogs since 1.4.table-format explains
what a table format is as distinct from a file format, and what the Hive
directory convention could not do; pushdown shows the three
levels a filter acts at and how to confirm each with
icebergr_scan_plan() rather than assume it;
time-travel covers snapshot history and the two ways
as_of is easy to misread; writing covers
appends, creation and registration, and why the operations it cannot do
are refused before anything reaches the warehouse; types
covers the R, Arrow and Iceberg correspondence and the five cases that
need care. Each carries a bibliography.icebergr_create_table()’s behaviour is now documented
rather than implied: it reads a schema off the data frame and writes
no rows, leaving a table with no snapshot at all.pkgdown site at https://pursuitofdatascience.github.io/icebergr/.data-raw/logo.R.FEASIBILITY.md, where it was already
duplicated.First release. A deliberately narrow but correct subset of Apache
Iceberg for R, built on iceberg-rust 0.10.0 via
extendr.
icebergr_catalog() for REST catalogs,
in-process memory catalogs (used for local warehouses and
the bundled test fixture), and AWS Glue when the package is compiled
with the optional glue Cargo feature.icebergr_list_namespaces() and
icebergr_list_tables().icebergr_table(),
icebergr_table_exists(), icebergr_schema(),
icebergr_partitions(), icebergr_properties(),
and icebergr_reload() to re-read metadata so a handle can
see a commit made by another session.
icebergr_schema(snapshot_id = ) reports the schema as it
was at an earlier snapshot, which is also what
icebergr_scan() resolves filter and
select against when reading one: Iceberg keeps a schema per
snapshot, so a column another engine has since renamed or dropped is
still nameable as of the snapshot that had it.icebergr_scan() with predicate and projection
pushdown, materialised with icebergr_collect() or
as.data.frame(). Arrow is the interchange layer throughout,
using the Arrow C stream interface, so no serialisation round trip
occurs between Rust and R. long columns come back as
bit64::integer64 at any depth, including inside a
struct, so a value past 2^53 stays exact. A filter on a
decimal column runs with iceberg-rust’s
row-level selection disabled, since in 0.10.0 that stage discards every
row of an ordering comparison against a decimal; file and row group
pruning still apply. case_sensitive = FALSE prefers an
exact match, so a table holding both id and ID
resolves each to itself, and a name matching two columns and neither
exactly is an error rather than a silent choice.icebergr_snapshots() for snapshot history,
and icebergr_scan(snapshot_id = ) or
icebergr_scan(as_of = ) to read an earlier state of a
table. as_of resolves against Iceberg’s snapshot
log, so a snapshot a rollback abandoned, or one that only ever
existed on another branch, is not selected even though the snapshot list
still carries it with a matching timestamp.icebergr_append(), to an
unpartitioned table whose metadata file is named the way Iceberg names
them. Both of those are checked before any data file is written, because
Iceberg only discovers them at the commit, which would leave orphan
Parquet in the warehouse and report the cause in terms of neither the
table nor the file.struct and list columns read
and write, a struct arriving as a data frame column.
Iceberg cannot push a filter or a projection down onto a nested
field, so read the parent column and subset it in R. Nanosecond
timestamps read, write and filter, but an R POSIXct is a
double of seconds, so sub-microsecond precision is lost.icebergr_spec_support() reports the supported Iceberg
spec version and the full supported/unsupported feature matrix
programmatically.Row-level deletes, MERGE/upsert, full schema evolution, partition
evolution, compaction and maintenance operations, and a
dbplyr backend. Several of these are also absent from
iceberg-rust itself; see
icebergr_spec_support() and the README for which is
which.
The package is named icebergr, not iceberg.
Apache Software Foundation trademark policy does not permit third
parties to use Apache marks as the primary branding of their own
products, and a bare iceberg package would also imply that
this is an ASF-governed client, which it is not. See
inst/NOTICE.
Installing from source compiles Apache Iceberg’s Rust implementation,
so a Rust toolchain is required: rustc 1.92 or newer.
iceberg-rust 0.10.0 declares 1.94, but nothing in the tree
uses a feature newer than 1.92, so cargo is passed
--ignore-rust-version and configure gates on
the version the package is actually tested against. The Rust
dependencies a default install compiles are vendored in
src/rust/vendor.tar.xz, so that install never touches the
network; it compiles 264 crates and takes a while the first time. CI
exercises that exact offline path on every commit. The optional
s3 and glue features are the exception: they
draw in about a hundred further crates, which would have tripled the
source tarball, so enabling one of them resolves those from crates.io
and needs network access.