First release of the package after its archival on CRAN, and the first to include the spatially-clustered STEM (SC-STEM) model family. The classical STEM workflow of version 1.0 is unchanged in its statistical content; what changes is its numerical robustness, the whole SC-STEM layer, and the packaging.
A (it was K):
STEM_Model(..., A = ), skeleton$A,
STEM_Kriging(..., A.newlocations = ). Scripts written for
version 1.0 must replace K = by A =.K, with index
k = 1, ..., K, as in the paper and as in the SCDA package:
STEM_Fit(K = ), SCSTEM_Estimation(K = ),
SCSTEM_Infocrit(K_grid = ),
SCSTEM_Select()$K_selected, and the columns K,
K_eff of the grid. The transition matrix stays
G.df:
info_crit["df"] (it was "k") and the column
df of the grid (it was npar).mink and maxk of
SCSTEM_Infocrit() are removed.k, which R would otherwise
complete silently to knn.STEM_Fit() is a single entry point for the pooled and
the clustered model, with or without a penalty on the regression
coefficients. K = 1 fits the pooled STEM model and
K > 1 the spatially-clustered one;
lambda = 0, the default, gives the ordinary maximum
likelihood estimator and reproduces STEM_Estimation()
exactly.alpha and lambda follow the
glmnet parameterization: alpha = 0 is ridge,
alpha = 1 the lasso, anything between the elastic net. Only
the regression coefficients are penalized; the intercept is excluded by
default. In the clustered model the penalty acts within each regime,
with one (alpha, lambda) shared by all of them.lambda is dimensionless. Its L1 part is measured
against the largest partial gradient, so lambda in
(0, 1] traverses the whole lasso path and means the same
thing at any error variance and in any regime; its L2 part is left
alone, being already unit-free. lambda_scale = "absolute"
recovers the raw convention.lambda_by = "size" spreads the penalty over the regimes
in proportion to 1/n_g instead of equally, shrinking the
smaller regimes more. It remains one hyperparameter.latent = FALSE switches the latent process off and
spatial = FALSE replaces the spatial correlation by the
identity. Together with regularization = 0 they reduce the
model exactly to penalized linear regression, and with
K > 1 to clusterwise penalized regression.lambda means the same thing for every
covariate and every regime.dev/regularization/stem-elastic-net.tex in the source
repository.SCSTEM_Estimation() fits an SC-STEM model: the
monitoring locations are partitioned into K latent spatial
regimes and a separate STEM model is estimated within each of them, so
that regression coefficients, variance components and latent temporal
dynamics are all cluster-specific. Labels and parameters are estimated
jointly by maximizing a Potts-penalized log-likelihood.SCSTEM_Infocrit() now fits a full (K, phi)
grid and returns exact log-likelihoods, AIC, BIC and KIC, an
admissibility flag, the cluster sizes and the estimated partitions.SCSTEM_Select() implements a two-step rule for choosing
the hyperparameters: (S1) the modal BIC-minimizing
K inside a moderate-penalty band, ties resolved towards the
smaller K; (S2) the phi with
the smallest BIC at that K over the whole grid, ties
resolved towards the smaller phi. Only admissible
configurations enter the rule. The pooled K = 1 model
competes in (S1) and is returned when no partition improves on it.SCSTEM_Bootstrap() is now a
refit-with-clustering parametric bootstrap: data are
generated cluster by cluster from the fitted model and the
entire procedure, endogenous partitioning included, is
re-estimated on every draw, so that the uncertainty of the partition is
propagated. It replaces the previous bootstrap, which conditioned on the
estimated partition and therefore understated the uncertainty. The
refits use the same alpha, lambda,
penalize, lambda_scale,
lambda_by, latent and spatial as
the original fit, so a fit with a ridge on the coefficients is resampled
with it. Its draws can be computed in parallel: cores
(default 1) shares them among that many R processes. Every draw runs
under its own seed, drawn from seed, so the draws are the
same whatever the number of processes.SCSTEM_BootInference() aligns every refit onto the
original clusters by the majority rule and returns bootstrap standard
errors; normal, basic, percentile and bias-corrected confidence
intervals; pairwise percentile tests for the differences between
clusters; the co-clustering matrix; and the ARI of each refit against
the original partition.SCSTEM_CV() is an optional validation of predictive
accuracy, separate from the choice of (K, phi): blocked
cross-validation leaving out locations ("LKLO"), time
blocks ("LKTO"), both ("LKLHTO") or random
cells, of one fit or of a named list of fits on the same data (the
pooled model, the selected one, models with different covariates), all
refitted on the same folds and ranked by root mean squared error. A
removed location inherits the regime of its nearest retained location
and is kriged within it.SCSTEM_Estimation,
SCSTEM_Infocrit, SCSTEM_Select,
SCSTEM_Bootstrap, SCSTEM_BootInference and
SCSTEM_CV, each with a print() method.NA), so the panel
may be unbalanced. The Kalman filter restricts each time point to the
locations observed there (Durbin and Koopman 2012), and the M-step
completes the sufficient statistics with the conditional moments of the
missing values, which the spatial covariance couples to the observed
ones. Kriging and both bootstraps follow the same rule. Every location
must keep at least one observation; covariates and coordinates cannot be
missing.STEM_Complete() and SCSTEM_Complete()
return the conditional expectation of the response given the observed
data: the observed value where there is one, the model-based prediction
where there is not.STEM_Signal() and SCSTEM_Signal() return
the fitted values, the regression surface plus the smoothed latent
process, without the measurement error.label_update = "simultaneous".Q = sum_i l_{i,k_i} + phi * c * #{concordant neighbor pairs}
is now computed explicitly and traced along the iterations
(obj_trace). Convergence is declared on label stability, on
the improvement of Q, on cycle detection (the best visited
partition is returned) or at max_iter.enforce_min_size = TRUE). Within-cluster homogeneity is
what the assignment step seeks, so the cluster-wise variance components
shrink and, left unconstrained, the cluster with the smallest residual
variance attracts every location. On the pm10 example the
unconstrained sweep collapses to a single cluster even at
phi = 0. A location may now leave its cluster only if that
cluster stays at or above min_cluster_size.swap_pass = TRUE). Since the constraint above freezes any
location sitting in a minimum-size cluster, each sweep is followed by a
pass that exchanges the labels of two locations whenever this strictly
increases Q. Swaps leave cluster sizes unchanged, so
feasibility and monotonicity both hold. An exact pruning bound keeps the
scan affordable.knn graph is symmetrized: the
Potts penalty is defined on an undirected graph, whereas
spdep::knearneigh() returns an asymmetric one.knn graph is built with the same metric as
the covariance. The nearest neighbors used to be taken on raw
coordinates, so on longitude and latitude they were the neighbors of a
planar metric in which a degree of longitude and a degree of latitude
count the same; at 45 degrees of latitude the first is about 78 km and
the second about 111 km, so the graph preferred north-south neighbors
while Sigma_e was measured on the sphere. The
distance argument now governs both.phi_scale argument. Each location contributes
T observations to the likelihood, so a penalty calibrated
for cross-sectional models is not transferable. The default
"auto" normalizes the penalty by the median spread of the
location-wise log-likelihood contributions;
"per-observation" and "raw" are also
available. The penalty actually applied is reported in
phi_effective.init_method = "departures"): the mean of
its residual from the pooled signal, the slopes of that residual on its
covariates, and the lag-one autocorrelation and log-variance of what the
slopes leave. The covariate means, the previous default, carry no
information on the regimes when the covariates are exogenous to them;
they remain available as init_method = "kmeans".
SCSTEM_Infocrit() computes the departures once, from its
fit at K = 1.phi_penalty > 0 starts from the
solution of the unpenalized fit at the same K
(SCSTEM_Estimation() fits phi = 0 first;
SCSTEM_Infocrit() passes on the partition of the
phi = 0 fit it already has). A penalty that is strong from
the first sweep froze whatever partition it was given, so a start
unrelated to the regimes stayed where it was. The automatic scale,
computed at the first sweep, is then the one of the unpenalized fit and
no longer depends on the initialization. Measured there it is several
times larger than before, so the defaults are recalibrated:
phi_penalty = 0.05,
phi_grid = c(0, 0.025, 0.05, 0.1, 0.2, 0.5, 1) and the band
c(0.025, 0.2) of SCSTEM_Select(); the strong
values 0.5 and 1 lie outside the band and compete only in the choice of
phi.The score used to allocate locations to clusters before 2.0.0 contained four defects, all fixed:
m0 instead of Sigmaeta, which flips its sign
whenever m0 < 0;sigma2omega and sigma2eps were taken from
the pooled fit rather than from the candidate cluster, which made the
corresponding term constant across clusters and turned the
nugget_var switch into a no-op;y_{t-1}
instead of the contemporaneous K_i y_t of the measurement
equation;phi without a comparable scale.The contribution is now an explicit conditional pseudo-likelihood, documented as such, and used only to rank clusters: coefficients, variance components and information criteria all come from the exact cluster-wise likelihoods.
K_eff * (ncov + 3 + 3p) free
parameters and, in the BIC, the number of observed values of the
response as the sample size (d * T when the panel is
complete). The previous implementation used the number of
clusters as the number of parameters and the number of locations as
the sample size.STEM_Estimation()estimates$loglik is now the log-likelihood. Every
earlier version, 1.0 included, returned four times it: the EM loop
stored -2 times the log-likelihood and the value returned multiplied
that by -2 again. The iterations were not affected (the stopping rule
used differences of the stored value, halved), but everything computed
from the returned value was: the AIC, BIC and KIC of SC-STEM weighted
the likelihood four times against the penalty, which favored more
regimes. On the 951 replications of an earlier run of the simulation
study, the BIC on the true log-likelihood selects fewer regimes in 40,
mostly in the weakly separated scenarios. The log-likelihood, the
smoothed states and the maximum are now tested against dense
computations on the full covariance of the data
(tests/testthat/test-exact.R).STEM_control(): the computational settings of every
estimation function in one object, in the spirit of
optim(control = ). All the estimation functions take
control; options(Stem.control = list(...))
sets them for a whole session.STEM_control(algorithm = ): the EM algorithm; the ECME
algorithm (Liu and Rubin 1994), in which the regression coefficients and
m0 are updated on the observed likelihood by a generalized
least-squares step computed with the Kalman filter; and both accelerated
by SQUAREM (Varadhan and Roland 2008). They reach the same maximum. The
default is "SQUAREM": in the simulation study it fitted the
regimes in about a third of the time of the EM algorithm, closer to the
maximum, and ran the whole grid of SCSTEM_Infocrit() in
about half the time, on small and large cells alike, with the same
selections. The algorithm applies to the pooled fit and to the final
refits; inside the alternation of SC-STEM, whose fits stop after a few
iterations, the iterations are not accelerated. With a ridge on the
coefficients (lambda > 0, alpha = 0) the
safeguard of SQUAREM compares the penalized log-likelihood, the scale of
the penalty measured at each point: the accelerated iterations reach the
fixed point of the plain ones, in about half the time. Lasso and
elastic-net fits (alpha > 0) run the plain
iterations.regularization defaults to 0 (it was 0.01) in
STEM_Estimation(), SCSTEM_Estimation() and
STEM_Bootstrap(). The constant was added to every matrix
the EM algorithm inverts but not to the filter that computes the
likelihood, so the iterations converged to a point that is not the
maximum: the variance parameters were off by 3 to 13% on the regimes of
the simulation study, and the variance of the latent innovations of a
pooled fit by 84%, whatever the tolerance.convergence.par$theta.bound of
STEM_Estimation() and theta_bound of
SCSTEM_Estimation() report a range at a limit. The
Newton-Raphson step of the spatial parameters accepts a step only if it
does not worsen the objective, and allows one grid search where it used
to allow 30; nr_hess_maxit is removed from
STEM_control().STEM_Estimation() refer to the same parameters: the
log-likelihood and the smoothed states used to be those of the
last-but-one iteration.kalman(), R/kalman.R), the
E-step (R/estep.R), the conditional updates of the M-step
(R/mstep-updates.R), the M-steps of the EM and ECME
algorithms (R/mstep-em.R, R/mstep-ecme.R),
SQUAREM (R/squarem.R) and the wrapper that runs the
algorithm chosen (R/em-fit.R). kalman() used
to run a whole EM iteration; it now runs the filter and the smoothers,
which the E-step and the fit at the estimates use. With
algorithm = "EM" and regularization = 0.01 the
iterations reproduce those of the previous code to 1e-13. The smoother
gain of the initial state now uses the transpose of G,
which matters only for a non-symmetric G.1e-3 on both by default. With em_stop = "any"
(the default) the algorithm stops when either criterion is met, with
em_stop = "all" only when both are; in either case after
em_maxit iterations (default 500). The previous rule
stopped when the relative change of the whole parameter vector and the
relative change of the log-likelihood fell below precision.
That vector contained the fixed loadings, so the rule loosened with the
number of locations, and with log-likelihoods of the order of
1e4 a relative tolerance stopped the algorithm while
several units could still be gained.SCSTEM_Infocrit()).K = 1, so that the log-likelihoods compared across
K are computed to the same accuracy; the fit reports in
em_converged whether the EM algorithm of each regime met
its stopping rule.SCSTEM_Infocrit() shares the refits
across its grid, and a penalized fit that ends at the partition of the
unpenalized fit it starts from reuses that refit. The refit depends on
the partition and the settings, not on phi_penalty; two
refits of one partition from different starts stopped at different
points of the likelihood, and that difference decided between penalties
with the same partition. The fit reports refit_reused; the
argument refit_cache carries the shared refits and is not a
setting to tune.precision,
max.iter, precision_full_dataset,
max_iter, abs_tol, rel_tol) still
work, as overrides of control.control (nr_maxit).These were pre-existing defects, harmless for a single pooled fit but
fatal for a clusterwise algorithm, which refits the model on hundreds of
different subsets of locations. Both aborted the fit with
missing value where TRUE/FALSE needed:
kalman(): the Newton-Raphson step is validated before
use; a singular or non-finite Hessian, or a non-finite step, now exits
the loop keeping the last valid iterate instead of propagating
NaN into the convergence test. The relative criterion has a
floor on its denominator, the Hessian condition is evaluated through
isTRUE(), and the fallback grid search is guarded against
an underflowed scale that produced log(0).STEM_Estimation(): a non-finite EM iterate is discarded
with a warning and the last valid one is returned; both relative
convergence criteria have a floor on the denominator and are wrapped in
isTRUE().On the pm10 example these fixes take the parametric
bootstrap from 4 usable draws out of 12 to 12 out of 12.
STEM_Simulation() called mvrnorm()
unqualified, relying on a NAMESPACE import; it now calls
MASS::mvrnorm().STEM_Kriging(): the checks on the number of columns of
coord.newlocations and covariates.newlocations
never fired; they do now. The documented predictor had the wrong sign on
the latent term of the residual (the code was right), and the
documentation now states what se.pred measures: the
standard error of the spatially structured component, without the nugget
and the uncertainty of the smoothed state.d x d
predictive covariance. Since the measurement covariance does not depend
on time and the state contribution has rank p, the Woodbury
identity and the matrix determinant lemma reduce each step to
p x p algebra against a Cholesky factor computed once per
pass, so a forward pass costs O(d^3 + T d^2) instead of
O(T d^3). On 200 stations over 365 days an EM iteration is
about ten times faster; the estimates are unchanged to floating point,
the two forms being algebraic identities.d
observations is of order exp(-d), so past roughly 300
locations it underflowed to zero and the fit aborted with a non-finite
log-likelihood. Networks of that size can now be fitted at all.T matrices of size
d x d at once. The derivative helpers receive the inverse
of the scaled covariance from the caller instead of recomputing it up to
seven times each, and every trace of a matrix product is evaluated
without forming the product.log b is carried as the multiplier of the identity that it
is rather than as a dense matrix, the products against the inverse
covariance are formed once instead of three times each, and the log
determinant and the inverse now come from a single Cholesky
factorization. The d x d matrix products per inner
iteration go from twelve to five.mvtnorm moves from Imports to
Suggests: no function in the package uses it any more,
though the test suite still does, as an independent reference
implementation of the likelihood.Addresses the review comments received on the previous submission:
DESCRIPTION now carry
<doi:...>, <https:...> and ISBN
links in the required format;T and F have been replaced by
TRUE and FALSE throughout the code and the
documentation;print()/cat() diagnostics have been
replaced by message() behind a verbose = FALSE
argument, threaded through kalman(),
STEM_Estimation(), STEM_Bootstrap() and the
SC-STEM routines. The only remaining cat() calls are inside
print() methods, where they belong;.GlobalEnv. Where a
seed is needed for reproducibility, the RNG stream is saved and restored
on exit.Further packaging work required before submission:
pm10 moved from a 250 KB source file duplicated in
R/ and data/ to a proper
data/pm10.rda (25 KB), with R/pm10.R reduced
to documentation;init_partition argument;dplyr and sf were likewise dropped, as
nothing in the package needs them any more; every external call is
written as package::function() and NAMESPACE
no longer carries importFrom directives;URL and BugReports fields added, along
with NEWS.md and a testthat suite.STEM_Model() to the
selection rule and the bootstrap), a map of the package, computational
notes, and numerical experiments on the estimation algorithms.inst/extdata/,
showing how the STEM and SC-STEM functions call one another, and
reachable from R with
system.file("extdata", package = "Stem").system.file("extdata", "Stem-computational-supplement.pdf", package = "Stem").
The source repository adds documentation pages on the models, the
computations and the functions (docs/), and the data and
code that reproduce the paper
(papers/sc-stem-local-leader).