R package Stem · technical note
Where each equation of the STEM model lives in the package: the covariance structure, the EM algorithm, the Newton-Raphson step for the spatial parameters, the kriging predictor and the parametric bootstrap, mapped onto the functions that implement them. Written as a reading guide for anyone who wants to follow the source with the papers open.
They build on one another; only one is the direct reference
| Work | What it contributes | Relation to the package |
|---|---|---|
| Fassò, Cameletti, Nicolis (2007) Environmetrics 18, 245–264 |
The geostatistical dynamical calibration (GDC) model, the most general of the three: instrument calibration through an additive bias A(t) and a multiplicative bias B(t), loadings obtained from empirical orthogonal functions, and a network mixing gravimetric and TEOM monitors. |
The package implements the special case without calibration bias, with K playing the role of the EOF loading matrix. |
| Fassò, Cameletti (2007) GRASPA Technical Report n. 27 |
The three-stage hierarchical model with covariates, the EM algorithm with closed-form M-steps, and the spatio-temporal parametric bootstrap. | The direct reference. It announces this package by name: “Software availability: R package Stem — Developer: Michela Cameletti”. |
| Fassò, Cameletti (2010) Simulation 86, 139–153 |
The fullest published statement: the same model and EM, plus the kriging predictor, the bootstrap and a sensitivity analysis of the model components. | Its Equations (12)–(21) are the ones the estimation code follows, and the equation numbers used below refer to it unless marked TR. |
Three stages, collapsed into a state-space form
Concentrations are observed at n locations over T days. The first stage separates the measurement error from the underlying level; the second gives that level a regression part, a latent temporal part and a spatial part; the third makes the latent part Markovian.
Substituting (2) into (1) gives the two-stage state-space form actually estimated,
with three mutually independent, temporally white error components: the instrumental error epsilon, with variance sigma2eps; the innovation eta, with variance-covariance Sigma_eta; and the small-scale spatial component omega, white in time but spatially correlated,
The parameter set is
with C_0 held fixed. Setting p = 1 and K = 1, the usual choice, gives a single latent process shared by the whole network.
Equation (8), and Equation (7) of the technical report
The two error components of e_t combine into a single variance-covariance matrix, written as a common variance times a scaled spatial covariance function:
Two things follow, and both are visible in the code.
First, sigma2eps is the nugget of the spatial process in the geostatistical sense. Second, the technical report notes that “for positive definiteness reasons, it is preferable to estimate log(gamma) instead of sigma2eps”, which is why the internal parameterization carries logb = log(gamma) rather than the variance itself. The package applies the same reasoning to the range parameter and works with logtheta = log(theta).
Sigmastar.exp(d, logb, logtheta, dist) returns Gamma as diag(exp(logb), d) + exp(-exp(logtheta) * dist). The diagonal is exp(logb) + 1 = 1 + gamma and the off-diagonal is C_theta(h), matching the definition exactly. Everywhere the full Sigma_e is needed, the code multiplies this by sigma2omega.
A useful consequence for the clustered layer: the marginal variance of a single observation is Sigma_e[i,i] = sigma2omega * (1 + gamma) = sigma2omega + sigma2eps, which is the variance used by the per-location contribution in SCSTEM_Estimation.
Equation (9)
The likelihood is written through the prediction-error decomposition of the Kalman filter:
Each term is the log density of the one-step-ahead predictive distribution of Z_t, whose mean adds the regression part to the state contribution and whose variance adds the state prediction uncertainty to the observation covariance.
filterstep computes a = G m and R = G C G' + W, so that F'a + X beta is mu_t and Q = F'RF + V is Omega_t. It never forms Q: since V is constant in t and F'RF has rank p, the Woodbury identity and the matrix determinant lemma give Q^-1 and log|Q| from a Cholesky factor of V that filtering computes once per pass, so the step costs O(d^2) rather than O(d^3). The log density is evaluated in closed form rather than as the logarithm of a density, which on a network of a few hundred locations would underflow to -Inf; filtering accumulates the sum over t.
Equations (11)–(19)
Direct maximization of (9) is awkward, so estimation goes through EM on the complete-data likelihood, treating the latent process as the missing data. The E-step assembles the objective from the Kalman smoother output y_t^T, P_t^T and P_{t,t-1}^T; the M-step maximizes it in two blocks. Most parameters have closed forms; only the two spatial-covariance parameters need a numerical step.
| Eq. | Update | In the code |
|---|---|---|
| (14) | beta = [SUM_t X_t' Sigma_e^-1 X_t]^-1 {SUM_t X_t' Sigma_e^-1 (Z_t - K y_t^T)} — a generalized least squares step on the data net of the latent process. |
kalman: MM is the left factor, v the right one, beta_j the solution. |
| (15) | sigma2omega <- sigma2omega / (T n) * tr[Sigma_e^-1 W] |
sum(diag(Gamma^-1 %*% BB)) / (n*d). The sigma2omega factor cancels against the one inside Sigma_e^-1, which is why the code inverts the scaled Gamma rather than Sigma_e. Note the local naming: in the sources n counts time points and d counts locations, the opposite of the papers. |
| (16) | G = S10 S00^-1 |
G_j. With flag.Gdiag = TRUE only the diagonal is retained. |
| (17) | Sigma_eta = S11 - S10 S00^-1 S10' |
Sigmaeta_j. The S matrices in the code are unnormalized sums, and the division by the number of time points restores the identity. |
| (18) | mu_0 = y_0^T, the smoothed state at time zero. |
m0_j, from the backward recursion. |
No closed form exists for {theta, sigma2eps}, so they are obtained by minimizing Qtilde numerically, holding the other parameters at their updated values:
where Delta and H are the gradient and Hessian of Qtilde with respect to (logtheta, log gamma). Differentiating (12) gives, for a generic parameter of the scaled covariance,
d1_Q, d2_Q and d12_Q implement those three expressions; d1_Sigmastar_logtheta.exp, d2_Sigmastar_logtheta.exp, d1_Sigmastar_logb.exp and d2_Sigmastar_logb.exp supply the derivatives of Gamma.
Two structural facts make these simple. The range parameter affects only the off-diagonal, so the derivative with respect to logtheta has a zero diagonal; and the two parameters enter Gamma additively, one on the diagonal and one off it, so the mixed second derivative of Gamma vanishes and d12_Q keeps only the two trace terms that survive.
The outer loop stops when two relative criteria are met jointly: one on the parameter vector, one on the log-likelihood. Relative rather than absolute criteria, so that parameters on very different scales are treated alike.
STEM_Estimation returns them as convergence.par$conv.par and convergence.par$conv.log, together with the number of EM iterations and the number of inner Newton-Raphson iterations per EM step.
Equations (20)–(21)
Mapping the process at an unobserved location s_0 follows from the joint Gaussian distribution of the observed vector and the unobserved value, conditionally on the latent state. The result is the simple kriging predictor and its prediction error variance:
Because the parameter set and the latent process are not known, the plug-in approach substitutes the ML estimates and the smoother output. The resulting variance therefore ignores the uncertainty of the estimates themselves, which is what the bootstrap of the next section is for.
STEM_Kriging builds the distance matrix between the new locations and the network, then calls spatial.pred with Sigma11 = sigma2omega * Gamma, Sigma12 = sigma2omega * exp(-theta * h) and Sigma22 = sigma2omega * I. It returns the predictions and their standard errors at the requested time point.
Section 4 of the 2010 paper
The EM algorithm returns point estimates but no standard errors, and the asymptotic distribution of the estimates is not a good guide here: the papers note that the estimate distributions are markedly asymmetric. The bootstrap resamples directly from the fitted model.
Each replicate draws a new latent path and a new dataset from Equations (6) and (7) with the parameter set replaced by its ML estimate, starting from an initial state drawn from its own fitted distribution; the whole EM estimation, and if wanted the spatial prediction, is then repeated on the simulated data. Across replicates this gives standard errors, percentile confidence intervals free of normality assumptions, and the full empirical distribution of each parameter.
STEM_Simulation performs the draw: it builds Vmat = sigma2omega * Gamma, draws y_0 from its initial distribution, then propagates the state and generates the observations period by period. STEM_Bootstrap repeats simulate, then re-estimate over the requested number of replicates and returns the list of fits.
The spatially-clustered counterpart, SCSTEM_Bootstrap, uses the same generator regime by regime and additionally re-runs the clustering on every draw, so that the uncertainty of the estimated partition is propagated as well.
Reading the sources alongside the papers
STEM_Model assembles the data and the starting values; the validation of dimensions lives in STEM_Skeleton and STEM_Data.STEM_Estimation is the outer EM loop and the convergence test; kalman is one full E-step plus M-step.kalman: filtering and smoothing for the recursions, Q_function_addendo1/2/3 for the three pieces of the objective, the d1_*/d2_* family for the Newton-Raphson step.vignette("function-map", package = "Stem") shows how these call one another.?SCSTEM_Estimation and vignette("SCSTEM", package = "Stem").A note on notation, worth keeping in mind when reading the sources: the papers use n for the number of locations and T for the number of time points, whereas the code uses n for time points and d for locations. The formulas are the same; only the letters differ.