This vignette covers running ggmlR across more than one
GPU. For the single-GPU basics — installation, device
discovery, Vulkan build flags — see vignette("gpu-vulkan").
For data-parallel training specifically, see
vignette("data-parallel-training").
The three modes differ in what gets split across devices, and that choice follows from why one GPU is not enough.
| Mode | What is split | Use when | Entry point |
|---|---|---|---|
| Tensor parallelism (TP) | Individual weight matrices, by rows | One layer is too big, or you want to cut per-layer latency | ggml_vulkan_split_mul_mat() |
| Graph-level TP (meta backend, experimental) | Every weight of a whole graph, by a rule per tensor | As TP, but for a multi-layer model without a gather after each matrix | ggml_tp_forward(),
ggml_meta_backend() |
| Pipeline parallelism (PP) | Whole layers, by depth | The model does not fit on one card | ggml_pp_forward() |
| Data parallelism (DP) | The batch | The model fits, you want throughput | dp_train(), ggml_tp_dp_forward() |
They compose. ggml_tp_dp_forward() and
ggml_pp_dp_forward() run DP replicas on top of TP or PP
groups — for example four GPUs as two replicas of TP=2.
The decisive practical difference is how much data crosses between devices:
That ordering matters more than it looks, because of the transport.
ggmlR provides the tensor layer: primitives that split individual matrices, graph stages and batches across devices. If your goal is to run a whole language model across several GPUs, you almost certainly want llamaR instead, which is built on ggmlR and handles model loading, KV cache, sampling and serving:
# llamaR — whole-LLM inference; the split happens inside the model loader
model <- llama_load_model("model.gguf", n_gpu_layers = -1L,
devices = c("Vulkan0", "Vulkan1"),
split_mode = "layer") # or "tensor"split_mode = "layer" is pipeline-style: whole layers per
card. split_mode = "tensor" is real tensor parallelism
through ggml’s meta backend (section 4) — every weight split across the
cards, each layer run on all of them. split_mode = "row"
asks the backend for a row-split buffer type; Vulkan does not offer one,
so on Vulkan "row" silently behaves like
"layer".
None of these calls into the ggml_vulkan_split_mul_mat()
/ ggml_pp_forward() primitives shown below;
"tensor" shares the meta backend with
ggml_tp_forward(). Reach for the ggmlR primitives when you
are building a custom parallel computation — a model architecture of
your own, a research prototype, a non-LLM workload — rather than serving
an existing GGUF.
llamaR ships a full PP/TP/DP sweep over a real model in
system.file("examples", "bench_pp_tp_dp.sh", package = "llamaR").
Copying a tensor from GPU 0 to GPU 1 needs a path. ggmlR implements
three, selected by the transport argument:
"host-staging" (default) — device →
host RAM → device. Portable and always correct."opaque-fd" — zero-copy via
VK_KHR_external_memory_fd."device-group" — one logical device spanning several
cards (NVLink / LDA).The default is not a conservative placeholder; on the hardware we tested it is the only path that works. Measured on a 4×P100 server:
"opaque-fd" shares memory correctly in loopback (same
device) but silently transfers zeros across devices — a
hardware/driver limitation on NVIDIA, not a bug in the import path."device-group" enumerates five groups, every one of
them single-device: no NVLink/LDA aggregation is offered at all.Verify your own hardware before assuming a zero-copy path exists:
# Loopback sanity check: does the transport work at all on device 0?
r <- ggml_vulkan_p2p_selftest(0L, 0L)
cat(r$report)
# The real question: does a cross-device copy carry the bytes?
if (ggml_vulkan_device_count() >= 2) {
r <- ggml_vulkan_p2p_selftest(0L, 1L, transport = "opaque-fd")
cat(r$report) # "verified" vs. a mismatch tells you immediately
}
# Are there any multi-device (LDA) groups?
ggml_vulkan_device_groups()Because host-staging routes through host RAM, TP
is the mode that suffers most and PP the least. If your model
does not fit on one card, prefer PP.
ggml_vulkan_split_mul_mat() splits the weight matrix
W by rows across devices, computes each slice on its own
GPU, and gathers the result. It is a standalone C routine, deliberately
outside the ggml graph.
set.seed(1)
W <- matrix(rnorm(2048 * 64), nrow = 2048) # [N x K] weights
X <- matrix(rnorm(4 * 64), nrow = 4) # [M x K] activations
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2)
# Contract: identical to the single-device result, up to f32 rounding.
max(abs(Y - X %*% t(W))) # ~3.8e-6 on 2 devicesPick a subset of GPUs rather than “the first N” with
device_ids, and give uneven cards uneven shares with
weights:
Y <- ggml_vulkan_split_mul_mat(W, X, device_ids = c(0L, 1L))
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2, weights = c(0.7, 0.3))
# Inspect the row ranges the split math will use (0-based, half-open):
ggml_vulkan_split_row_ranges(nrows = 2048, n_devices = 2)ggml_vulkan_split_buffer_type() exposes the same
row-split as a ggml buffer type, for callers that allocate weights
themselves.
ggml_vulkan_split_mul_mat() splits one matrix and
gathers the result right away, so a two-layer model built from it pays
two gathers. ggml’s meta backend (upstream since ggml
0.27; experimental here) splits a whole graph instead. It wraps several
devices into one backend and:
The classic case is the Megatron MLP: split the first weight by its output features and the second by its input features. The hidden activation then stays split between the two layers, and the only reduction is after the second one.
Shapes follow ggml: an R array’s dim() is the tensor’s
ne, fastest dimension first. A weight with
dim = c(n_in, n_out) and an input with
dim = c(n_in, batch) give
ggml_mul_mat(ctx, w, input) of shape
c(n_out, batch) — t(W) %*% x in R terms. The
split rules name a ggml axis: "rows" is ne[1]
(here: output features), "cols" is ne[0]
(input features).
n_in <- 512L; n_hid <- 2048L; n_out <- 512L; batch <- 16L
W1 <- array(rnorm(n_in * n_hid), c(n_in, n_hid))
W2 <- array(rnorm(n_hid * n_out), c(n_hid, n_out))
X <- array(rnorm(n_in * batch), c(n_in, batch))
mlp <- function(ctx, input, w) # w: the weight tensors, by name
ggml_mul_mat(ctx, w$w2, ggml_relu(ctx, ggml_mul_mat(ctx, w$w1, input)))
Y <- ggml_tp_forward(
weights = list(w1 = W1, w2 = W2), # names are what `split` refers to
forward = mlp,
x = X, # the input tensor is named "input"
devices = c(0L, 1L),
split = list(w1 = "rows", w2 = "cols"))
max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0)))Rules are "mirror" (the default: a full copy on each
device), "rows", "cols", "axis2",
"axis3", or
list(axis = "rows", weights = c(0.7, 0.3)) for uneven
cards; split may also be a function(name, ne).
A rule that cannot apply — the axis is shorter than the number of
devices — falls back to "mirror" for that tensor with a
warning.
devices takes Vulkan ids, device names
("Vulkan0", "CPU") or device pointers, and may
list the same device twice. That is how to try the API on one GPU, or
with no GPU at all:
Y <- ggml_tp_forward(list(w1 = W1, w2 = W2), mlp, X,
devices = c("CPU", "CPU"),
split = list(w1 = "rows", w2 = "cols"))For a graph you drive yourself — a scheduler, repeated computes, several outputs — use the backend directly:
meta <- ggml_meta_backend(c(0L, 1L), split = list(w1 = "rows", w2 = "cols"))
sched <- ggml_backend_sched_new(list(meta, ggml_backend_cpu_init()))
# Weights (and inputs) in a context allocated on `meta`:
# buf <- ggml_backend_alloc_ctx_tensors(ctx_w, meta)
# The graph in a SEPARATE context, allocated through the scheduler:
# ggml_backend_sched_alloc_graph(sched, gf); ggml_backend_sched_graph_compute(sched, gf)Keep the two contexts apart. The meta backend asks for a rule for
every tensor in a buffer allocated on it; an intermediate result that
ended up there would be laid out by its name rather than by the op that
produced it. ggml_backend_meta_device() is the raw API
underneath, with a callback that returns numeric axis codes
(GGML_SPLIT_AXIS_*).
One more rule for a graph you drive yourself: a matrix product whose
inputs are both split along the reduced axis ("cols" on the
weight and on its input) leaves per-device partial sums. The meta
backend reduces them before their first consumer, but never after the
graph’s last node, and cannot read them back — so do not read such a
tensor, and do not end a graph with one. ggml_tp_forward()
appends a trivial consumer for you.
What it costs. Vulkan has no device-to-device
all-reduce, so the meta backend’s reductions copy through host RAM — the
same host-staging transport as the rest of this vignette (section 2).
Graph-level TP moves less data than
ggml_vulkan_split_mul_mat() layer by layer, but it still
pays off only when the model does not fit on one card.
Status. Forward graphs only:
ggml_fit(), the autograd engine and ONNX models do not run
on it. Not yet measured on real multi-GPU hardware. llamaR reaches the
same backend through split_mode = "tensor".
PP assigns layers to devices: GPU 0 runs stages 1–16, GPU 1
runs 17–32, and exactly one activation tensor crosses between them. Each
stage is described by a list with its device, its
in_shape, and a build function that constructs
the sub-graph and returns a set_weights closure.
The one non-obvious rule: weights are set after
allocation, which is why build returns
set_weights instead of filling tensors immediately.
K <- 64L; M <- 8L
W1 <- matrix(rnorm(K * K), nrow = K)
W2 <- matrix(rnorm(K * K), nrow = K)
X <- matrix(rnorm(K * M), nrow = K) # ggml ne = c(K, M): column m is sample m
make_stage <- function(dev, Wt, relu) {
list(
device = dev,
in_shape = c(K, M),
build = function(ctx, input) {
w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K)
z <- ggml_mul_mat(ctx, w, input) # == t(Wt) %*% input in R terms
list(
output = if (relu) ggml_relu(ctx, z) else z,
set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt))
)
})
}
stages <- list(make_stage(0L, W1, relu = TRUE),
make_stage(1L, W2, relu = FALSE))
y <- ggml_pp_forward(stages, x = as.numeric(X), out_shape = c(K, M))
Y <- matrix(y, nrow = K, ncol = M)
max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0))) # ~1.8e-5With four GPUs, two replicas of TP=2 usually beat one TP=4 group: DP adds no cross-device traffic, so the hybrid hides the weakness of host-staging.
# Replica A = {GPU0, GPU1}, replica B = {GPU2, GPU3}.
# Each replica takes half the batch; no traffic crosses between replicas.
Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0L, 1L), c(2L, 3L)))
# The PP equivalent takes a factory: make_stages(devices, m_shard) -> stage list
Y <- ggml_pp_dp_forward(make_stages, x, replicas = list(c(0L, 1L), c(2L, 3L)),
out_ncol = M)For multi-GPU training rather than inference, use
dp_train() — see
vignette("data-parallel-training").
--enable-hard-exit
flag)A multi-GPU script can segfault after it has printed its results. The cause is a race at process exit: the Vulkan loader and the driver’s ICD libraries get unmapped while driver worker threads are still winding down, so late destructors call into unmapped memory. The computed results are already out; the crash is harmless but noisy, and it turns a successful run into a non-zero exit code.
Call ggml_vulkan_shutdown() to tear Vulkan down
while the loader is still mapped. It is idempotent, safe
mid-session, and Vulkan transparently re-initializes on the next
operation:
That narrows the race but does not close it: no R exit hook runs
before R unmaps the loader. For a guaranteed-clean exit, make
hard = TRUE the last statement of a
standalone script — after teardown it calls _exit(status),
terminating immediately without running exit handlers or unmapping
libraries, so there is no teardown phase left to crash.
This path is compiled out by default. CRAN
Repository Policy forbids a package from terminating the user’s R
session, so the released package must not link _exit(). In
a default build, hard = TRUE performs the normal teardown
and emits a warning() — it is never silently ignored.
Compile it in with:
On Windows R ignores configure.args; set
Sys.setenv(GGML_VK_HARD_EXIT = "1") before installing from
source. Check what the current build has:
Two caveats. hard = TRUE bypasses R’s shutdown entirely:
no .RData, no on.exit() handlers, no flushed
connections beyond what ggmlR flushes itself. Use it only as the final
line of a script, never mid-session, never inside a package. And
although the race is observed in multi-GPU scripts — more devices mean
more driver threads and a wider window — nothing in the mechanism is
specific to multiple GPUs; single-GPU runs simply have not been seen to
trip it.
Runnable demos ship with the package, under
system.file("examples", package = "ggmlR"):
multi_gpu_example.R — device discovery and per-device
worktp_dp_hybrid.R — TP×DP hybrid on four GPUspp_pipeline.R — two-stage pipeline parallelismtp_p2p_diagnose.R — transport diagnostics; drop
hard = TRUE to observe the exit-time crash described
abovedp_train_demo.R — data-parallel trainingThe test suite documents the numeric contracts, and skips gracefully when fewer than two GPUs are present:
tests/testthat/test-vulkan-tensor-parallel.R — split
math, split_mul_mat, buffer-type factory, TP×DP and PP
forwardstests/testthat/test-meta-backend-compute.R — meta
backend: compute against the plain CPU on 2×CPU and 2× the same GPU,
ggml_meta_backend(), ggml_tp_forward()tests/testthat/test-vulkan.R,
test-vulkan-caps.R — backend and capabilities