Multi-GPU: Tensor, Pipeline and Hybrid Parallelism

This vignette covers running ggmlR across more than one GPU. For the single-GPU basics — installation, device discovery, Vulkan build flags — see vignette("gpu-vulkan"). For data-parallel training specifically, see vignette("data-parallel-training").

library(ggmlR)

1. Which mode do you need?

The three modes differ in what gets split across devices, and that choice follows from why one GPU is not enough.

Mode What is split Use when Entry point
Tensor parallelism (TP) Individual weight matrices, by rows One layer is too big, or you want to cut per-layer latency ggml_vulkan_split_mul_mat()
Graph-level TP (meta backend, experimental) Every weight of a whole graph, by a rule per tensor As TP, but for a multi-layer model without a gather after each matrix ggml_tp_forward(), ggml_meta_backend()
Pipeline parallelism (PP) Whole layers, by depth The model does not fit on one card ggml_pp_forward()
Data parallelism (DP) The batch The model fits, you want throughput dp_train(), ggml_tp_dp_forward()

They compose. ggml_tp_dp_forward() and ggml_pp_dp_forward() run DP replicas on top of TP or PP groups — for example four GPUs as two replicas of TP=2.

The decisive practical difference is how much data crosses between devices:

That ordering matters more than it looks, because of the transport.

Are you sure you want this layer?

ggmlR provides the tensor layer: primitives that split individual matrices, graph stages and batches across devices. If your goal is to run a whole language model across several GPUs, you almost certainly want llamaR instead, which is built on ggmlR and handles model loading, KV cache, sampling and serving:

# llamaR — whole-LLM inference; the split happens inside the model loader
model <- llama_load_model("model.gguf", n_gpu_layers = -1L,
                          devices = c("Vulkan0", "Vulkan1"),
                          split_mode = "layer")   # or "tensor"

split_mode = "layer" is pipeline-style: whole layers per card. split_mode = "tensor" is real tensor parallelism through ggml’s meta backend (section 4) — every weight split across the cards, each layer run on all of them. split_mode = "row" asks the backend for a row-split buffer type; Vulkan does not offer one, so on Vulkan "row" silently behaves like "layer".

None of these calls into the ggml_vulkan_split_mul_mat() / ggml_pp_forward() primitives shown below; "tensor" shares the meta backend with ggml_tp_forward(). Reach for the ggmlR primitives when you are building a custom parallel computation — a model architecture of your own, a research prototype, a non-LLM workload — rather than serving an existing GGUF.

llamaR ships a full PP/TP/DP sweep over a real model in system.file("examples", "bench_pp_tp_dp.sh", package = "llamaR").

2. The transport, and why it constrains everything

Copying a tensor from GPU 0 to GPU 1 needs a path. ggmlR implements three, selected by the transport argument:

The default is not a conservative placeholder; on the hardware we tested it is the only path that works. Measured on a 4×P100 server:

Verify your own hardware before assuming a zero-copy path exists:

# Loopback sanity check: does the transport work at all on device 0?
r <- ggml_vulkan_p2p_selftest(0L, 0L)
cat(r$report)

# The real question: does a cross-device copy carry the bytes?
if (ggml_vulkan_device_count() >= 2) {
  r <- ggml_vulkan_p2p_selftest(0L, 1L, transport = "opaque-fd")
  cat(r$report)          # "verified" vs. a mismatch tells you immediately
}

# Are there any multi-device (LDA) groups?
ggml_vulkan_device_groups()

Because host-staging routes through host RAM, TP is the mode that suffers most and PP the least. If your model does not fit on one card, prefer PP.

3. Tensor parallelism

ggml_vulkan_split_mul_mat() splits the weight matrix W by rows across devices, computes each slice on its own GPU, and gathers the result. It is a standalone C routine, deliberately outside the ggml graph.

set.seed(1)
W <- matrix(rnorm(2048 * 64), nrow = 2048)   # [N x K] weights
X <- matrix(rnorm(4 * 64),    nrow = 4)      # [M x K] activations

Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2)

# Contract: identical to the single-device result, up to f32 rounding.
max(abs(Y - X %*% t(W)))                     # ~3.8e-6 on 2 devices

Pick a subset of GPUs rather than “the first N” with device_ids, and give uneven cards uneven shares with weights:

Y <- ggml_vulkan_split_mul_mat(W, X, device_ids = c(0L, 1L))
Y <- ggml_vulkan_split_mul_mat(W, X, n_devices = 2, weights = c(0.7, 0.3))

# Inspect the row ranges the split math will use (0-based, half-open):
ggml_vulkan_split_row_ranges(nrows = 2048, n_devices = 2)

ggml_vulkan_split_buffer_type() exposes the same row-split as a ggml buffer type, for callers that allocate weights themselves.

4. Graph-level tensor parallelism: the meta backend

ggml_vulkan_split_mul_mat() splits one matrix and gathers the result right away, so a two-layer model built from it pays two gathers. ggml’s meta backend (upstream since ggml 0.27; experimental here) splits a whole graph instead. It wraps several devices into one backend and:

The classic case is the Megatron MLP: split the first weight by its output features and the second by its input features. The hidden activation then stays split between the two layers, and the only reduction is after the second one.

Shapes follow ggml: an R array’s dim() is the tensor’s ne, fastest dimension first. A weight with dim = c(n_in, n_out) and an input with dim = c(n_in, batch) give ggml_mul_mat(ctx, w, input) of shape c(n_out, batch) — t(W) %*% x in R terms. The split rules name a ggml axis: "rows" is ne[1] (here: output features), "cols" is ne[0] (input features).

n_in <- 512L; n_hid <- 2048L; n_out <- 512L; batch <- 16L
W1 <- array(rnorm(n_in * n_hid),  c(n_in,  n_hid))
W2 <- array(rnorm(n_hid * n_out), c(n_hid, n_out))
X  <- array(rnorm(n_in * batch),  c(n_in,  batch))

mlp <- function(ctx, input, w)          # w: the weight tensors, by name
  ggml_mul_mat(ctx, w$w2, ggml_relu(ctx, ggml_mul_mat(ctx, w$w1, input)))

Y <- ggml_tp_forward(
  weights = list(w1 = W1, w2 = W2),            # names are what `split` refers to
  forward = mlp,
  x       = X,                                 # the input tensor is named "input"
  devices = c(0L, 1L),
  split   = list(w1 = "rows", w2 = "cols"))

max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0)))

Rules are "mirror" (the default: a full copy on each device), "rows", "cols", "axis2", "axis3", or list(axis = "rows", weights = c(0.7, 0.3)) for uneven cards; split may also be a function(name, ne). A rule that cannot apply — the axis is shorter than the number of devices — falls back to "mirror" for that tensor with a warning.

devices takes Vulkan ids, device names ("Vulkan0", "CPU") or device pointers, and may list the same device twice. That is how to try the API on one GPU, or with no GPU at all:

Y <- ggml_tp_forward(list(w1 = W1, w2 = W2), mlp, X,
                     devices = c("CPU", "CPU"),
                     split = list(w1 = "rows", w2 = "cols"))

For a graph you drive yourself — a scheduler, repeated computes, several outputs — use the backend directly:

meta  <- ggml_meta_backend(c(0L, 1L), split = list(w1 = "rows", w2 = "cols"))
sched <- ggml_backend_sched_new(list(meta, ggml_backend_cpu_init()))
# Weights (and inputs) in a context allocated on `meta`:
#   buf <- ggml_backend_alloc_ctx_tensors(ctx_w, meta)
# The graph in a SEPARATE context, allocated through the scheduler:
#   ggml_backend_sched_alloc_graph(sched, gf); ggml_backend_sched_graph_compute(sched, gf)

Keep the two contexts apart. The meta backend asks for a rule for every tensor in a buffer allocated on it; an intermediate result that ended up there would be laid out by its name rather than by the op that produced it. ggml_backend_meta_device() is the raw API underneath, with a callback that returns numeric axis codes (GGML_SPLIT_AXIS_*).

One more rule for a graph you drive yourself: a matrix product whose inputs are both split along the reduced axis ("cols" on the weight and on its input) leaves per-device partial sums. The meta backend reduces them before their first consumer, but never after the graph’s last node, and cannot read them back — so do not read such a tensor, and do not end a graph with one. ggml_tp_forward() appends a trivial consumer for you.

What it costs. Vulkan has no device-to-device all-reduce, so the meta backend’s reductions copy through host RAM — the same host-staging transport as the rest of this vignette (section 2). Graph-level TP moves less data than ggml_vulkan_split_mul_mat() layer by layer, but it still pays off only when the model does not fit on one card.

Status. Forward graphs only: ggml_fit(), the autograd engine and ONNX models do not run on it. Not yet measured on real multi-GPU hardware. llamaR reaches the same backend through split_mode = "tensor".

5. Pipeline parallelism

PP assigns layers to devices: GPU 0 runs stages 1–16, GPU 1 runs 17–32, and exactly one activation tensor crosses between them. Each stage is described by a list with its device, its in_shape, and a build function that constructs the sub-graph and returns a set_weights closure.

The one non-obvious rule: weights are set after allocation, which is why build returns set_weights instead of filling tensors immediately.

K <- 64L; M <- 8L
W1 <- matrix(rnorm(K * K), nrow = K)
W2 <- matrix(rnorm(K * K), nrow = K)
X  <- matrix(rnorm(K * M), nrow = K)     # ggml ne = c(K, M): column m is sample m

make_stage <- function(dev, Wt, relu) {
  list(
    device   = dev,
    in_shape = c(K, M),
    build = function(ctx, input) {
      w <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, K, K)
      z <- ggml_mul_mat(ctx, w, input)     # == t(Wt) %*% input in R terms
      list(
        output      = if (relu) ggml_relu(ctx, z) else z,
        set_weights = function() ggml_backend_tensor_set_data(w, as.numeric(Wt))
      )
    })
}

stages <- list(make_stage(0L, W1, relu = TRUE),
               make_stage(1L, W2, relu = FALSE))

y <- ggml_pp_forward(stages, x = as.numeric(X), out_shape = c(K, M))
Y <- matrix(y, nrow = K, ncol = M)

max(abs(Y - t(W2) %*% pmax(t(W1) %*% X, 0)))   # ~1.8e-5

6. Hybrid: DP over TP or PP

With four GPUs, two replicas of TP=2 usually beat one TP=4 group: DP adds no cross-device traffic, so the hybrid hides the weakness of host-staging.

# Replica A = {GPU0, GPU1}, replica B = {GPU2, GPU3}.
# Each replica takes half the batch; no traffic crosses between replicas.
Y <- ggml_tp_dp_forward(W, X, replicas = list(c(0L, 1L), c(2L, 3L)))

# The PP equivalent takes a factory: make_stages(devices, m_shard) -> stage list
Y <- ggml_pp_dp_forward(make_stages, x, replicas = list(c(0L, 1L), c(2L, 3L)),
                        out_ncol = M)

For multi-GPU training rather than inference, use dp_train() — see vignette("data-parallel-training").

7. Clean shutdown (and the --enable-hard-exit flag)

A multi-GPU script can segfault after it has printed its results. The cause is a race at process exit: the Vulkan loader and the driver’s ICD libraries get unmapped while driver worker threads are still winding down, so late destructors call into unmapped memory. The computed results are already out; the crash is harmless but noisy, and it turns a successful run into a non-zero exit code.

Call ggml_vulkan_shutdown() to tear Vulkan down while the loader is still mapped. It is idempotent, safe mid-session, and Vulkan transparently re-initializes on the next operation:

ggml_vulkan_shutdown()     # safe anywhere; releases devices

That narrows the race but does not close it: no R exit hook runs before R unmaps the loader. For a guaranteed-clean exit, make hard = TRUE the last statement of a standalone script — after teardown it calls _exit(status), terminating immediately without running exit handlers or unmapping libraries, so there is no teardown phase left to crash.

ggml_vulkan_shutdown(hard = TRUE, status = 0L)   # never returns

This path is compiled out by default. CRAN Repository Policy forbids a package from terminating the user’s R session, so the released package must not link _exit(). In a default build, hard = TRUE performs the normal teardown and emits a warning() — it is never silently ignored. Compile it in with:

R CMD INSTALL . --configure-args="--enable-hard-exit"

On Windows R ignores configure.args; set Sys.setenv(GGML_VK_HARD_EXIT = "1") before installing from source. Check what the current build has:

ggml_vulkan_hard_exit_available()

Two caveats. hard = TRUE bypasses R’s shutdown entirely: no .RData, no on.exit() handlers, no flushed connections beyond what ggmlR flushes itself. Use it only as the final line of a script, never mid-session, never inside a package. And although the race is observed in multi-GPU scripts — more devices mean more driver threads and a wider window — nothing in the mechanism is specific to multiple GPUs; single-GPU runs simply have not been seen to trip it.

8. Where to look next

Runnable demos ship with the package, under system.file("examples", package = "ggmlR"):

list.files(system.file("examples", package = "ggmlR"),
           pattern = "^(tp_|pp_|multi_gpu|dp_)")

The test suite documents the numeric contracts, and skips gracefully when fewer than two GPUs are present:

9. Summary

ggml_vulkan_status()

Choose PP when the model does not fit, TP when a layer does not fit or latency matters, DP for throughput — and remember that on current NVIDIA hardware every cross-device byte goes through host RAM, so the mode that moves the least data wins.