Giant language mannequin (LLM) coaching workloads more and more run into GPU reminiscence limits earlier than compute is totally used. Mannequin weights, gradients, optimizer states, communication buffers, and intermediate activations all compete for GPU high-bandwidth reminiscence (HBM). As mannequin dimension, sequence size, and batch dimension develop, HBM capability typically turns into the first scaling bottleneck.
This put up explains how host offloading within the open supply Python library JAX reduces HBM stress. This course of, which is very advantageous on NVIDIA Blackwell, strikes chosen activations to pinned host reminiscence in the course of the ahead move and streams them again when wanted within the backward move. Host offloading is an alternative choice to activation rematerialization. As an alternative of recomputing chosen activations, the coaching step reloads them from host reminiscence.
Why is host offloading advantageous on NVIDIA Grace Blackwell techniques?
Host offloading is very advantageous on NVIDIA Grace Blackwell techniques. The NVIDIA Grace CPU and NVIDIA Blackwell GPU are related by NVLink-C2C with 900 GB/s of bidirectional bandwidth, making pinned host reminiscence a sensible staging space for chosen activations. The Vera CPU and Rubin GPU additional enhance on this by doubling the bidirectional velocity to 1.8 TB/s of coherent bandwidth.
Excessive-bandwidth CPU-GPU connectivity helps make host offloading sensible, however bandwidth alone isn’t sufficient. To enhance efficiency, activation transfers should overlap with helpful GPU work.
Efficiency outcomes on MaxText workloads
The experiments use MaxText, a JAX LLM coaching framework that makes use of the Accelerated Linear Algebra (XLA) compiler for coaching at scale on NVIDIA GPUs. All outcomes have been measured on NVIDIA GB200 NVL72 techniques utilizing 128 GPUs.
The analysis makes use of two MaxText workloads. Llama 3.1 405B is a dense decoder-only transformer mannequin used to review focused question, key, and worth (QKV) activation offloading at a set batch dimension. DeepSeek-V3 671B is a sparse mixture-of-experts (MoE) mannequin with multihead latent consideration (MLA), used to review each throughput and memory-capacity results.
DeepSeek-V3 671B offloading coverage
DeepSeek-V3 671B has 61 decoder layers: the primary three use dense multilayer perceptron (MLP) blocks, whereas the remaining layers use MoE blocks.
Determine 1 reveals the activation offloading coverage for the repeated MoE decoder layer, which dominates the stack. The primary three dense MLP layers aren’t proven; they use the same coverage, with chosen up and down projection outputs offloaded.
Within the MoE layer proven in Determine 1, the coverage offloads chosen MLA question and key/worth projection intermediates and chosen MoE up projection intermediates. These activations are massive sufficient to impression whether or not bigger batch configurations match.
Coaching throughput enhancements
MaxText stories throughput as TFLOPs/s/gadget, computed from mannequin FLOPs per coaching step divided by measured step time.
DeepSeek-V3 671B
Determine 2 compares DeepSeek-V3 671B throughput throughout activation placement insurance policies. With offloading, LHS, and pipelined transfers enabled, the workload reached 908.2 TFLOPs/s/gadget. This was 57% sooner than activation rematerialization on the similar batch configuration and 67.7% sooner than offloading with out LHS or pipelining.
Not like the dense Llama workload the place LHS alone was enough to cover latency, the large activation footprint of DeepSeek-V3 MoE and MLA layers implies that enabling pipelined transfers supplies a definite, optimistic impression on complete throughput.
These efficiency leaps spotlight a basic NVIDIA benefit: the tight co-design of software program and {hardware}. In a typical commodity cluster, streaming huge MoE activation layers to host reminiscence would stall your entire coaching pipeline. On Blackwell, XLA customized scheduling flags work hand-in-hand with the {hardware} to make sure knowledge is moved asynchronously over devoted copy streams.
This permits NVIDIA platforms to unlock huge batch configurations that stay fully out of attain for architectures missing this tight compiler-to-interconnect integration.
Rising possible batch dimension
Host offloading can even change which batch configurations are possible. Desk 1 reveals DeepSeek-V3 671B outcomes throughout activation placement insurance policies.
Rows 4 and 5 present the capability comparability. Saving chosen activations on gadget match micro batch 2 and international batch 256, whereas optimized host offloading match micro batch 8 and international batch 1024. With out utilizing both offloading or rematerialization, the gadget will hit an Out-of-Reminiscence (OOM) error when making an attempt to make use of a micro batch dimension of 8 and international batch dimension of 1024.
Host offloading made the micro batch 8, international batch 1024 configuration possible by shifting chosen activation storage out of GPU reminiscence. On this DeepSeek configuration, the offload coverage targets massive intermediate activations from MLA, MoE, and MLP blocks. Maintaining these activations on gadget limits the possible batch configuration, whereas offloading them leaves extra HBM obtainable for mannequin state, communication buffers, runtime workspaces, and lively computation.
With LHS and pipelined transfers enabled, the offload run makes use of 165.2 GiB of GPU reminiscence, in contrast with 145.6 GiB with out these optimizations. The rise comes from maintaining extra copy buffers and prefetched activations in GPU reminiscence whereas transfers overlap with computation. This additional HBM use trades some reminiscence capability for higher overlap and better throughput.
Llama 3.1 405B
The Llama 3.1 405B experiment ran 10 steps on artificial knowledge with batch dimension 2, sequence size 8,192, totally sharded knowledge parallelism (FSDP) set to 128, and bfloat16 activations with NVFP4 4-bit weight quantization.
As proven in Desk 2, QKV activation offloading with Latency Hiding Scheduler (LHS) improved throughput from 2,669 to 2,746 TFLOPs/s/gadget, a 2.9% enhance over the baseline with out offloading. Disabling LHS lowered QKV offload throughput to 2,569 TFLOPs/s/gadget, highlighting that host offloading will depend on efficient overlap with different GPU work.
For this Llama 3.1 405B run, LHS alone supplied one of the best throughput: 2,746 TFLOPs/s/gadget with out pipelining in contrast with 2,718 TFLOPs/s/gadget with pipelining. On this configuration, LHS already hides most switch latency behind compute and communication, leaving little uncovered latency for pipelining to cover.
The 70.9 GiB host reminiscence worth is the overall QKV activation storage throughout all 126 layers, not the quantity of GPU reminiscence saved at a single second. At batch dimension 2 and sequence size 8,192, one layer’s bfloat16 QKV activations require about 576 MiB: 512 MiB for question and 32 MiB every for key and worth.
With the scan loop enabled for layers (scan_layers=True), the backward move processes one layer at a time, so QKV activations of just one layer are wanted on the GPU directly. On this workload, QKV offloading is principally a efficiency optimization: it replaces backward move QKV rematerialization with transfers that may overlap with compute and communication. GPU peak reminiscence stays dominated by mannequin state, communication buffers, and runtime workspaces.
Llama 3.1 405B supplies a dense-model fixed-batch instance. The acquire is smaller than DeepSeek V3 671B, nevertheless it reveals the identical mechanism: focused QKV offloading replaces backward-pass rematerialization with transfers that overlap with compute and communication.
When is host offloading most helpful?
Host offloading is most helpful when GPU reminiscence limits mannequin dimension, sequence size, or batch dimension, and when chosen tensors are massive sufficient to cut back HBM stress. It’s particularly helpful when offloading can change costly activation rematerialization or make a bigger batch configuration possible.
Efficiency will depend on overlap. Host offloading works greatest when the workload has sufficient compute, communication, or different unbiased work to cover switch latency. On NVIDIA GPUs, XLA helps create this overlap through the use of devoted copy streams, scheduling transfers with LHS, and enabling pipelined host offloading.
This pipeline is especially advantageous on platforms just like the NVIDIA Blackwell GB200 and NVIDIA Blackwell Extremely GB300, which leverage devoted NVLink-C2C interconnect to bypass conventional PCIe bottlenecks fully. With the NVIDIA Vera Rubin platform delivering even greater interconnect efficiency, host offloading supplies a predictable architectural lever for decoupling coaching throughput from strict bodily reminiscence limits.
Host offloading is much less possible to assist when tensors are small, when little unbiased work is on the market to overlap transfers, or when the workload is bottlenecked someplace apart from reminiscence. Runtime reminiscence ought to nonetheless be validated with actual runs as a result of static estimates could not embody NVIDIA Collective Communications Library (NCCL) communication scratch house, NVIDIA cuDNN consideration workspace, and framework-managed buffers.
Find out how to get began with host offloading
Begin with a small consultant JAX coaching run. Select massive activations from costly ahead paths, allow offloading, and measure runtime GPU reminiscence, host reminiscence use, and end-to-end step time.
For open supply JAX APIs, see the JAX host offloading tutorial. It covers activation offloading with jax.remat, checkpoint insurance policies, and memory_kind=”pinned_host”, in addition to parameter and optimizer state offloading with jax.device_put().
The MaxText experiments on this put up used the NGC JAX container. The DeepSeek-V3 671B runs used ghcr.io/nvidia/jax:deepseek_v3_maxtext, a personalized container with further MaxText integration for Transformer Engine MoE permutation optimizations.
The optimized offload configurations used the next XLA flags:
–xla_gpu_enable_pipelined_host_offloading=true
–xla_gpu_experimental_parallel_async_compute_limit=8
The final setting will increase the quantity of asynchronous work that may be in flight, giving LHS extra room to overlap activation copies and NCCL collectives. Use profiling instruments resembling NVIDIA Nsight Methods to verify that device-to-host and host-to-device copies overlap with compute and NCCL communication.
Be taught extra
Host offloading is a robust match when HBM capability limits batch dimension, context size, or mannequin scale, and when chosen activations are massive sufficient to cut back GPU reminiscence stress. It’s handiest when it replaces costly activation rematerialization or makes a bigger batch configuration possible.
Deal with host offloading as a reminiscence placement alternative that ought to be validated with measurements. Select massive activations, allow overlap with LHS and pipelined host offloading, and profile each runtime reminiscence and step time.
Whereas the outcomes introduced on this put up concentrate on Llama 3.1 405B and DeepSeek-V3 671B, the identical strategy can assist different JAX workloads when chosen activations are massive sufficient to matter and switch price can overlap with helpful computation.
To run JAX on NVIDIA GPUs, NVIDIA JAX-Toolbox supplies maintained containers, documentation, and optimized JAX and MaxText examples.
Acknowledgments
We want to thank Jaroslav Sevcik, Sevin Varoglu, Tj Xu, Haixin Liu, Abhinav Goel, Md Fahim Faysal Khan, Stefano Bosisio, and Jinxin Yang for his or her technical contributions.
