Each byte moved has a price. As mannequin checkpoints develop to a whole lot of gigabytes or perhaps a terabyte, that price provides up shortly. To make issues even worse, shifting these mannequin weights across the cluster is extraordinarily frequent. As an example, a chilly begin could pull weights from distant storage into GPU reminiscence; autoscaling and rolling updates should populate every new duplicate; and RL post-training repeatedly strikes up to date weights from trainers to roll out staff. These could seem like totally different workflows, however they impose the identical recurring tax: time spent shifting weights earlier than helpful work can start.
ModelExpress: Accelerating the mannequin weight lifecycle
NVIDIA ModelExpress (MX) is constructed round a easy thought: Earlier than loading a mannequin, first ask the place a appropriate copy of its weights already lives. Moderately than treating each duplicate as an impartial chilly begin, MX chooses the quickest obtainable supply and switch path.
When a serving peer already holds appropriate weights in GPU, MX transfers them straight from GPU to GPU over P2P RDMA by way of NVIDIA Inference Xfer Library (NIXL), bypassing redundant entry to object storage, native disk, and host reminiscence. When no peer is obtainable, MX bootstraps from the quickest supported path by streaming from an object retailer with out touchdown on disk or studying native information straight into GPU reminiscence.
MX transfers DeepSeek-V4 Professional weights and JIT Kernel cache artifacts from a serving duplicate right into a recent duplicate in below 10 seconds, lowering the whole startup time to 1 minute 44 seconds from 8 minutes. The remainder of the publish reveals how MX selects the quickest obtainable path to GPU reminiscence, prioritizing P2P RDMA from a serving duplicate and eliminating redundant downloads and copies alongside the best way. It then extends the identical strategy to reusing kernel caches and distributing RL weight updates.
Accelerating each stage from distant storage to GPU reminiscence
Each new employee should get its weights from one in every of three locations: distant storages (e.g. HF or S3), native storage, or one other employee already serving the mannequin. For the primary employee, there is no such thing as a peer but, so it should bootstrap from storage. MX can stream the checkpoints from object storage or load it from quick native storage, eradicating avoidable copies alongside both path.
As soon as that first employee is serving, the popular supply modifications. Its weights are already resident, post-processed, and specified by GPU reminiscence, so each appropriate employee after it ought to load straight from that peer over P2P RDMA. MX makes this transition robotically: bootstrap as soon as from storage, then scale out GPU to GPU, falling again to storage solely when no appropriate peer is obtainable.
Beginning the primary employee: Bootstrap from storage
Distant object storage to GPU: Avoiding native disk
When the checkpoint lives in a cloud bucket and you’ll quite not provision and handle a disk cache tier, MX makes use of the Mannequin Streamer to tug safetensors by a reusable CPU staging buffer and into GPU. The checkpoint by no means lands on native disk, eliminating the intermediate obtain, reload, and storage quantity.
Mannequin Streamer makes use of a multithreaded tensor reader to fetch tensor ranges concurrently throughout checkpoint shards. As tensors arrive, it pipelines distant reads with GPU placement: accomplished tensors are handed to the inference engine whereas later tensors are nonetheless being fetched. This retains the storage, community, and GPU copy paths busy whereas reusing a bounded quantity of host reminiscence.
In tensor-parallel deployments, the taking part ranks divide the distant reads and share the outcomes, usually over NCCL, as a substitute of getting each rank obtain the complete checkpoint independently. MX connects this distributed stream on to the inference engine’s weight loader, making ready the primary employee to develop into the P2P supply for each appropriate duplicate that follows.
Cluster ingress: Obtain as soon as, not N occasions
When a cluster maintains a shared disk cache tier (e.g. persistent volumes in K8s), MX ensures that the fleet populates it solely as soon as. If 10 replicas concurrently wish to fetch the 806 GiB DeepSeek-V4 Professional mannequin, they might want to pull roughly 8 TiB of similar information throughout the community whereas competing for a similar ingress bandwidth. The MX Mannequin Cache Service collapses these requests into one coordinated obtain: an atomic declare in Metadata Retailer selects a downloader, whereas the remaining replicas observe its progress and reuse the cached copy. The cluster pays the exterior obtain price as soon as, then each duplicate can start from the identical cached checkpoint.
Native storage to GPU: Bypassing host-memory staging
When GPUDirect Storage (GDS) is supported within the system, MX reads checkpoint information straight from native storage into GPU reminiscence by NIXL’s multithreaded GDS backend. NIXL executes batched tensor reads in parallel straight into GPU reminiscence, bypassing host reminiscence and the staging copy required by a traditional loader. Customers don’t must allow GDS explicitly: MX detects the potential robotically and falls again to a different loading technique when it’s unavailable.
Native storage to GPU: Pipelining native reads with ModelStreamer
MX can even load native checkpoints by ModelStreamer. A number of OS threads learn safetensors concurrently right into a configurable CPU buffer whereas accomplished tensors transfer to the GPU and later reads proceed in parallel. Not like GDS, this path nonetheless levels by host reminiscence, nevertheless it overlaps disk I/O with GPU placement, advantages from the OS web page cache, and supplies a conveyable quick path when direct storage-to-GPU entry is unavailable.
Beginning each employee after the primary: Fetch from a serving peer
That is the important thing characteristic of MX. As soon as one other duplicate is already serving the identical mannequin, the weights have accomplished most of their journey: they’re resident in GPU reminiscence, post-processed, and laid out for the inference engine. MX treats that duplicate as a stay weight supply. After confirming compatibility, it transfers the tensors straight from the supply GPU to the goal GPU. As soon as its weights are loaded, the brand new duplicate joins the supply pool, giving subsequent replicas one other peer to load from. With each profitable switch, that pool grows alongside the deployment, turning scale-out into GPU-to-GPU fan-out as a substitute of repeated chilly hundreds.
The MX management airplane discovers appropriate friends, exchanges switch metadata, and tracks supply readiness, however by no means handles the burden bytes themselves. On the information airplane, MX makes use of NIXL as a default switch engine whose pluggable backends permit for peak efficiency throughout a wide range of networks, resembling Infiniband, RoCE, NVLink, EFA, and so forth. MX has a first-class transport interface that enables libraries resembling fabric-lib and standalone Mooncake to combine with MX.
Earlier than any switch begins, MX computes an mx_source_id from the mannequin and runtime settings that decide tensor structure, then considers solely friends with an identical ID. The management airplane discovers these friends by Redis, Kubernetes CRDs, or k8s-service (serverless) metadata backends.
Optimizing NIXL reminiscence registration overhead
Earlier than NIXL can RDMA a tensor, the GPU reminiscence backing it needs to be registered: an ibv_reg_mr name that returns the Distant Key (rkey) used for distant entry. A big mannequin has tens of 1000’s of tensors, and registering them separately is gradual sufficient to point out up within the price range. By default, MX registers every tensor individually. Two opt-in methods cut back that registration price:
Pool registration registers every underlying cudaMalloc allocation as soon as as a substitute of every tensor, slicing registration rely by 80 to 99 % on typical fashions with no change to switch semantics.
VMM enviornment registration goes additional. It installs a CUDAPluggableAllocator that routes each load-time allocation right into a single 16 TiB virtual-address enviornment, then registers the entire used vary as one dmabuf-backed reminiscence area at finish of load. Registration collapses from one name per tensor to at least one name, whole; every tensor descriptor merely carries an offset into that single area.
Utilizing DeepSeek-V4-Professional TP=8 on the vLLM engine, as proven in Determine 3, beneath, we measured the common NIXL registration time for every strategy.
Runtime path choice and secure fallback
At startup, MX probes the obtainable capabilities, robotically skipping any path the surroundings doesn’t assist. The primary relevant technique runs within the present precedence order: P2P RDMA -> ModelStreamer -> GDS -> default loader (host-staged POSIX I/O). If a path is unavailable or fails earlier than modifying the mannequin state, MX falls by robotically. If a failure happens after weights have begun touchdown, it reinitializes the mannequin earlier than persevering with, so partially written weights are by no means served.
P2P retries alternate friends just for metadata failures earlier than switch begins, and the native loader stays the ultimate fallback. This capability-driven design retains the MX core {hardware} and software program agnostic, with platform-specific quick paths enabled solely the place supported.
Finish-to-end outcomes
We ran DeepSeek-V4-Professional on an 8xB200 GPU node with NVIDIA ConnectX-7 NICs and in contrast the whole mannequin loading time throughout totally different chilly begin situations. Every duplicate used vLLM 0.23.0 with TP=8 and –enable-flashinfer-autotune. See Determine 4, beneath.
Heat, not simply loaded: Inheriting the compiled kernels
Getting weights into GPU reminiscence is important, however a loaded mannequin shouldn’t be but able to serve. Throughout its first ahead passes, the engine JIT-compiles and autotunes kernels (e.g. torch.compile, Triton, DeepGEMM, TileLang, and and so forth.) and captures CUDA graphs for the precise mannequin, dtype, quantization, and GPU. For fashions resembling DeepSeek-V4 Professional, this could take a number of minutes and might develop into the dominant startup price as soon as MX reduces weight-loading latency (see Determine 5, beneath).
That repeated warmup is avoidable. When the mannequin, software program stack, and GPU structure match, one duplicate pays the compilation price and the remainder can inherit the ensuing caches.
MX’s Artifact Switch API packages these file-backed artifacts, transfers them straight between registered host-memory buffers over NIXL’s CPU-to-CPU RDMA path, then verifies and installs them within the goal engine’s cache listing. This eliminates the necessity for a shared ReadWriteMany (RWX) quantity in Kubernetes, whereas an artifact-specific mx_source_id prevents reuse throughout incompatible replicas. MX detects commonplace cache areas robotically when a Redis or Kubernetes metadata backend is configured.
We ran utilizing the identical setup to measure how a lot the kernel artifact switch can cut back the startup time. The artifact-enabled run transferred the Triton/DeepGEMM/TileLang/CuTe DSL/FlashInfer caches. The chart compares the main startup levels and whole wall-clock time from course of begin till the API was prepared.
When the weights change each Step: RL post-training
The whole lot up to now assumes a mannequin’s weights are mounted as soon as loaded. RL post-training breaks that assumption. A coach updates the coverage each step, and the inference actors producing rollouts should decide up these weights earlier than the subsequent spherical of technology. As with inference startup, weight motion is on the important path in RL: rollout staff wait whereas up to date weights transfer from the coach’s distributed structure (whether or not FSDP/DTensor shards or Megatron TP, PP, and EP partitions) into the inference engine’s structure.
MX drives the refit by the next 4 levels:
Publish: Every coach rank advertises the tensors or shards it already owns, along with metadata describing their form, dtype, placement, and parameter mapping to MX.
Uncover: A rollout employee seems up the requested weight model and its obtainable sources by MX.
Plan: The receiver maps the printed possession info onto its personal goal structure and identifies which sources comprise the required tensors or ranges.
Pull, convert, and cargo: The receiver points one-sided reads straight towards these sources.
MX consists of the core constructing blocks for receiver-driven refit, and prospects are evaluating them in lively integrations. We’re additionally testing delta weight diff refits for cross-cluster weight switch, a method utilized by Fireworks/Cursor, Cognition, and extra in current RL runs.
Contributing to Dynamo and our roadmap
MX has native integrations with vLLM and SGLang and helps serving frameworks together with Dynamo and llm-d.
The Dynamo open supply group is actively working towards deeper TensorRT-LLM integration and broader inference capabilities. Discover the present Dynamo documentation and roadmap, strive the obtainable workflows in your personal surroundings, and contribute suggestions to assist form the undertaking’s course.
AcknowledgmentsModelExpress is a crew effort. Thanks to the remainder of the MX crew, Zhongdongming Dai and Tanushriya Singh for his or her core work on the undertaking. We’re grateful to Itay Neeman, Anish Maddipoti, Istvan Haller, and Omri Kahalon for his or her steerage on the undertaking’s technical course, and Will Eaton at Crimson Hat for his assist on the llm-d integration.
