When an LLM engine course of fails, the usual restoration path includes a chilly restart. This requires loading weights into HBM from storage, compiling kernels, and capturing NVIDIA CUDA graphs. For giant fashions, initialization can take a number of minutes, throughout which surviving employees should take up the displaced visitors.
Shadow engine restoration, obtainable as a preview function in NVIDIA Dynamo, strikes most of this restoration work off the serving path. It retains a totally initialized shadow engine idle on the identical GPUs because the energetic engine. The GPU Reminiscence Service (GMS) shares the prevailing weights between the engines with out creating one other copy in HBM. If the energetic course of fails, the shadow takes over inside seconds. Re-initialization happens within the background solely off the serving path.
We measured the influence by intentionally terminating one employee in a two-worker GLM-5.2 deployment. With out shadow engine restoration, the remaining employee served all incoming visitors throughout the 283-second chilly restart, rising TTFT and decreasing per-user decode fee all through the outage. With shadow engine restoration, a second employee resumed serving in 7.3 seconds, almost 39 instances sooner, minimizing disruption to service high quality.
Why LLM inference restoration is sluggish: two core issues
Manufacturing LLM engines generally expertise recoverable software program faults, together with course of crashes, recoverable CUDA errors, and transient collective failures. In these instances, the {hardware}, drivers, and node stay wholesome; solely the method holding the corrupted state is misplaced, and a substitute engine can sometimes begin on the identical GPUs.
So why can’t that contemporary engine skip the initialization price? Two issues stand in the way in which:
Weights are tied to the engine course of. GPU reminiscence is linked to the engine’s CUDA context, which is itself tied to the engine course of. When the method exits, the motive force releases all sources, together with weights already resident in GPU reminiscence. Consequently, a substitute engine course of should repeat the total weight-loading process.
Some initialization states are non-transferable. NCCL and torch.distributed communicators bind to the particular operating course of, and CUDA graphs are fastened to the digital addresses current throughout seize. These states can’t be handed off from a earlier engine and should be recreated throughout each restart.
Shadow engine restoration addresses every downside with a focused optimization: decoupling weight lifetime from the engine course of, and finishing non-transferable initialization earlier than a failure.
How shadow engine restoration works
Shadow engine restoration combines persistent GPU reminiscence, a pre-warmed standby engine, and worker-level coordination to get better with no chilly restart.
GPU Reminiscence Service: Persistent GPU reminiscence for LLM inference
The GPU Reminiscence Service (GMS) manages particular reminiscence areas, corresponding to weights, independently of the engine course of. Through the use of a course of distinct from the engine to personal these areas, weights stay resident in reminiscence whilst engines are restarted. Consequently, a brand new engine on the identical GPU can connect to current reminiscence.
GMS is a per-GPU sidecar that owns bodily GPU reminiscence on behalf of inference engines. It’s principally dormant and has no CUDA context of its personal; it allocates bodily pages, palms out handles to them, and arbitrates which engines could learn or write at any time. Engines join, import handles, and map the underlying pages at digital addresses in their very own CUDA contexts. Mapping occurs as soon as, at startup, and GMS is just not concerned in any subsequent entry.
This performance is constructed on the CUDA Digital Reminiscence Administration API. With this API, bodily GPU reminiscence and its related digital addresses can have impartial lifetimes. Since bodily allocations are reference-counted, they survive so long as any course of maintains a mapping. Two engines mapping the identical weight tensor entry the identical bodily bytes, every utilizing digital addresses native to its context. A kernel studying a weight dereferences an unusual pointer into the identical HBM the burden would have occupied anyway, so a GMS-backed learn prices not more than an engine-allocated one.
This structure gives two advantages. First, weights persist past engine failure. Whereas the kernel removes the failed engine’s CUDA context, the GMS reference ensures the bodily pages keep resident so a contemporary engine can instantly map them. Second, weights will be shared between concurrent engines; due to this fact, a secondary engine on the identical GPU incurs zero marginal weight price.
Integrating GMS into inference frameworks requires solely a slim change. vLLM, SGLang, and NVIDIA TensorRT-LLM every combine GMS by means of a customized torch.cuda.CUDAPluggableAllocator sure to the burden reminiscence pool. From contained in the engine, weights stay unusual torch.Tensors. Adopting GMS is so simple as flipping a flag at startup.
GMS is just not restricted to weights. The present preview doesn’t help utilizing GMS for the KV cache, however that functionality is underneath energetic improvement. The objective is for a promoted shadow to map the outgoing engine’s cache as an alternative of rebuilding it as visitors arrives.
Shadow engines: preinitialized standbys with zero marginal weight price
A shadow engine is a totally initialized engine course of that is still idle and co-resident on the identical GPUs because the energetic engine. Weight sharing makes this configuration possible: with out it, the second engine would require one other full copy of the weights, considerably decreasing the reminiscence obtainable for processing requests.
A shadow runs by means of the identical startup path as an energetic engine. On every of its GPUs, it connects to the native GMS and imports the burden mappings, establishes communicators (NCCL and NIXL for KV switch between employees), captures CUDA graphs, and performs any vital warm-up. By the top of the startup, it is able to serve. Then, as an alternative of serving, it parks: it releases the materializable elements of its reminiscence and blocks, ready for its flip.
What a shadow has precomputed earlier than it parks:
CUDA context, captured graphs, and communicators. These non-transferable parts are prepared when the shadow engine is activated as a result of they can’t be inherited from one other course of.
Weight mappings. The GMS handles are already imported, so waking the shadow requires solely remapping them to digital addresses established throughout initialization.
What it has deferred:
KV cache materialization. The KV cache is the biggest reclaimable allocation held by an engine. The shadow reserves its handle vary with out bodily backing whereas parked and materializes the cache when promoted.
A parked shadow due to this fact retains solely its CUDA context, captured graphs, communicators, and weight mappings—no separate copy of the weights and no KV cache. This footprint is sufficiently small for the shadow to stay alongside an energetic engine on the identical units, enabling restoration inside seconds.
The employee: a single deployable unit
The next determine reveals how these parts match inside every employee and scale behind a shared router.
These foundations are built-in into one pod. The employee holds two engine containers, a GMS sidecar to mediate GPU reminiscence entry, and a shared lock to elect the energetic engine.
At regular state, one engine holds the lock and stays awake, related to GMS, holding a materialized KV cache, and registered with the frontend router. The opposite stays totally initialized and related to GMS however is dormant, holds no KV cache, and waits on the lock.
Restoration in depth
The next sections hint the restoration sequence and clarify the synchronization and memory-management mechanisms that make it dependable.
Sequence
A employee strikes by means of 4 phases earlier than returning to regular state.
T₀ Regular. Engine A holds the lock and is awake, registered with the router. Engine B is dormant, blocked on the lock.
T₁ Failure. Engine A’s course of exits, both as a result of it crashed outright or as a result of a liveness probe discovered it hung and killed it. Both means, the kernel releases its lock as the method is reaped. The employee is briefly unroutable, till the shadow registers.
T₂ Cutover. Engine B acquires the lock, wakes, remaps weights by means of GMS, materializes its KV cache, and re-registers with the router. Engine A’s container is restarted by the orchestrator.
T₃ Restarted. Engine A finishes initialization and enters the shadow state. The system returns to a gradual state, with the roles swapped.
The shadow’s benefit is that it enters T₂ already initialized. The one work on the important path is buying the lock, remapping weights, and materializing the KV cache.
Synchronization
The employee requires each mutual exclusion, making certain just one engine is awake at a time, and dependable launch to make sure the standby engine takes over if the energetic one fails. A POSIX flock on a shared file gives these ensures. When the energetic course of exits on account of a shutdown, segfault, or SIGKILL, the kernel reaps its file descriptors, and the shadow engine acquires the lock to start serving.
Every engine’s startup path is due to this fact a brief chief election:
…
# put the engine to sleep whereas we wait on the lock
await engine.sleep()
lock = FlockFailoverLock(lock_path)
await lock.purchase(engine_id=engine.id) # wait on the lock to wake
await engine.wake()
A deadlocked engine whose course of continues to be alive falls to the Kubernetes liveness probe, which cascades to a SIGKILL and journeys the identical kernel-managed launch.
Reminiscence accounting
Becoming two engine processes on one GPU with out exhausting HBM takes cautious accounting throughout the lifecycle.
Weights. Allotted as soon as by GMS and mapped read-only by each engine within the employee; by no means duplicated.
KV cache. Held solely by the energetic engine in the present day: materialized when it wakes, launched when it dies, releasing the area for the shadow to take over.
Buffers and graphs. NCCL buffers, the CUDA context, and captured graphs. Held by every engine even whereas dormant, and the entire of a parked shadow’s standing price.
Benchmark outcomes: shadow engine restoration vs. chilly restart on GLM-5.2
To quantify the profit, we in contrast shadow engine restoration with a chilly restart after an engine failure in a two-worker fleet.
Setup
We ran two employees serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes: one employee per node, TP=8, 200K max context, and an FP8 KV cache. A single frontend distributes requests round-robin throughout the 2. The load is artificial: 32,000 enter tokens and 1,000 output tokens per request, arriving at 0.7 requests per second.
Each arms run similar engine builds and configurations; the one distinction is the shadow engine. The baseline has it off, and the killed employee cold-restarts. Within the Shadow Engine Restoration configuration, every employee pod hosts a preinitialized shadow engine that may take over if the energetic engine fails.
We injected the fault solely after the workload reached its steady-state working level: a SIGKILL to one of many two employees, adopted by 600 seconds of statement. With two employees within the fleet, shedding one leaves the survivor carrying each request till its accomplice returns.
Outcomes
In comparison with the chilly restart baseline of 283 seconds, shadow engine restoration solely takes 7.3 seconds (1.7 seconds to detect the fault and 5.6 seconds to advertise the shadow). This leads to considerably higher TTFT and decode fee after the fault, and permits us to largely keep away from SLA violations.
Present scope and subsequent steps
Having validated that quick restoration for inference workloads on Kubernetes is possible, we’re working to stabilize the implementation and increase help to a wider vary of workloads. Shadow engine restoration will roll out incrementally over the approaching months.
The present preview has a number of limitations and deployment necessities.
Shadow engine restoration addresses widespread engine course of failures however doesn’t cowl {hardware}, node, or multi-node failures, which nonetheless depend on customary rescheduling.
It requires Dynamic Useful resource Allocation (DRA) on Kubernetes, so the cluster wants Kubernetes 1.34 or newer with DRA enabled and the NVIDIA GPU DRA driver put in.
Dynamo Snapshot will be composed with the restoration function to reduce competition throughout the initialization of the shadow whereas serving.
As a result of promoted shadows begin with empty KV caches, the post-cutover TTFT experiences a slight bump. Carrying cache state throughout a promotion, each the prefix-cache index and the cache reminiscence itself, is the energetic line of labor.
vLLM is the first supported backend.
To attempt Shadow Engine Restoration, begin with the Kubernetes quickstart to create a operating deployment. Then comply with the Shadow Engine Restoration deployment workflow and use the vLLM failover instance as an entire manifest. Go to the ai-dynamo/dynamo repository to ask questions, report points, or contribute.
