Two AI computing clusters constructed from equivalent NVIDIA H100, GB200 NVL72, or GB300 NVL72 methods can ship materially completely different coaching throughput. We routinely see 8% to 12% gaps between accomplice deployments and the corresponding NVIDIA reference structure (RA) on the identical workload, similar mannequin, similar world batch measurement.
The trigger is commonly a stack of configuration selections within the kernel, hypervisor, BIOS, and NVIDIA Collective Communications Library (NCCL) settings, every costing a number of %, that compound into a niche massive sufficient to overlook the 95% threshold required for NVIDIA Exemplar Cloud validation.
This publish walks by way of 4 debugging investigations from actual accomplice clusters. Every diagnostic isolates a definite layer of the stack: system reminiscence administration unit (SMMU) and page-table habits on NVIDIA Grace CPU; energy administration and non-uniform reminiscence entry (NUMA) placement on x86-based CPU; NVIDIA NCCL queue-pair concurrency on 1.6 Tbps materials; and silent hardware-installation defects. The publish additionally reveals the precise sign in perf, NVIDIA Nsight Programs, or NVIDIA NCCL checks that pointed to the basis trigger, alongside the tuning change that closed the hole.
Infrastructure engineers and efficiency architects who already run these benchmarks can profit from these diagnostic patterns we use internally to run towards their very own clusters earlier than formal RA validation.
Stipulations
To breed the diagnostics on this publish, you will want:
An NVIDIA HGX H100, HGX H200, HGX B200, GB200 NVL72, or GB300 NVL72 methods cluster with NVIDIA Quantum InfiniBand or RoCE interconnect.
A distributed coaching workload with steady iteration timing—NVIDIA NeMo on Llama 3 mannequin, NVIDIA Nemotron, or DeepSeek configuration is an inexpensive reference.
Root entry on no less than one node for perf, BIOS/UEFI adjustments, and kernel parameter adjustments.
nccl-tests constructed towards the identical NCCL model your coaching stack makes use of, NVIDIA Nsight Programs, and Linux perf with kernel symbols obtainable.
Widespread patterns behind coaching efficiency gaps
Latest Exemplar coaching engagements present that efficiency gaps not often come from a single apparent failure. Extra typically, they arrive from configuration particulars that grow to be seen solely underneath workload strain. Some recurring patterns embrace:
Grace and virtualization readiness: Lacking platform capabilities, SMMU overhead, IOMMU habits, or page-size settings that don’t match the anticipated configuration.
CPU energy and course of placement: Cores operating under anticipated turbo frequency, ranks or helper threads positioned on the flawed cores, or NUMA/PCT bindings that don’t match the platform topology.
Runtime topology: Host topology information or NCCL settings which can be appropriate on the node however lacking contained in the workload container or launcher atmosphere.
Cloth and collective habits: NCCL settings that don’t match the goal material, message measurement, or scale of the coaching workload.
Software-to-platform binding: Coaching processes binding by core ID or rank order as an alternative of topology-aware affinity.
These aren’t the one causes of coaching efficiency gaps, and checking them doesn’t exchange validation with actual purposes.
4 case research under present how these patterns appeared in current coaching work: what sign uncovered the difficulty, what modified, and the way the repair was verified. The order isn’t a common triage sequence; the correct place to begin relies on the workload, platform, and first profiler sign.
Layer: Virtualization and SMMU
A GB200 NVL72 accomplice deployment operating DeepSeek-V3 Combination-of-Consultants (MoE) FP8 pre-training inside a VM was producing iteration occasions 12% to 14% longer than the bare-metal RA. Pre-training recipes for dense fashions like Llama 3 70B ran inside 3% of RA efficiency, whereas DeepSeek-V3 MoE, which points many small kernels per iteration, was the outlier.
Nsight Programs traces captured on the accomplice cluster confirmed considerably increased CPU overhead for tiny kernel areas of the workload. Microbenchmarks focusing on simply the CPU single thread efficiency demonstrated close to equivalent efficiency on the accomplice and RA cluster nodes. This indicated {that a} 30-second perf document -a -g seize on the host, considered with perf report, surfaced an sudden prime body: 24% of CPU cycles spent on arm_smmu_cmdq_issue_cmdlist.
arm_smmu_cmdq_issue_cmdlist is the operate that submits invalidation instructions to the Arm SMMU’s command queue. Underneath virtualization, each map/unmap leading to visitor invalidation traps the host and serializes by way of a single command queue, producing the spinlock competition seen within the profile. Digital Command Queue (VCMDQ) is a characteristic obtainable by way of the Command Queue Virtualization extension to the usual Arm SMMUv3 and permits the visitor to subject SMMU invalidation instructions on to {hardware} with out VM exits.
The repair: Allow CMDQV/VCMDQ within the host kernel on the accomplice cluster and expose it to the visitor. This requires a kernel constructed with the tegra241-cmdqv driver and the corresponding hypervisor assist; current QEMU/libvirt variations have added a cmdqv IOMMU attribute to show it to visitors.
After this modification, linux perf confirmed arm_smmu_cmdq_issue_cmdlist falling out of the highest frames and dTLB miss charges returning to bare-metal parity. The MoE iteration-time hole narrowed to inside RA tolerance from 12%.
The takeaway is that Grace-based virtualized deployments want the VM stack to show the correct SMMU capabilities for memory-mapping-heavy workloads. With CMDQV/VCMDQ enabled within the host kernel and uncovered to the visitor, the platform can keep away from pointless SMMU serialization and return MoE coaching efficiency to inside RA tolerance.
The subsequent layer down is the CPU itself, the place the failure mode seems utterly completely different.
Case examine 2: H100 cluster shedding 12% to CPU competition and NUMA misbinding
Layer: CPU energy and course of placement.
A accomplice’s H100 SXM5 cluster, operating the identical NCCL model and NeMo container as NVIDIA’s HGX RA, was operating Llama 3 70B pre-training 12% slower than reference. Not like the GB200 NVL72 case, this wasn’t a kernel-level subject; every part occurred in consumer house and BIOS.
Two issues stood out:
CPU frequency: turbostat -i 1 throughout coaching confirmed busy cores pegged at 3.0 GHz, regardless of the SKU being rated for 3.8 GHz turbo. Idle cores had been additionally at 3.0 GHz, with C-states sitting in C1 slightly than dropping to C6.
NUMA-remote site visitors: numastat -p confirmed roughly 18% of the coaching course of’s reminiscence accesses going to the distant NUMA node
Root trigger:
The CPU on the accomplice cluster was configured with C-states restricted to C1 in BIOS. It is a frequent “low-latency” default that’s actively flawed for AI coaching workloads. With idle cores held in C1, they continued to attract bundle energy; the busy cores feeding the GPU with kernels couldn’t declare sufficient of the bundle energy funds to hit turbo. Permitting the idle cores to drop to C6 freed energy headroom, enabling the busy cores to climb to three.8 GHz and recuperate roughly 4% on this workload.
The hypervisor housekeeping threads had been pinned to the identical bodily cores because the coaching course of’s information loader staff. Contained in the VM this appeared like sporadic 50–100 ms stalls within the python threads, which then propagated because the lengthy tail in step time. The repair was a cpuset separation: hypervisor and host providers on cores 0–7 and 56–63, coaching processes on the rest.
End result: The 12% hole shrank to three%, with the residual traced to a special NCCL tuning subject lined within the subsequent case examine.
The sample right here is that no single repair recovered the entire hole. The C-state change was the biggest single contributor at ~4%, and the remainder got here from course of isolation by way of NUMA binding. With CPU and virtualization addressed, the following ceiling is the community.
Case examine 3: GB300 NVL72 with NVIDIA ConnectX-8 SuperNIC under-utilizing 1.6 Tbps material
Focus: ConnectX-8 SuperNIC collective tuning
A GB300 NVL72 deployment with NVIDIA ConnectX-8 SuperNICs (1.6 Tbps per node) confirmed a 31% coaching efficiency hole on Nemotron-4 15B pre-training. Single-node throughput appeared wholesome; the hole appeared at 512 GPUs, the place the profiler confirmed uncovered AllGather and ReduceScatter time. That pointed to the collective path on the ConnectX-8 material slightly than compute.
The investigation examined a number of variables with NCCL Exams (nccl-tests), together with iteration rely, UCX/UCC habits, NUMA mapping, NVLS, and NCCL variations. For the workload’s networking efficiency, the related tuning change was narrower: rising NCCL_IB_QPS_PER_CONNECTION to 4 from the default worth of 1.
Nsight Programs hint exhibiting communication overhead uncovered at decrease QPS values, contributing to longer coaching iteration time
Sign was seen in each the workload and the nccl-tests collective measurements. On the NVIDIA reference cluster, the default configuration ran at about 1.09s per iteration. With QPS=4, the identical reference workload improved to about 0.83s. Within the profile, AllGather time dropped from about 375ms to 262ms, and ReduceScatter dropped from about 389ms to 273ms. The comparability run was about 0.76s and used a special NCCL model. The remaining distinction was subsequently partly attributable to an NCCL model mismatch between the comparability and reference environments; aligning the variations narrowed the residual hole additional. As a result of NCCL model adjustments are outdoors the conventional Exemplar tuning scope, the beneficial tuning retains the deployed NCCL model unchanged.
Lesson: Don’t enhance QPS in every single place. QPS is fabric- and workload-dependent. On this GB300 ConnectX-8 workload, QPS=4 improved large-message AllGather and ReduceScatter habits. On different materials or message-size profiles, the identical setting could add CPU overhead with out bettering coaching throughput. The appropriate method is to check the collective on the workload’s actual message sizes, sweep the setting on the goal material, and confirm the outcome within the coaching workload.
Case examine 4: The atmosphere variable that by no means made it inside
In a virtualized B200 deployment, coaching throughput was 13%–53% under the NVIDIA reference although nccl-tests run on the host confirmed anticipated efficiency. Contained in the enroot workload container, AllGather and ReduceScatter had been 2–4× slower, shifting the investigation from material well being to a direct comparability of the NCCL topology configuration seen on the VM and contained in the coaching job.
───────── ─────────
NCCL_TOPO_FILE=/and many others/nccl/topo.xml →NCCL_TOPO_FILE (not propagated)
/and many others/nccl/topo.xml current →/and many others/nccl/topo.xml(not mounted)
↓
NCCL falls again to auto-detection
→ 13–53% under reference
Lesson: Run checks from inside the identical container, launcher, and Slurm allocation that can run the benchmark not from the host. Operating echo $NCCL_TOPO_FILE && cat $NCCL_TOPO_FILE contained in the job container is the quickest sanity examine. If the trail doesn’t resolve, NCCL fails silently with no error making this one of many more durable gaps to diagnose with out realizing the place to look.
Abstract of fixes
Preflight checks earlier than full-scale coaching debug
When a cluster underperforms relative to its NVIDIA reference structure specs, these checks assist rule out frequent platform points earlier than full-scale workload tuning.
Debug early, debug much less
Efficiency gaps between a cloud coaching deployment and the corresponding NVIDIA reference structure are sometimes cumulative with a number of % from CPU energy settings, one other from NUMA or PCT binding, extra from a lacking kernel functionality, container-visible topology, or material configuration. These points are price checking earlier than validation as a result of they’ll flip into costly full-scale debug classes.
On the similar time, preflight diagnostics don’t assure an Exemplar Cloud cross. Some points solely seem within the validation workloads themselves, underneath the precise mannequin, precision, topology, container, launcher, and community situations used for the run. The sensible aim is to take away recognized platform dangers early, then use the coaching workload traces to debug the gaps that solely seem at scale.
