AI infrastructure spans a number of layers, from compute and networking to storage, orchestration, and purposes. When efficiency degrades, figuring out the supply could be troublesome as a result of a symptom noticed at one layer could originate elsewhere within the stack.
A full-stack observability technique connects telemetry throughout these layers, serving to infrastructure and operations groups detect issues, isolate their causes, and keep dependable AI workloads. This publish presents a sensible observability framework for NVIDIA AI infrastructure and reveals how you can apply it to frequent monitoring and troubleshooting situations.
Take into account a distributed coaching job that’s three days into execution. GPU utilization and queue wait occasions stay regular. After six hours of diminished throughput, the workforce traces the trigger to a single InfiniBand hyperlink drifting into an elevated bit error price.
This can be a traditional grey failure. The {hardware} is degraded, however the system doesn’t report it as “down.” AI coaching follows the majority synchronous parallel (BSP) mannequin. These tightly coupled techniques are delicate to stragglers: one sluggish rank holds again the job. Hyperlink-level retransmissions stall a single rank throughout synchronous collective operations corresponding to NVIDIA Collective Communications Library (NCCL) all-reduce. Throughput then falls to the slowest rank, and the opposite ranks block. That may be a cascading failure.
You see this failure mode usually in AI factories. The required telemetry usually already exists; the problem is deciding on the best indicators from the best instruments early sufficient to behave. Operators don’t want each metric from each product. They want a call path that maps elements to instruments, instruments to a concise alert set, and that alert set right into a single triage dashboard.
This publish reveals that path. You’ll learn to:
Enumerate the failure domains that have to be observable earlier than deciding on software program.
Map AI infrastructure elements to telemetry instruments sources utilizing a call framework derived from NVIDIA DGX deployments.
Apply the observability framework to an InfiniBand cluster.
Scale back telemetry to a top-k alert set and correlate indicators in a single triage dashboard.
Maintain detailed catalogs, protocol matrices, and per-tool enablement guides in product documentation. Right here, you’ll give attention to how to decide on an observability stack. NVIDIA DGX and NVIDIA HGX deployments share the identical observability floor, even when {hardware} configurations differ (Determine 1):
Establish AI manufacturing unit failure domains
Earlier than deciding on monitoring software program, enumerate the domains during which silent failure consumes GPU hours:
Platform well being: Followers, PSUs, BMC, chassis, CPU, reminiscence, native storage.
GPU well being and efficiency: Utilization, temperature, energy, XID/ECC, and NVIDIA NVLink throughput.
Cloth: InfiniBand or Ethernet hyperlink integrity, congestion, and swap/cable well being; rack-scale NVLink the place current.
Cluster and jobs: Scheduling, reservations, idle allotted GPUs, queue wait.
Inference providers: Latency, success price, and cache habits when NVIDIA NIM microservices or comparable providers are in manufacturing.
In advanced techniques, protection gaps are hardly ever closed in a single move. They seem later, when latent failure modes present up below load. Analyzing these modes early shortens discovery. This evaluation might help forestall failures from recurring below load.
Operations groups want a transparent mapping from element to telemetry supply. Desk 1 maps NVIDIA Knowledge Middle GPU Supervisor (DCGM), NVIDIA System Administration (NVSM), NVIDIA Unified Cloth Supervisor (UFM), NVIDIA NetQ, NVIDIA NMX, NVIDIA Base Command Supervisor (BCM), and NVIDIA Run:ai to these elements. Inexperienced signifies full help for the area; yellow signifies partial or oblique protection. Use the framework to pick out the minimal device set that eliminates protection gaps.
Key tradeoffs embody:
DCGM in comparison with NVSM for GPUs: Desire DCGM for utilization, energy, temperature, NVLink, and XID/ECC export into Prometheus. Retain NVSM for system well being on DGX-class nodes (drives, energy, total well being). The instruments overlap on GPU metrics; neither substitutes for platform BMC information.
UFM in comparison with NetQ: Choose by material sort. InfiniBand makes use of UFM. Spectrum Ethernet/RoCE makes use of NetQ. Deploy each solely when each materials are current.
NMX: Required for rack-scale NVLink. Omit on traditional multi-node NVLink topologies the place DCGM already covers node interconnects.
BCM: Deal with BCM because the aggregator and cluster/job aircraft, not because the supply of low-level counters. Specialised instruments stay chargeable for deep telemetry.
Run:ai and NIM: Introduce when workload scheduling equity or inference SLOs are first-class operational necessities. Neither replaces DCGM or material monitoring.
A helpful rule: cowl each required inexperienced cell with the fewest instruments. Additional exporters and not using a clear triage path add noise, not observability. That noise results in alert fatigue.
Groups maintain including metrics and dashboards, but nonetheless can’t reply what’s damaged and why. The result’s “watermelon metrics”: dashboards that look inexperienced exterior whereas providers fail inside. The corrective precept is the one said in “As Easy as Attainable, No Easier”: maintain monitoring easy, and take away unused indicators slightly than accumulate them.
Apply the observability framework to an InfiniBand cluster
Take into account a DGX cluster with InfiniBand, BCM, and Slurm. Most jobs are coaching; inference isn’t but in manufacturing. The operational requirement is a single triage dashboard and alerts that detect material and GPU well being regressions earlier than multi-hour job waste accumulates.
Right here’s the choice course of:
Domains in scope: platform, GPU, InfiniBand material, cluster/jobs. Out of scope for the preliminary deployment: NetQ, NMX, Run:ai, NIM.
Device choice from the framework
Redfish/IPMI on each node for followers, PSU, chassis, and BMC state.
DCGM on each GPU node for utilization, energy, temperature, XID/ECC, and NVLink.
NVSM on DGX nodes for system well being aggregation.
UFM for InfiniBand port well being, BER, congestion, and routing.
BCM because the cluster aggregator for jobs, reservations, and consolidated {hardware} alerts.
Rationale: DCGM alone would miss the BER regression described within the introduction. This can be a tail-at-scale downside. When work is synchronized throughout many ranks, end-to-end throughput follows the slowest rank, not common element well being. UFM alone would miss GPU XID storms and node energy faults. BCM alone wouldn’t generate the low-level counters. Mixed, they cowl the domains that waste GPU hours on this atmosphere.
Exclusions: Skip NetQ, NMX, and inference metrics till Ethernet, rack-scale NVLink, or inference providers are launched.
That provides you an preliminary stack of IPMI, DCGM, NVSM, UFM, and BCM, unified in Prometheus/Grafana.
Construct an actionable AI infrastructure alert set
Most instruments expose a whole lot of metrics. Desire a brief top-k set tied to service-level indicators (SLIs) and service-level targets (SLOs), not a dump of each {hardware} counter. Every alert ought to map to a transparent remediation motion. UFM Telemetry exposes a whole lot of fields; begin with the documented high-frequency telemetry fields.
For the apply the observability framework to an InfiniBand cluster, start with:
Platform: fan pace, PSU standing, key temperatures (SPD_FAN_*, PWR_*, TEMP_* by way of Redfish/IPMI).
GPU: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_XID_ERRORS, plus NVSM GPU/system well being.
InfiniBand: PortXmitDataExtended, SymbolErrorCounterExtended, Effective_BER, Total_Raw_BER, Chip_Temp.
Jobs: BCM/Slurm indicators for working jobs, GPU reservations, and wait time so material or GPU alerts correlate to workload influence.
Broaden the set solely when an incident demonstrates a protection hole. Per Signs Versus Causes, alert on signs that map to an outlined motion (drain node, substitute cable, open material case), not on each counter a collector can emit.
Desire Prometheus exporters the place out there: DCGM and NVSM each expose Prometheus endpoints; UFM and BCM can feed the identical scrape mannequin by way of exporters or APIs. This retains protocol choice easy for the preliminary deployment and avoids introducing a second management aircraft earlier than gNMI or SNMP integration is required.
Construct a unified AI infrastructure triage dashboard
With instruments and top-k metrics chosen, construct a unified AI infrastructure triage dashboard (Determine 2):
A sensible structure seems like this:
Set up IPMI and DCGM exporters on each GPU node.
Run UFM Telemetry the place the material is reachable; scrape or export into Prometheus.
Retain BCM because the cluster administration and aggregation aircraft.
Level Grafana at Prometheus for dashboards and alerts throughout GPU, node, and material indicators.
Add a neighborhood Slurm dashboard if job-level context isn’t already out there in BCM.
Use a two-layer monitoring strategy. Layer 1 offers a high-level view for triage; Layer 2 retains the element wanted to examine particular person elements. Layer 1 is the Grafana dashboard used first: is the fault within the GPU, node, or material? Layer 2 is the seller UI deep dive (UFM net UI, BCM Base View, and comparable instruments) as soon as the failing area is thought. Day 2 operations is quicker when Layer 1 solutions the triage query from one board. Layer 2 stays out there for root-cause evaluation.
Outline observability acceptance standards
You’re prepared to maneuver on when:
Each failure area in scope has at the least one full-support device from the framework.
Alerts are sure to a brief top-k metric listing with house owners and actions.
GPU, node, and material indicators share a typical timeline in a single view.
Additional instruments (Ethernet, rack-scale NVLink, Run:ai, NIM) are added solely when wanted, not by default.
Don’t measure observability maturity by the variety of dashboards. Measure it by whether or not your indicators present the failing element and the following motion earlier than important compute capability is wasted.
Broaden the NVIDIA AI infrastructure observability stack
After your preliminary deployment meets the outline observability acceptance standards , develop protection on this order:
Use the choice framework to justify every addition. Maintain detailed metric dictionaries and protocol matrices in runbooks or product docs. Maintain the manufacturing alert set brief sufficient for on-call use.
A choice framework beats a metric catalog. It’s what will get you to some well-chosen indicators and one triage board as an alternative of fifty dashboards no one reads.
3-step rollout guidelines:
Set up protection: Choose one full-support device for every in-scope area (see Desk 1).
Combine exporters: Wire Redfish/IPMI, DCGM, NVSM, UFM, and BCM into Prometheus, and level Grafana on the unified telemetry endpoint.
Implement possession: Bind each alert to an proprietor and a playbook motion earlier than including the following exporter.
Don’t measure observability maturity by the variety of dashboards. Measure it by whether or not your indicators title the failing element and the following motion earlier than important compute capability is wasted.
