AI has reworked how organizations function, driving unprecedented ranges of productiveness and innovation. Nonetheless, AI adoption could be impeded by issues surrounding information privateness, sovereignty and the right way to safe information whereas it’s in use, or throughout inference and engagement with AI fashions. NVIDIA Confidential Computing (CC) was engineered to be a safe and performant resolution for the period of agentic AI to scale any mannequin securely.
CC permits the safety of enterprise information and proprietary mannequin weights and the mannequin itself throughout energetic inference. On this publish, we’ll present an outline of CC and show benchmarks that present its inference efficiency is sort of equivalent (as much as 98%) to options that don’t allow CC safety.
Knowledge, code, and mannequin integrity
CC offers a safety layer that spans silicon, interconnect, and system software program. Right here’s the way it works:
{Hardware} root of belief
NVIDIA Blackwell GPUs, together with the NVIDIA RTX PRO 6000, HGX B200, and HGX B300, are engineered with CC embedded within the {hardware}. The HGX B200 and HGX B300 GPUs help confidential computing throughout a number of GPUs (as much as 8) with NVIDIA NVLink encryption. On the silicon stage, the GPU maintains a personal signing key that’s fused on the time of producing and by no means uncovered to software program, firmware, or the host system. This key’s the muse of the attestation chain.
Attestation: Verification earlier than execution
Earlier than a confidential workload receives any secrets and techniques, it undergoes distant attestation. The NVIDIA Distant Attestation Service (NRAS) verifies a signed proof bundle—the GPU’s {hardware} report mixed with CPU TEE measurements (AMD SEV-SNP or Intel TDX)—in opposition to a known-good reference integrity manifest (RIM).
As soon as the Confidential VM (CVM) is in a verified, unmodified state, secrets and techniques equivalent to mannequin decryption keys could be deployed into the CVM. The attestation handshake is usually a one-time startup occasion. As soon as the workload is operating, attestation doesn’t add latency to particular person inference requests.
Determine 2. Attestation companies remotely validate the id, configuration, and integrity of Trusted Execution Environments and challenge cryptographic proof
Optimizing AI inference efficiency in Confidential Computing
CC modifications to AI inference efficiency on Blackwell GPUs can come from two areas:
Safe work submission latency: For inference, safe work submission latency is usually the bigger issue and as a result of added overhead from encryption and kernel launches, smaller items of labor are extra affected. Rising the quantity of labor carried out per GPU work launch reduces the influence of the safe launch overhead.
Diminished host-to-device CPU-to-GPU bandwidth: If a workload relies upon closely on transferring inputs to the GPU, efficiency will depend upon whether or not the required bandwidth to maintain the GPU totally utilized exceeds the encrypted switch bandwidth accessible in CC mode.
A number of improvements optimize inference efficiency with CC together with:
CC-safe autotuner timing: FlashInfer replaces occasion timers in CC mode with the GPU world timer register, permitting autotuners to precisely examine kernel candidates and choose the quickest implementation for every form.
Async D2H copy employee: SGLang strikes per-step token readback off the scheduler’s crucial path. This helps restore compute/copy overlap as a result of CC can in any other case make many host-to-device and device-to-host copies successfully synchronous throughout cudaMemcpyAsync.
Piecewise CUDA graph help: SGLang provides CUDA graph replay for prefill and blended batches, lowering kernel launch overhead that’s amplified in CC mode.
NVIDIA continues to work with upstream communities for inference frameworks to make sure these frameworks are optimized for efficiency.
We measured the inference efficiency of CC throughout completely different key metrics. Beneath are the small print on the check setup and measurements.
Benchmark outcomes
Throughout all workload configurations examined, enabling CC mode produced minimal throughput and time per output token overhead throughout steady-state inference.
The next desk summarizes CC throughput, TTFT, TPOT overhead on Blackwell Extremely (HGX B300) for mannequin Qwen/Qwen3.5-397B-A17B-FP8
Relative Efficiency of Confidential Computing
Take a look at Setup
Benchmark: Qwen 3.5 397B-A17B mannequin at FP8 precisionEnvironment: Digital Machine with GPU passthroughBaseline: Confidential Computing OffExperiment: Confidential Computing On
All different variables held fixed.
{Hardware} Configurations
HGX B300 with Blackwell Extremely.
Software program Stack
Observe: Please observe the CPU energy and vCPU pinning configuration described on this doc.
Workload Parameters
Every configuration was examined throughout a variety of situations consultant of actual enterprise inference workloads:
Enter/output token lengths: 8192/1024, 1024/1024Batch sizes: 4, 8, 16, 32, 64, 128 and 256 concurrent requests. Inference framework (Mode): SGLang (Server)Baseline: With out –enable-symm-mem
Metrics Collected
Output Throughput per GPU (tokens/sec/gpu)Median Time to First Token (TTFT) — latency from request submission to first token generated, in msMedian Time Per Output Token (TPOT) — per-token technology latency in steady-state streaming, in ms
Path ahead
{Hardware}-level safety with CC protects delicate AI workloads whereas preserving the efficiency wanted for manufacturing AI workloads.
CC offers a stronger safety basis for manufacturing inference workloads with minimal efficiency overheads. In our analysis utilizing Qwen 3.5 on SGLang, we noticed this throughout a sweep of concurrency ranges, enter sequence lengths, and output sequence lengths, proving that organizations can safe their AI workloads and information, and keep compliant to regulation with out compromising on efficiency.
Be a part of NVIDIA and our companions to safe your AI workloads with CC on Blackwell by accessing the sources beneath.
Assets
NVIDIA Confidential Computing DocumentationNVIDIA Blackwell Structure WhitepaperNVIDIA GPU Operator and Container ToolkitNVIDIA Distant Attestation Service (NRAS)NIST SP 800-207 Zero Belief ArchitectureHIPAA Safety Rule (HHS)GDPR Article 32 — Safety of Processing
