AI has reworked how organizations function, driving unprecedented ranges of productiveness and innovation. Nonetheless, AI adoption could be impeded by issues surrounding information privateness, sovereignty and the right way to safe information whereas it’s in use, or throughout inference and engagement with AI fashions. NVIDIA Confidential Computing (CC) was engineered to be a safe and performant resolution for the period of agentic AI to scale any mannequin securely.\u00a0<\/p>\n
CC permits the safety of enterprise information and proprietary mannequin weights and the mannequin itself throughout energetic inference. On this publish, we’ll present an outline of CC and show benchmarks that present its inference efficiency is sort of equivalent (as much as 98%) to options that don\u2019t allow CC safety.\u00a0<\/p>\n
Knowledge, code, and mannequin integrity<\/h2>\n
CC offers a safety layer that spans silicon, interconnect, and system software program. Right here\u2019s the way it works:<\/p>\n
![\"Diagram](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image1.webp\")
{Hardware} root of belief<\/h3>\n
NVIDIA Blackwell GPUs, together with the NVIDIA RTX PRO 6000, HGX B200, and HGX B300, are engineered with CC embedded within the {hardware}. The HGX B200 and HGX B300 GPUs help confidential computing throughout a number of GPUs (as much as 8) with NVIDIA NVLink encryption. On the silicon stage, the GPU maintains a personal signing key that’s fused on the time of producing and by no means uncovered to software program, firmware, or the host system. This key’s the muse of the attestation chain.<\/p>\n
Attestation: Verification earlier than execution<\/h3>\n
Earlier than a confidential workload receives any secrets and techniques, it undergoes distant attestation. The NVIDIA Distant Attestation Service (NRAS) verifies a signed proof bundle\u2014the GPU\u2019s {hardware} report mixed with CPU TEE measurements (AMD SEV-SNP or Intel TDX)\u2014in opposition to a known-good reference integrity manifest (RIM).<\/p>\n
As soon as the Confidential VM (CVM) is in a verified, unmodified state, secrets and techniques equivalent to\u00a0 mannequin decryption keys could be deployed into the CVM. The attestation handshake is usually a one-time startup occasion. As soon as the workload is operating, attestation doesn’t add latency to particular person inference requests.<\/p>\n Determine 2. Attestation companies remotely validate the id, configuration, and integrity of Trusted Execution Environments and challenge cryptographic proof<\/p>\n CC modifications to AI inference efficiency on Blackwell GPUs can come from two areas:\u00a0<\/p>\n Safe work submission latency: \u00a0For inference, safe work submission latency is usually the bigger issue and as a result of added overhead from encryption and kernel launches, smaller items of labor are extra affected. Rising the quantity of labor carried out per GPU work launch reduces the influence of the safe launch overhead.\u00a0<\/p>\n Diminished host-to-device CPU-to-GPU bandwidth: If a workload relies upon closely on transferring inputs to the GPU, efficiency will depend upon whether or not the required bandwidth to maintain the GPU totally utilized exceeds the encrypted switch bandwidth accessible in CC mode.<\/p>\n A number of improvements optimize inference efficiency with CC together with:<\/p>\n CC-safe autotuner timing: FlashInfer replaces occasion timers in CC mode with the GPU world timer register, permitting autotuners to precisely examine kernel candidates and choose the quickest implementation for every form.<\/p>\n Async D2H copy employee: SGLang strikes per-step token readback off the scheduler\u2019s crucial path. This helps restore compute\/copy overlap as a result of CC can in any other case make many host-to-device and device-to-host copies successfully synchronous throughout cudaMemcpyAsync.<\/p>\n Piecewise CUDA graph help: SGLang provides CUDA graph replay for prefill and blended batches, lowering kernel launch overhead that’s amplified in CC mode.<\/p>\n NVIDIA continues to work with upstream communities for inference frameworks to make sure these frameworks are optimized for efficiency.\u00a0<\/p>\n We measured the inference efficiency of CC throughout completely different key metrics. Beneath are the small print on the check setup and measurements.\u00a0<\/p>\n Throughout all workload configurations examined, enabling CC mode produced minimal throughput and time per output token overhead throughout steady-state inference.<\/p>\n The next desk summarizes CC throughput, TTFT, TPOT overhead on Blackwell Extremely (HGX B300) for mannequin Qwen\/Qwen3.5-397B-A17B-FP8<\/p>\n Benchmark: Qwen 3.5 397B-A17B mannequin at FP8 precisionEnvironment: Digital Machine with GPU passthroughBaseline: Confidential Computing OffExperiment: Confidential Computing On<\/p>\n All different variables held fixed.\u00a0<\/p>\n HGX B300 with Blackwell Extremely.\u00a0<\/p>\n Observe: Please observe the CPU energy and vCPU pinning configuration described on this doc.\u00a0<\/p>\n Every configuration was examined throughout a variety of situations consultant of actual enterprise inference workloads:<\/p>\n Enter\/output token lengths: 8192\/1024, 1024\/1024Batch sizes: 4, 8, 16, 32, 64, 128 and 256 concurrent requests.\u00a0Inference framework (Mode): SGLang (Server)Baseline: With out \u2013enable-symm-mem<\/p>\n Output Throughput per GPU (tokens\/sec\/gpu)Median Time to First Token (TTFT) \u2014 latency from request submission to first token generated, in msMedian Time Per Output Token (TPOT) \u2014 per-token technology latency in steady-state streaming, in ms<\/p>\n {Hardware}-level safety with CC protects delicate AI workloads whereas preserving the efficiency wanted for manufacturing AI workloads.\u00a0<\/p>\n CC offers a stronger safety basis for manufacturing inference workloads with minimal efficiency overheads. In our analysis utilizing Qwen 3.5 on SGLang, we noticed\u00a0 this throughout a sweep of concurrency ranges, enter sequence lengths, and output sequence lengths, proving that organizations can safe their AI workloads and information, and keep compliant to regulation with out compromising on efficiency.\u00a0<\/p>\n Be a part of NVIDIA and our companions to safe your AI workloads with CC on Blackwell by accessing the sources beneath.<\/p>\n NVIDIA Confidential Computing DocumentationNVIDIA Blackwell Structure WhitepaperNVIDIA GPU Operator and Container ToolkitNVIDIA Distant Attestation Service (NRAS)NIST SP 800-207 Zero Belief ArchitectureHIPAA Safety Rule (HHS)GDPR Article 32 \u2014 Safety of Processing<\/p>\n<\/div>\n![\"Diagram](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image-1-1-625x267.png\")
<\/p>\n<\/figure>\n<\/figure>\nOptimizing AI inference efficiency in Confidential Computing<\/h2>\n
Benchmark outcomes<\/h2>\n
Relative Efficiency of Confidential Computing<\/h3>\n
Take a look at Setup<\/h2>\n
{Hardware} Configurations<\/h3>\n
Software program Stack<\/h3>\n
Workload Parameters<\/h3>\n
Metrics Collected<\/h3>\n
Path ahead<\/h2>\n
Assets<\/h2>\n