Builders constructing for AR glasses and wearable units face an infrastructure hole. The {hardware} is prepared, however creating AI experiences requires integrating stay digital camera and microphone streams, multimodal AI fashions, enterprise knowledge, device use, deployment infrastructure, and device-specific runtimes.
NVIDIA XR AI is designed to deal with this problem by offering a reusable basis for connecting prolonged actuality (XR) units to GPU-accelerated AI providers working within the cloud, knowledge middle, workstation, or edge.
Now publicly out there in beta, builders have entry to an open supply library for constructing clever brokers for AI glasses, AR glasses, and XR headsets. These clever XR brokers can see what customers see, perceive spoken or typed intent, name enterprise instruments, and reply throughout the identical XR session. They will help frontline workforce members discover the precise info, information staff by procedures, confirm outcomes, and seize the proof.
XR AI brings intelligence to folks the place they work, whether or not in area service, distant help, industrial operations, healthcare, coaching, or different hands-busy environments.
NVIDIA companions in healthcare and manufacturing present helpful examples of how this sample could be utilized. Researchers within the Cong Lab on the Stanford College of Medication and the Wang Lab at Princeton College have explored XR and AI workflows for stem cell remedy analysis, serving to researchers entry contextual info and work together with laboratory techniques whereas remaining centered on advanced procedures.
In manufacturing, Siemens is exploring in a analysis context how NVIDIA XR AI and NVIDIA DGX Spark will help manufacturing unit engineers discover upkeep info, troubleshoot points, confirm work, and seize what occurred on the store flooring.
This put up walks by the method of constructing an clever XR Agent in your use case. It additionally explores how XR AI combines visible grounding utilizing NVIDIA Cosmos, voice-first interplay with NVIDIA Nemotron fashions, enterprise connectivity utilizing Mannequin Context Protocol (MCP), and versatile agent orchestration with frameworks resembling NVIDIA NeMo Agent Toolkit.
Parts and structure of an clever XR Agent
An clever XR Agent begins with stay context from the person’s XR system. Digital camera frames, microphone audio, and knowledge messages movement into the XR Media Hub, the place they are often routed to fashions, instruments, and brokers that perceive the person’s setting and intent. NVIDIA Cosmos fashions present visible grounding; NVIDIA Nemotron fashions present language understanding, reasoning, and gear calling; and MCP servers expose enterprise instruments and knowledge sources. Agent frameworks resembling NVIDIA NeMo Agent Toolkit can orchestrate workflows throughout fashions and instruments, whereas NVIDIA CloudXR can add rendered spatial content material when an utility wants wealthy 3D interplay.
XR AI retains this structure modular by separating media transport, mannequin providers, device entry, agent orchestration, and consumer supply. Video pixels can stay in shared reminiscence whereas light-weight metadata strikes by the system, so brokers retrieve picture knowledge solely when a process requires it. This reduces pointless mannequin inference and knowledge motion whereas letting builders swap shoppers, fashions, MCP servers, orchestration frameworks, and deployment environments with out rebuilding the complete agent.
The identical design additionally helps multi-user and multi-agent situations. Participant id acts because the routing boundary: a number of shoppers can hook up with the identical hub, a number of brokers can observe the identical streams, and every response is routed again to the proper participant. This sample allows one basis to assist visible understanding, voice interplay, enterprise device use, real-time reasoning, context-aware XR responses, and versatile deployment throughout AI glasses, AR glasses, XR headsets, cellular units, internet shoppers, and CloudXR-powered experiences.
Get began
XR AI is now out there in public beta. The next sections stroll by how you should utilize XR AI to shortly get to a working clever XR Agent, together with:
Dwell digital camera, microphone, and system knowledge streams
Actual-time multimodal interplay
Visible grounding by Cosmos-powered VLMs
Voice interplay by speech recognition and Nemotron fashions
Enterprise connectivity by MCP
Searchable visible information seize and retrieval workflows
Elective agent orchestration by NeMo Agent Toolkit or different frameworks
Elective CloudXR-rendered spatial content material
Whereas implementation particulars fluctuate throughout industries, the underlying structure stays largely the identical.
Construct your first clever XR agent with the general public beta
Step 1. Clone the XR AI repository
The GitHub repository consists of pattern brokers, model-server launchers, MCP servers, internet shoppers, XR workflows, and the core media infrastructure. The quickest solution to perceive the system is to begin with a easy multimodal agent after which add capabilities one layer at a time.
Step 2. Begin the AI providers
The bigger examples use shared AI providers that may be began independently:
This begins the mannequin processes utilized by the heavier demos and leaves the weights loaded within the background.
Within the present repository, the mannequin server stack consists of:
nvidia/parakeet-tdt-0.6b-v3 for speech-to-text
nvidia/Cosmos-Reason1-7B for vision-language reasoning
nvidia/Llama-3.1-Nemotron-Nano-8B-v1 for quick, latency-sensitive language responses
NVIDIA-Nemotron-3-Nano-30B-A3B for deeper tool-calling workflows
The agent-sdk/xr-ai-models package deal retains the mannequin layer versatile. Staff reference logical providers resembling llm, agent_llm, vlm, stt, and tts by configuration, letting builders swap endpoints, use cloud-hosted fashions, or deliver OpenAI-compatible APIs with out altering agent logic.
The core AI providers to energy visible understanding, speech recognition, language reasoning, and voice responses are in place.
Step 3. Run a sensor-first XR agent
Begin the only working agent:
When the service begins, it prints an online consumer URL and authentication token.
Open the online consumer, join, and ship a immediate resembling ping or ask a query by the microphone.
The workflow is easy:
The consumer streams digital camera, microphone, and knowledge messages.
XR AI routes media by the XR Media Hub.
Speech is transformed to textual content.
The most recent digital camera body is analyzed utilizing the Cosmos-powered VLM path.
The agent generates a response.
The response returns as each textual content and synthesized audio.
That is now a working clever XR agent. It could pay attention, perceive what the person sees, cause over visible context, and reply by the identical session utilizing each textual content and speech.
Earlier than including enterprise techniques, RAG pipelines, or spatial rendering, this validates an important functionality: real-time multimodal interplay grounded within the person’s setting.
Step 4. Join enterprise knowledge by MCP
Most enterprise brokers want greater than stay notion. A researcher might have protocol steps, experiment metadata, or dataset entry. A area technician might have upkeep data. A producing engineer might have work directions, controller state, or digital-twin info. XR AI makes use of Mannequin Context Protocol (MCP) as the mixing layer for these workflows.
The repository consists of MCP servers for XR-specific capabilities:
vlm-mcp for visible query answering
video-mcp for video evaluation and queries
render-mcp for scene manipulation
oxr-mcp for OpenXR spatial info
vec-mcp for vector and spatial utilities
transcript-mcp for transcript ingestion and retrieval
Builders may construct customized MCP servers for enterprise techniques, retrieval-augmented era (RAG), databases, digital twins, asset-management techniques, and domain-specific workflows.
Many organizations are additionally curious about capturing and understanding visible info from the bodily world. An XR agent can observe procedures, inspections, upkeep actions, or analysis workflows, then use applied sciences resembling NVIDIA Video Search and Summarization (VSS) to index, summarize, and retrieve that info later. Over time, this creates a searchable visible information base that may assist reporting, coaching, compliance, operational critiques, and retrieval-augmented era workflows.
That is the place the agent begins to maneuver past notion and into enterprise motion and organizational reminiscence.
Step 5. Add agent orchestration
The next instance is tailored from the NeMo Agent Toolkit MCP consumer workflow sample. In apply, this configuration would stay inside a NeMo Agent Toolkit workflow definition and allow the agent to find instruments uncovered by XR AI MCP servers.
xr_tools:
_type: mcp_client
server:
transport: streamable-http
url: “http://localhost:8220/mcp”
workflow:
_type: react_agent
tool_names:
– xr_tools
The vital level isn’t the framework, however that XR AI offers a constant basis for real-time media, multimodal notion, and enterprise connectivity whereas enabling builders to decide on the orchestration strategy that most closely fits their setting.
Builders curious about extra superior orchestration workflows ought to evaluation the NeMo Agent Toolkit documentation, which incorporates detailed examples for MCP integration, device calling, multi-agent techniques, and RAG-based workflows.
Step 6. Add CloudXR-rendered spatial experiences
Not each XR workflow requires rendered 3D content material. Some brokers solely want a digital camera, microphone, language, and enterprise instruments. When a workflow advantages from spatial visualization, XR AI can pair the agent layer with NVIDIA CloudXR.
This workflow launches the XR Media Hub, CloudXR runtime, mannequin providers, MCP servers, and an agent employee.
The agent can name rendering instruments by MCP to create, replace, and manipulate objects in a person’s spatial setting. CloudXR streams the ensuing expertise from GPU infrastructure to the consumer system.
The demo additionally reveals a helpful manufacturing sample. A smaller mannequin handles fast acknowledgments and standing updates whereas a bigger mannequin performs deeper reasoning and gear use. Customers obtain quick suggestions whereas extra advanced operations proceed within the background. At this stage, the XR agent can work together with each the bodily setting and rendered spatial content material.
You now have a working clever XR agent, able to customise to your use case. You may as well study extra or attain out to us for a deeper partnership.
