Lengthy-running AI brokers typically spend most of their time on routine execution quite than troublesome reasoning. After making a plan, they might carry out a whole lot of device calls, file reads, validations, instructions, and formatting steps, so utilizing a frontier reasoning mannequin for each motion can turn into unnecessarily sluggish and costly.<\/p>\n
NVIDIA\u2019s Nemotron 3.5 Lightning takes a distinct strategy: a quick, environment friendly mannequin designed for high-volume agent execution. The concept is easy: use the costly mannequin to assume and the quick mannequin to work. On this article, we look at whether or not that structure can cut back price with out sacrificing agentic efficiency.<\/p>\n
What’s NVIDIA Nemotron 3.5 Lightning?<\/h2>\n
Moreover, NVIDIA Nemotron 3.5 Lightning is an open-weight reasoning and instruction mannequin designed primarily for the execution layer of agentic programs.<\/p>\n
Its core specs are:<\/p>\n
NVIDIA\u2019s official NVFP4 mannequin card additionally lists single-GPU deployment on a DGX Spark GB10 or H100, with help spanning Blackwell, Hopper and Ampere {hardware} relying on quantization.<\/p>\n
The mannequin is primarily supposed for English and programming languages, whereas Spanish, French, German, Italian and Japanese are additionally formally supported.<\/p>\n
That is necessary as a result of Nemotron 3.5 Lightning shouldn’t be evaluated as merely \u201cone other 30B mannequin.\u201d<\/p>\n
After all, its supposed job is way more particular.<\/p>\n
Why NVIDIA Constructed an Execution-Targeted Mannequin<\/h2>\n
Take into account a coding agent.<\/p>\n
It might first want to grasp a bug and develop a plan. That could be a troublesome reasoning downside.<\/p>\n
However after the plan exists, the agent might have to:<\/p>\n
![\"Coding](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2026\/08\/agent-loop-flowchart.webp\")
<\/figure>\n<\/div>\nStep one might deserve a frontier mannequin.<\/p>\n
Do all of the others?<\/p>\n
In all probability not.<\/p>\n
NVIDIA argues that long-running brokers spend a considerable portion of their workloads on precisely these high-volume execution operations, akin to device calls, validation and delegation. Utilizing a frontier reasoning mannequin for each execution step will increase each price and latency.<\/p>\n
Briefly, Nemotron 3.5 Lightning is NVIDIA\u2019s reply.<\/p>\n
A potential manufacturing structure turns into:<\/p>\n
![\"AI](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2026\/08\/image-1-xhmgqa.webp\")
<\/figure>\n<\/div>\nSubsequent, this modifications how we must always take into consideration mannequin choice.<\/p>\n
As an alternative of asking:<\/p>\n
Lastly, which single mannequin ought to energy my agent?<\/p>\n
the extra helpful query turns into:<\/p>\n
Equally, which mannequin ought to deal with every kind of labor inside my agent?<\/p>\n
That’s the architectural thought behind Lightning.<\/p>\n
Structure Deep Dive<\/h2>\n
In the meantime, Nemotron 3.5 Lightning makes use of one of many extra attention-grabbing architectures amongst present smaller agent fashions.<\/p>\n
NVIDIA describes it as a hybrid:<\/p>\n
Mamba-2
\n +
\nCombination-of-Specialists
\n +
\nSelective Consideration
\n +
\nMulti-Token Prediction<\/p>\n
The mix issues as a result of every element solves a distinct effectivity downside.<\/p>\n
1. Combination-of-Specialists: 30B Parameters, Solely 3B Lively<\/h3>\n
Nemotron 3.5 Lightning incorporates roughly 30 billion whole parameters however prompts solely round 3 billion for every token.<\/p>\n
In a dense 30B mannequin, basically the entire community participates in inference.<\/p>\n
In an MoE mannequin:<\/p>\n
![\"Mixture](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2026\/08\/image-2-xhmgqa-scaled.webp\")
<\/figure>\n<\/div>\nThen again, the router chooses solely a small subset of consultants.<\/p>\n
You due to this fact retain a lot of the representational capability of a bigger mannequin whereas doing computation nearer to a considerably smaller mannequin.<\/p>\n
That’s central to Lightning\u2019s throughput benefit.<\/p>\n
Printed runtime configuration additionally exposes 128 routed consultants plus a shared knowledgeable, with six routed consultants chosen per token. The configuration incorporates 52 hidden layers. Its hybrid layer sample resolves to Mamba, MoE and sparse Consideration parts quite than utilizing full self-attention at each layer. These are implementation-level configuration particulars, so builders ought to confirm them towards the precise checkpoint and runtime they deploy.<\/p>\n
2. Mamba-2 Layers<\/h3>\n
Conventional Transformers rely closely on consideration.<\/p>\n
Consideration is extraordinarily highly effective, however lengthy sequences turn into computationally costly.<\/p>\n
Though Mamba relies on state-space modeling and might course of sequences extra effectively.<\/p>\n
Nonetheless, Nemotron 3.5 Lightning doesn’t abandon consideration totally. As an alternative, NVIDIA makes use of Mamba-2 for a lot of the sequence processing whereas preserving chosen Consideration layers the place international token interplay stays helpful.<\/p>\n
Conceptually:<\/p>\n This hybrid design is especially related for long-context brokers.<\/p>\n As an alternative of paying full consideration prices all through your complete community, the mannequin mixes mechanisms optimized for various jobs.<\/p>\n Consideration continues to be necessary when tokens should instantly evaluate info throughout distant elements of the sequence.<\/p>\n That issues for:<\/p>\n lengthy paperwork<\/p>\n source-code repositories<\/p>\n multi-step device trajectories<\/p>\n dialog historical past<\/p>\n retrieved paperwork<\/p>\n agent reminiscence<\/p>\n Nemotron due to this fact retains chosen consideration layers as a substitute of switching to a pure state-space structure.<\/p>\n The architectural philosophy is just not \u201cMamba as a substitute of Transformer.\u201d<\/p>\n It’s use costly international consideration solely the place it provides enough worth.<\/p>\n Regular autoregressive LLMs study:<\/p>\n Token 1 \u2192 predict Token 2Token 2 \u2192 predict Token 3Token 3 \u2192 predict Token 4<\/p>\n Nemotron 3.5 Lightning contains Multi-Token Prediction, or MTP, layers that study to foretell a number of future tokens throughout coaching. Furthermore, NVIDIA added a devoted continued-pretraining stage for these MTP layers.<\/p>\n MTP improves coaching alerts, however it additionally turns into helpful throughout inference.<\/p>\n As an alternative of proposing solely:<\/p>\n subsequent token<\/p>\n the system can speculate about:<\/p>\n token t+1token t+2token t+3…<\/p>\n These candidates can then be verified effectively.<\/p>\n Consequently, this is likely one of the mechanisms behind Lightning\u2019s excessive era throughput.<\/p>\n Then again, its velocity doesn’t come from one optimization. It’s the mixture of a number of.<\/p>\n The most suitable choice due to this fact is determined by concurrency.<\/p>\n There isn’t any universally quickest configuration.<\/p>\n MoE Sparsity: 30B parameters present capability, however solely about 3B are lively.<\/p>\n Briefly, Hybrid Mamba Structure: Mamba reduces the necessity to carry out full consideration throughout each layer.<\/p>\n Moreover, NVFP4 Quantization: Decrease-precision inference reduces reminiscence and compute necessities.<\/p>\n As an alternative, Multi-Token Prediction: A number of future tokens could be proposed collectively.<\/p>\n After all, Speculative Decoding: NVIDIA supplies three speculative approaches:<\/p>\n MTP: Built-in instantly into the mannequin. NVIDIA recommends it notably for medium to excessive concurrency.<\/p>\n Specifically, DSpark: A devoted draft mannequin optimized for DGX Spark and lower-concurrency data-center inference.<\/p>\n Consequently, DFlash: A further draft mannequin that builders can benchmark towards MTP and DSpark for his or her workload.<\/p>\n Whereas Nemotron 3.5 Lightning combines robust intelligence with as much as 4x output velocity of similar-sized fashions, putting it on the accuracy-speed Pareto frontier for high-volume agent workloads.<\/p>\n NVIDIA publishes each BF16 and NVFP4 outcomes throughout data, reasoning, coding, brokers, instruction following and lengthy context.<\/p>\n In reality, the necessary statement is that quantization doesn’t dramatically collapse mannequin high quality.<\/p>\n Listed here are the official reported outcomes. Benchmark-native items are preserved, so not each worth must be interpreted as a proportion.<\/p>\n Furthermore, NVIDIA says these evaluations had been run by a constant NeMo Fitness center and NeMo Evaluator-based harness and has printed benchmark recipes for reproducibility.<\/p>\n In distinction, an attention-grabbing result’s how shut NVFP4 stays to BF16.<\/p>\n Nonetheless, pricing is barely extra sophisticated than a single quantity as a result of the mannequin is open-weight and obtainable by a number of routes.<\/p>\n The next displays publicly listed pricing on August 12, 2026.<\/p>\n Entry MethodCurrent CostContextBest ForNVIDIA Construct APIFree prototype endpoint1MTestingOpenRouter free routeFree1MQuick experimentationOpenRouter customary$0.05 enter \/ $0.20 output per 1M tokens262KSimple hosted APIFireworks serverlessSimilarly, $0.05 enter \/ $0.01 cached \/ $0.20 output per 1M262KProduction serverlessOllamaNo per-token mannequin feeRuntime dependentLocal\/personal useSelf-hosted vLLMInfrastructure costUp to 1MEnterprise\/self-hosting<\/p><\/div>\n Subsequent, NVIDIA at the moment gives a free API endpoint for prototyping by construct.nvidia.com.<\/p>\n In the meantime, OpenRouter lists each a free Nemotron 3.5 Lightning route with a 1M context and a normal route at the moment priced at $0.05 per million enter tokens and $0.20 per million output tokens. The usual OpenRouter route at the moment advertises a 262K context quite than the complete 1M mannequin functionality.<\/p>\n Lastly, Fireworks at the moment lists precisely $0.05 per million enter tokens, $0.01 per million cached enter tokens and $0.20 per million output tokens, with a 262K serverless context window.<\/p>\n Pricing and context limits can change shortly, notably in the course of the first weeks after a mannequin launch.<\/p>\n First, at launch, there are already a number of sensible methods to make use of the mannequin.<\/p>\n Go to https:\/\/construct.nvidia.com\/ and login or join<\/p>\n Click on in your profile image after which API keys.<\/p>\n Generate a brand new API key.<\/p>\n Now use this API for inference.<\/p>\n Set up Ollama in your system from <\/p>\n Run the next command in terminal to obtain and run Nemotron 3.5 lightening regionally.<\/p>\n ollama run nemotron-3.5-lightning\u201d<\/p>\n You may also use OpenRouter to run this mannequin. After all, its listed as a Free mannequin on OpenRouter. As an alternative, seize an API key and begin to use it<\/p>\n Nonetheless, NVIDIA exposes the mannequin by an OpenAI-compatible endpoint. The official instance makes use of nvidia\/nemotron-3.5-lightning-30b-a3b.<\/p>\n Set up the shopper:<\/p>\n pip set up openai<\/p>\n Set your API key:<\/p>\n export NVIDIA_API_KEY=”your_api_key”<\/p>\n Now create a easy request:<\/p>\n import os shopper = OpenAI( response = shopper.chat.completions.create( Purchase date: 2025-04-12 Determine whether the claim is within the warranty period. print(response.decisions[0].message.content material)<\/p>\n Output:<\/p>\n {“determination”: “permitted”,”rationale”: “The guarantee interval begins on the acquisition date of 2025-04-12 and lasts for 12 months, ending on 2026-04-12. The declare was submitted on 2026-03-02, which falls inside the lively guarantee interval. Moreover, the harm is listed as a producing defect, which is often lined beneath customary guarantee phrases.”}<\/p>\n This can be a higher first check than asking: Write a poem about AI.<\/p>\n Nemotron 3.5 Lightning is designed for structured agent workloads, so check it accordingly.<\/p>\n NVIDIA\u2019s primary argument is that future manufacturing AI programs might rely much less on a single big mannequin and extra on a coordinated structure of planners, routers, specialised staff, quick execution fashions, and verification layers. This represents a shift from maximizing mannequin dimension to optimizing how completely different fashions work collectively.<\/p>\n In that structure, Nemotron 3.5 Lightning doesn’t have to be the neatest mannequin obtainable. Its worth comes from being environment friendly, quick, and succesful sufficient to deal with most routine agent duties whereas recognizing when more durable work must be escalated. NVIDIA is due to this fact optimizing for sensible, scalable agent execution quite than merely competing for the biggest or most clever mannequin.<\/p>\n![\"Hybrid](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2026\/08\/hybrid-block-stack.webp\")
<\/figure>\n3. Selective Consideration<\/h3>\n
4. Multi-Token Prediction<\/h3>\n
Why Is Nemotron 3.5 Lightning So Quick?<\/h2>\n
![\"\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2026\/08\/serving-path-guidance.webp\")
<\/figure>\n<\/div>\nNVIDIA Nemotron 3.5 Lightning Benchmark Outcomes<\/h2>\n
Nemotron 3.5 Lightning Pricing<\/h2>\n
Working vLLM regionally<\/h3>\n
The right way to Entry NVIDIA Nemotron 3.5 Lightning<\/h2>\n
Choice 1: NVIDIA API<\/h3>\n
Choice 2: Ollama<\/h3>\n
Choice 3: OpenRouter<\/h3>\n
Palms-on: Utilizing Nemotron 3.5 Lightning By means of NVIDIA API<\/h2>\n
\nfrom openai import OpenAI<\/p>\n
\n base_url=”https:\/\/combine.api.nvidia.com\/v1″,
\n api_key=os.environ[“NVIDIA_API_KEY”]
\n)<\/p>\n
\n mannequin=”nvidia\/nemotron-3.5-lightning-30b-a3b”,
\n messages=[
\n {
\n “role”: “user”,
\n “content”: “””
\n A customer has submitted a warranty claim.<\/p>\n
\n Claim date: 2026-03-02
\n Warranty duration: 12 months
\n Damage type: manufacturing defect<\/p>\n
\n Return JSON with:
\n decision
\n rationale
\n “””
\n }
\n ],
\n temperature=1.0,
\n top_p=0.95,
\n max_tokens=2000,
\n extra_body={
\n “chat_template_kwargs”: {
\n “enable_thinking”: True
\n },
\n “reasoning_budget”: 4000
\n }
\n)<\/p>\nConclusion<\/h2>\n
Often Requested Questions<\/h2>\n
![\"Harsh](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_0fBqNLi.webp\")
<\/p><\/div><\/div>\n