Open AI fashions are evolving quicker than ever, however bringing them into native functions can nonetheless require model-specific conversion, preprocessing, post-processing, and runtime code.
NVIDIA TensorRT Mannequin Join open assortment of reference implementations helps to handle this problem. TensorRT Mannequin Join exhibits you the best way to run supported fashions with NVIDIA TensorRT in native C++ functions. You need to use, examine, modify, and prolong the implementations. Mannequin Join is designed to help the open mannequin ecosystem wherever TensorRT runs.
This publish explains what NVIDIA TensorRT Mannequin Join is and the best way to deploy a mannequin from Hugging Face mannequin ID to native C++ inference in two instructions. It additionally covers the 2 API ranges TensorRT Mannequin Join gives, the best way to combine customized GPU kernels, and the way the challenge is constructed to maintain tempo with the open mannequin ecosystem.
How one can deploy a mannequin from mannequin ID to native C++ inference in two instructions
Getting a mannequin into manufacturing mustn’t require deep compiler experience. Mannequin Join splits deployment into two phases with a single artifact between them.
1. Construct the bundle (Python CLI)
For a supported mannequin, the primary section is constructing a deployment bundle from a Hugging Face mannequin ID or native checkpoint:
The bundle incorporates the TensorRT engines and the model-specific belongings wanted at runtime.
2. Load and run (C++)
Within the second section, a local C++ software then masses the bundle and works with task-level inputs and outputs:
auto pipeline = trtmc::load(“qwen3-0.6b.bundle”);
auto consequence = pipeline->generate(“Clarify why native inference issues.”, {.max_new_tokens = 20});
std::cout << consequence.textual content << std::endl;
Mannequin Join handles checkpoint mapping, TensorRT engine building, preprocessing, runtime orchestration, and post-processing. You begin with an entire working implementation as a substitute of rebuilding this integration for each mannequin household.
You need to use Python to arrange the mannequin, however the deployed software runs natively with out requiring PyTorch or a Python interpreter in its manufacturing runtime.
Two API ranges, one place to begin
Mannequin Join gives two ranges of C++ APIs. With the semantic API, you possibly can work with acquainted inputs and outputs, resembling prompts, photos, and audio, whereas Mannequin Join handles model-specific preprocessing, execution, and post-processing.
If you happen to want extra management, the module-level API permits you to work straight with named tensors and particular person TensorRT parts to customise the inference pipeline. Each APIs use the identical Mannequin Join implementations, so you can begin with a easy task-level interface and customise the pipeline solely when wanted.
Lengthen TensorRT Mannequin Join with customized kernels
TVM FFI gives a language-agnostic interface for invoking GPU kernels with out tightly coupling the calling system to the kernel’s implementation framework or runtime. Utilizing TVM FFI by means of TensorRT Mannequin Join, you possibly can substitute a focused portion of a mannequin with a customized GPU kernel whereas TensorRT continues to execute the remainder of the inference pipeline. This makes it simpler to combine specialised or newly developed kernels with out rebuilding the appliance round a separate runtime. See the Deliver Your Personal Kernel tutorial for a labored instance.
Reference implementations for the open mannequin ecosystem
Mannequin Join will not be a brand new inference framework or a substitute for TensorRT. It’s a bridge between the end-to-end inference expertise for open fashions and the power of TensorRT to translate a computation graph into an accelerated engine on GPU.
Every mannequin’s implementation serves three functions:
Operating a supported open mannequin in a local TensorRT-enabled software
Studying from an entire, inspectable implementation of the mannequin and its inference pipeline
Extending the implementation for a associated structure, customized checkpoint, or software requirement
This gives the broader ecosystem with a clearer path to TensorRT deployment from a mannequin ID. Utility builders can start with working code. Group contributors can reuse present patterns so as to add help for brand spanking new fashions as a substitute of ranging from zero.
The aim is easy: wherever TensorRT is out there, you need to have a constant Mannequin Join path for supported open fashions.
Constructed AI-natively to maintain tempo with open fashions
The open mannequin ecosystem adjustments shortly. New architectures and checkpoints seem repeatedly, so a reference library should evolve simply as shortly.
Mannequin Join is constructed as an AI-native software program challenge. Coding brokers generate implementation code, checks, integrations, and documentation underneath human course and assessment. This allows the challenge to develop and validate a number of mannequin implementations in parallel whereas sustaining a constant structure and consumer expertise.
Mannequin Join makes use of nightly releases to shorten the trail from a brand new mannequin, consumer report, or contribution to an accessible implementation. Automated validation stays the discharge gate. The quicker cadence helps new mannequin help, fixes, and UX enhancements attain you sooner.
Delivering the whole TensorRT workflow
Mannequin Join is constructed on TensorRT, so efficiency stays central. For supported and validated workloads, Mannequin Join can ship quicker inference than torch.compile, and every implementation is repeatedly examined and optimized because the challenge evolves.
Efficiency mustn’t come on the expense of usability. Mannequin Join brings the whole workflow collectively: discover the mannequin ID, construct the mannequin, load it from C++, and adapt it when wanted. You get an accessible path to high-performance TensorRT inference whereas retaining the power to examine, customise, and optimize the underlying inference pipeline.
Get began with NVIDIA TensorRT Mannequin Join
Go to the NVIDIA/TensorRT-Mannequin-Join GitHub repo to search out supported implementations and construct a mannequin bundle. Use an implementation as-is, adapt it in your software, or contribute help that helps the following developer carry one other open mannequin to TensorRT.
Need to use an AI-native fast begin that doesn’t require an advanced setup? Open a terminal in any folder you possibly can entry, then paste the next immediate right into a coding agent. You need to have an entire deployment in minutes.
a brand new TensorRT-Mannequin-Join listing within the present workspace. Detect
the present GPU compute functionality, modify the repository improvement Docker
picture, construct and begin the container, set up TensorRT-Mannequin-Join, compile
the CLI, TensorRT backend, and all native mannequin DSOs just for that SM, then
construct and run an end-to-end Qwen/Qwen3-0.6B smoke check. Don’t commit or push
adjustments. Report the results of the check, present precise command, enter and output of
the inference run.
For extra about mannequin protection, structure particulars, and a full developer information, see the TensorRT Mannequin Join documentation.
