A TensorRT engine construct can take seconds to many minutes. Giant strongly typed fashions, deep tactic search, and a chilly timing cache on a brand-new GPU SKU can depart builders, finish customers, or AI brokers looking at a frozen terminal with no concept whether or not to attend, retry, or kill the method. Most NVIDIA TensorRT integrations report nothing throughout a construct or present no technique to abort early. In a long-running agent workflow, this turns into wasted GPU-hours and caught classes.
TensorRT offers IProgressMonitor, an API for fixing this difficulty, and it has been in NvInfer.h for a number of releases. This tutorial walks via a minimal drop-in implementation for Python and C++, provides a cancel path that responds to Ctrl-C or a programmatic cease sign from an outer occasion loop, and reveals the place to floor the ensuing progress stream so an IDE, a service, or an agent runtime can use it.
Each code block on this submit is lifted from or modeled on two NVIDIA-maintained OSS samples:
Python: samples/python/simple_progress_monitor/ (ResNet-50, strongly typed community)
C++: samples/sampleProgressMonitor/ (MNIST)
What IProgressMonitor offers you
IProgressMonitor is an summary base class that TensorRT calls throughout the engine construct. You subclass it and override three strategies. The form is an identical in Python and C++; solely the spelling differs.
A part whose parent_phase is non-null is nested inside one other part, so the monitor sees a tree of progress moderately than a flat record. The implementation have to be thread-safe as a result of TensorRT can name the identical monitor occasion from a number of inside threads.
Wire the monitor to the builder by setting it on the IBuilderConfig. It’s a single name in both language:
config->setProgressMonitor(&myMonitor); // C++
Learn the diagram from prime to backside. The builder opens the Constructing Engine part with phase_start, then opens Tactic Choice nested inside it with its parent_phase pointing again at Constructing Engine. Because the construct proceeds, the builder calls step_complete (the stable arrows) and your monitor returns a Boolean (the dashed arrows): true lets the construct proceed and false requests cancellation. Within the run proven right here, the monitor returns false at step 47, which is the purple cancel path, and the builder stops issuing new steps and unwinds. It calls phase_finish early on Tactic Choice after which on Constructing Engine, closing each lively part in reverse order.
What this tutorial builds
This tutorial reveals find out how to implement IProgressMonitor in Python and C++, add cancellation via step_complete, and route progress updates to a terminal, IDE, service, or agent runtime.
Conditions
One NVIDIA GPU.
TensorRT (present OSS launch) and its Python bindings, or a construct of the C++ samples.
Python 3.10 or newer (Python path).
The TensorRT pattern information: ResNet-50 ONNX for Python and MNIST ONNX for C++. Each ship with the sample-data archive or are mounted beneath /usr/src/tensorrt/information within the official NGC containers.
A terminal that helps ANSI virtual-terminal escapes. Any trendy Linux shell qualifies; Home windows Terminal works if VT is enabled.
1. Subclass IProgressMonitor in Python
The subclass is small. It solely tracks which phases are lively and what number of steps every part incorporates.
from dataclasses import dataclass, discipline
from threading import Lock
@dataclass
class _PhaseState:
num_steps: int
current_step: int = 0
guardian: str | None = None
class RichProgressMonitor(trt.IProgressMonitor):
def __init__(self):
tremendous().__init__()
self._lock = Lock()
self._phases: dict[str, _PhaseState] = {}
self._cancelled = False
self._rendered_lines = 0
def phase_start(self, phase_name, parent_phase, num_steps):
with self._lock:
self._phases[phase_name] = _PhaseState(
num_steps=num_steps, guardian=parent_phase
)
self._render()
def step_complete(self, phase_name, step) -> bool:
with self._lock:
if phase_name in self._phases:
self._phases[phase_name].current_step = step
self._render()
return not self._cancelled
def phase_finish(self, phase_name):
with self._lock:
self._phases.pop(phase_name, None)
self._render()
Two issues to note. First, the Lock isn’t optionally available. TensorRT will name into the monitor from a number of inside threads, and rendering from a thread that doesn’t personal the state will tear the show. Second, step_complete is the one callback that may cease the construct. phase_start returns None, so you can not reject a part earlier than it begins. The earliest cancellation level is the primary step_complete of that part.
2. Render nested progress bars with virtual-terminal escapes
The renderer is the half that varies most by surroundings, so this part offers the form and factors to the upstream pattern for the production-grade implementation. The sample is:
# Order phases by nesting depth so youngsters draw beneath dad and mom.
rows = sorted(
self._phases.objects(),
key=lambda kv: (kv[1].guardian or “”, kv[0]),
)
# Transfer the cursor up by the variety of traces the PREVIOUS render printed,
# not the present row depend — phases are added on nesting and eliminated on
# phase_finish, so the 2 differ precisely when the tree adjustments form.
if self._rendered_lines:
print(f”x1b[{self._rendered_lines}A”, end=””)
for name, st in rows:
# step is a 0-based index in [0, num_steps); +1 turns it into a
# completed count so the bar can actually reach 100%.
done = min(st.current_step + 1, st.num_steps)
pct = done / max(st.num_steps, 1)
bar = “█” * int(40 * pct) + “·” * (40 – int(40 * pct))
indent = ” ” if st.parent else “”
print(f”x1b[2K{indent}{name:<28} [{bar}] {performed}/{st.num_steps}”)
# Clear rows left behind when a part finishes and the depend shrinks.
for _ in vary(self._rendered_lines – len(rows)):
print(“x1b[2K”)
self._rendered_lines = len(rows)
The upstream simple_progress_monitor.py renders the same shape with improved color and width handling. The escape sequence x1b[NA moves the cursor up N lines, and x1b[2K clears a line. The first render call writes blank rows; subsequent calls overwrite them in place.
When this monitor is attached, do not redirect stdout to a file or pipe. The escape codes will be written verbatim into the log and make it unreadable. For non-terminal sinks, replace _render() with a structured emitter.
3. Add a cancel path
Cancellation is a three-line addition once the monitor exists. Install a SIGINT handler that flips the flag, then let step_complete honor it.
def install_cancel(monitor: RichProgressMonitor):
def handler(signum, frame):
monitor._cancelled = True
print(“nCancelling TensorRT build at next step boundary…”)
signal.signal(signal.SIGINT, handler)
Wire the monitor and run the builder:
network = builder.create_network(
1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
)
parser = trt.OnnxParser(network, TRT_LOGGER)
with open(onnx_path, “rb”) as f:
parser.parse(f.read())
config = builder.create_builder_config()
monitor = RichProgressMonitor()
config.progress_monitor = monitor
install_cancel(monitor)
serialized = builder.build_serialized_network(network, config)
if serialized is None:
if monitor._cancelled:
print(“Build cancelled cleanly.”)
else:
print(“Build failed.”)
build_serialized_network() returns None on cancellation. The builder unwinds at the next step boundary, usually quickly, but not instantaneously, especially inside a long tactic-search step.
Applications should surface cancellation latency to users. A simple “Cancelling…” message during the unwind window goes a long way.
The same flag can be set from any non-signal path, such as an IDE Stop button, an agent timeout, or a CI cancel webhook. Set monitor._cancelled = True, and the build aborts at the next step boundary.
4. The same pattern in C++
#include
#include
#include
class RichProgressMonitor : public nvinfer1::IProgressMonitor {
public:
void phaseStart(char const* phaseName,
char const* parentPhase,
int32_t nbSteps) noexcept override {
std::lock_guard g(mu_);
phases_[phaseName] = {nbSteps, 0, parentPhase ? parentPhase : “”};
render();
}
bool stepComplete(char const* phaseName,
int32_t step) noexcept override {
std::lock_guard g(mu_);
auto it = phases_.discover(phaseName);
if (it != phases_.finish())
it->second.present = step;
render();
return !cancelled_.load();
}
void phaseFinish(char const* phaseName) noexcept override {
std::lock_guard g(mu_);
phases_.erase(phaseName);
render();
}
void requestCancel() noexcept {
cancelled_.retailer(true);
}
non-public:
struct Part {
int32_t nbSteps;
int32_t present;
std::string guardian;
};
std::mutex mu_;
std::unordered_map phases_;
std::atomic cancelled_{false};
void render() noexcept;
};
Connect it the identical method:
std::unique_ptr(
builder->createBuilderConfig());
RichProgressMonitor monitor;
config->setProgressMonitor(&monitor);
std::atomic for the cancel flag issues as a result of requestCancel() could also be referred to as from one other thread or a sign handler. Every little thing else mirrors the Python model.
The place to wire it in actual methods
Determine 3. IProgressMonitor is the only integration level between the builder and an software’s surfaces
The cancel arrow is drawn from the agent runtime for concreteness, however the identical mechanism applies to each sink. A Ctrl-C from the terminal, an IDE Cease button, an HTTP cancel webhook, or an agent timeout all flip the identical monitor._cancelled flag, and the cancel takes impact on the subsequent step_complete return.The place to wire it in actual methods
The terminal is the straightforward case. The fascinating integrations route progress someplace else:
IDE extension: Override _render() to emit $/progress notifications within the Language Server Protocol, or equal window/showProgress in protocol. Every part turns into one progress token; step_complete() turns into a report message; phase_finish() turns into finish.
FastAPI / HTTP service: Run the construct on a background thread, and have _render() push entries into an asyncio.Queue that the request handler drains by way of Server-Despatched Occasions. The shopper will get a dwell stream; the cancel hook is only a POST /builds/{id}/cancel that calls monitor.requestCancel().
Agent device name: Emit one structured chunk per part transition ({“part”: …, “step”: …, “complete”: …}) into the tool-call stream. The agent runtime renders it within the user-visible hint, and the identical requestCancel() hook is what an agent timeout calls when the construct exceeds the finances. This sample additionally issues for agent runtimes. Lengthy-running builds must be observable and cancelable so brokers can report progress, implement time budgets, and cease cleanly.
In all three circumstances, IProgressMonitor is the fitting boundary. Something above it (rendering, streaming, transport) is application-level; something beneath it (tactic timing, kernel choice) is the builder’s enterprise.
Edge circumstances to deal with
These behaviors are frequent sources of integration bugs:
Don’t redirect stdout whereas the terminal renderer is hooked up. The escape sequences will pollute the log. For non-interactive sinks, swap the renderer for a structured emitter.
phase_start() can’t cancel. It returns None. The earliest cancel level is the primary step_complete() of that part. If the consumer cancels throughout an extended phase_start(), the construct will proceed till step one boundary.
phase_finish() might hearth earlier than all num_steps are reported. This may occur throughout error restoration, builder-internal short-circuits, or when step_complete() returns False. Deal with it because the authoritative end-of-phase sign; don’t assume current_step == num_steps.
Cancel latency is bounded however not zero. The builder finishes the present step earlier than checking the return worth. Lengthy tactic-search steps can push this into the seconds-to-tens-of-seconds vary.
Thread security is required. The identical monitor occasion known as from a number of builder threads; uninstrumented dict or unordered_map entry from _render() will finally crash or tear.
Get began
The quickest technique to run this finish to finish is:
cd TensorRT/samples/python/simple_progress_monitor
python3 simple_progress_monitor.py
This begins a dwell, animated construct of a ResNet-50 engine. Substitute simple_progress_monitor.py‘s monitor class with the model above or connect a cancel handler across the present class. C++ equal is on the market in samples/sampleProgressMonitor/.
For bigger methods, the fitting subsequent step is changing the terminal renderer with the transport the appliance already makes use of comparable to Language Server Protocol notifications, server-sent occasions, or structured tool-call chunks. IProgressMonitor turns into the purpose the place TensorRT construct progress is translated into the appliance’s progress mannequin.
Study extra
Confer with the next assets for extra info:
