A TensorRT engine construct can take seconds to many minutes. Giant strongly typed fashions, deep tactic search, and a chilly timing cache on a brand-new GPU SKU can depart builders, finish customers, or AI brokers looking at a frozen terminal with no concept whether or not to attend, retry, or kill the method. Most NVIDIA TensorRT integrations report nothing throughout a construct or present no technique to abort early. In a long-running agent workflow, this turns into wasted GPU-hours and caught classes.<\/p>\n
![\"Dark](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image-65.webp\")
TensorRT offers IProgressMonitor, an API for fixing this difficulty, and it has been in NvInfer.h for a number of releases. This tutorial walks via a minimal drop-in implementation for Python and C++, provides a cancel path that responds to Ctrl-C or a programmatic cease sign from an outer occasion loop, and reveals the place to floor the ensuing progress stream so an IDE, a service, or an agent runtime can use it.<\/p>\n
Each code block on this submit is lifted from or modeled on two NVIDIA-maintained OSS samples:<\/p>\n
Python: samples\/python\/simple_progress_monitor\/ (ResNet-50, strongly typed community)<\/p>\n
C++: samples\/sampleProgressMonitor\/ (MNIST)<\/p>\n
What IProgressMonitor offers you<\/h2>\n
IProgressMonitor is an summary base class that TensorRT calls throughout the engine construct. You subclass it and override three strategies. The form is an identical in Python and C++; solely the spelling differs.<\/p>\n A part whose parent_phase is non-null is nested inside one other part, so the monitor sees a tree of progress moderately than a flat record. The implementation have to be thread-safe as a result of TensorRT can name the identical monitor occasion from a number of inside threads.<\/p>\n Wire the monitor to the builder by setting it on the IBuilderConfig. It’s a single name in both language:<\/p>\n config->setProgressMonitor(&myMonitor); \/\/ C++\n<\/p><\/div>\n Learn the diagram from prime to backside. The builder opens the Constructing Engine part with phase_start, then opens Tactic Choice nested inside it with its parent_phase pointing again at Constructing Engine. Because the construct proceeds, the builder calls step_complete (the stable arrows) and your monitor returns a Boolean (the dashed arrows): true lets the construct proceed and false requests cancellation. Within the run proven right here, the monitor returns false at step 47, which is the purple cancel path, and the builder stops issuing new steps and unwinds. It calls phase_finish early on Tactic Choice after which on Constructing Engine, closing each lively part in reverse order.<\/p>\n This tutorial reveals find out how to implement IProgressMonitor in Python and C++, add cancellation via step_complete, and route progress updates to a terminal, IDE, service, or agent runtime.<\/p>\n One NVIDIA GPU.<\/p>\n TensorRT (present OSS launch) and its Python bindings, or a construct of the C++ samples.<\/p>\n Python 3.10 or newer (Python path).<\/p>\n The TensorRT pattern information: ResNet-50 ONNX for Python and MNIST ONNX for C++. Each ship with the sample-data archive or are mounted beneath \/usr\/src\/tensorrt\/information within the official NGC containers.<\/p>\n A terminal that helps ANSI virtual-terminal escapes. Any trendy Linux shell qualifies; Home windows Terminal works if VT is enabled.<\/p>\n The subclass is small. It solely tracks which phases are lively and what number of steps every part incorporates.<\/p>\n @dataclass class RichProgressMonitor(trt.IProgressMonitor): def phase_start(self, phase_name, parent_phase, num_steps): def step_complete(self, phase_name, step) -> bool: def phase_finish(self, phase_name): Two issues to note. First, the Lock isn’t optionally available. TensorRT will name into the monitor from a number of inside threads, and rendering from a thread that doesn\u2019t personal the state will tear the show. Second, step_complete is the one callback that may cease the construct. phase_start returns None, so you can not reject a part earlier than it begins. The earliest cancellation level is the primary step_complete of that part.<\/p>\n The renderer is the half that varies most by surroundings, so this part offers the form and factors to the upstream pattern for the production-grade implementation. The sample is:<\/p>\n The upstream simple_progress_monitor.py renders the same shape with improved color and width handling. The escape sequence x1b[NA moves the cursor up N lines, and x1b[2K clears a line. The first render call writes blank rows; subsequent calls overwrite them in place.<\/p>\n When this monitor is attached, do not redirect stdout to a file or pipe. The escape codes will be written verbatim into the log and make it unreadable. For non-terminal sinks, replace _render() with a structured emitter.<\/p>\n Cancellation is a three-line addition once the monitor exists. Install a SIGINT handler that flips the flag, then let step_complete honor it.<\/p>\n def install_cancel(monitor: RichProgressMonitor): signal.signal(signal.SIGINT, handler)\n<\/p><\/div>\n Wire the monitor and run the builder:<\/p>\n parser = trt.OnnxParser(network, TRT_LOGGER)<\/p>\n with open(onnx_path, “rb”) as f: config = builder.create_builder_config()<\/p>\n monitor = RichProgressMonitor() install_cancel(monitor)<\/p>\n serialized = builder.build_serialized_network(network, config)<\/p>\n if serialized is None: build_serialized_network() returns None on cancellation. The builder unwinds at the next step boundary, usually quickly, but not instantaneously, especially inside a long tactic-search step.<\/p>\n Applications should surface cancellation latency to users. A simple \u201cCancelling\u2026\u201d message during the unwind window goes a long way.<\/p>\n The same flag can be set from any non-signal path, such as an IDE Stop button, an agent timeout, or a CI cancel webhook. Set monitor._cancelled = True, and the build aborts at the next step boundary.<\/p>\n class RichProgressMonitor : public nvinfer1::IProgressMonitor { bool stepComplete(char const* phaseName, void phaseFinish(char const* phaseName) noexcept override { void requestCancel() noexcept { non-public: std::mutex mu_; void render() noexcept; Connect it the identical method:<\/p>\n RichProgressMonitor monitor;<\/p>\n config->setProgressMonitor(&monitor);\n<\/p><\/div>\n std::atomic for the cancel flag issues as a result of requestCancel() could also be referred to as from one other thread or a sign handler. Every little thing else mirrors the Python model.<\/p>\n Determine 3. IProgressMonitor is the only integration level between the builder and an software\u2019s surfaces<\/p>\n The cancel arrow is drawn from the agent runtime for concreteness, however the identical mechanism applies to each sink. A Ctrl-C from the terminal, an IDE Cease button, an HTTP cancel webhook, or an agent timeout all flip the identical monitor._cancelled flag, and the cancel takes impact on the subsequent step_complete return.The place to wire it in actual methods<\/p>\n The terminal is the straightforward case. The fascinating integrations route progress someplace else:<\/p>\n IDE extension: Override _render() to emit $\/progress notifications within the Language Server Protocol, or equal window\/showProgress in protocol. Every part turns into one progress token; step_complete() turns into a report message; phase_finish() turns into finish.<\/p>\n FastAPI \/ HTTP service: Run the construct on a background thread, and have _render() push entries into an asyncio.Queue that the request handler drains by way of Server-Despatched Occasions. The shopper will get a dwell stream; the cancel hook is only a POST \/builds\/{id}\/cancel that calls monitor.requestCancel().<\/p>\n Agent device name: Emit one structured chunk per part transition ({“part”: …, “step”: …, “complete”: …}) into the tool-call stream. The agent runtime renders it within the user-visible hint, and the identical requestCancel() hook is what an agent timeout calls when the construct exceeds the finances. This sample additionally issues for agent runtimes. Lengthy-running builds must be observable and cancelable so brokers can report progress, implement time budgets, and cease cleanly.<\/p>\n In all three circumstances, IProgressMonitor is the fitting boundary. Something above it (rendering, streaming, transport) is application-level; something beneath it (tactic timing, kernel choice) is the builder\u2019s enterprise.<\/p>\n These behaviors are frequent sources of integration bugs:<\/p>\n Don’t redirect stdout whereas the terminal renderer is hooked up. The escape sequences will pollute the log. For non-interactive sinks, swap the renderer for a structured emitter.<\/p>\n phase_start() can\u2019t cancel. It returns None. The earliest cancel level is the primary step_complete() of that part. If the consumer cancels throughout an extended phase_start(), the construct will proceed till step one boundary.<\/p>\n phase_finish() might hearth earlier than all num_steps are reported. This may occur throughout error restoration, builder-internal short-circuits, or when step_complete() returns False. Deal with it because the authoritative end-of-phase sign; don’t assume current_step == num_steps.<\/p>\n Cancel latency is bounded however not zero. The builder finishes the present step earlier than checking the return worth. Lengthy tactic-search steps can push this into the seconds-to-tens-of-seconds vary.<\/p>\n Thread security is required. The identical monitor occasion known as from a number of builder threads; uninstrumented dict or unordered_map entry from _render() will finally crash or tear.<\/p>\n The quickest technique to run this finish to finish is:<\/p>\n![\"UML](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image-67.webp\")
What this tutorial builds<\/h2>\n
Conditions<\/h2>\n
1. Subclass IProgressMonitor in Python<\/h3>\n
\nfrom dataclasses import dataclass, discipline
\nfrom threading import Lock<\/p>\n
\nclass _PhaseState:
\n num_steps: int
\n current_step: int = 0
\n guardian: str | None = None<\/p>\n
\n def __init__(self):
\n tremendous().__init__()
\n self._lock = Lock()
\n self._phases: dict[str, _PhaseState] = {}
\n self._cancelled = False
\n\t self._rendered_lines = 0<\/p>\n
\n with self._lock:
\n \tself._phases[phase_name] = _PhaseState(
\n \tnum_steps=num_steps, guardian=parent_phase
\n \t)
\n \tself._render()<\/p>\n
\n with self._lock:
\n \tif phase_name in self._phases:
\n self._phases[phase_name].current_step = step
\n \tself._render()
\n \treturn not self._cancelled<\/p>\n
\n with self._lock:
\n \tself._phases.pop(phase_name, None)
\n \tself._render()\n<\/p><\/div>\n2. Render nested progress bars with virtual-terminal escapes<\/h3>\n
\n # Order phases by nesting depth so youngsters draw beneath dad and mom.
\n rows = sorted(
\n self._phases.objects(),
\n key=lambda kv: (kv[1].guardian or “”, kv[0]),
\n )
\n # Transfer the cursor up by the variety of traces the PREVIOUS render printed,
\n # not the present row depend \u2014 phases are added on nesting and eliminated on
\n # phase_finish, so the 2 differ precisely when the tree adjustments form.
\n if self._rendered_lines:
\n print(f”x1b[{self._rendered_lines}A”, end=””)
\n for name, st in rows:
\n # step is a 0-based index in [0, num_steps); +1 turns it into a
\n # completed count so the bar can actually reach 100%.
\n done = min(st.current_step + 1, st.num_steps)
\n pct = done \/ max(st.num_steps, 1)
\n bar = “\u2588” * int(40 * pct) + “\u00b7” * (40 – int(40 * pct))
\n indent = ” ” if st.parent else “”
\n print(f”x1b[2K{indent}{name:<28} [{bar}] {performed}\/{st.num_steps}”)
\n # Clear rows left behind when a part finishes and the depend shrinks.
\n for _ in vary(self._rendered_lines – len(rows)):
\n print(“x1b[2K”)
\n self._rendered_lines = len(rows)\n<\/div>\n3. Add a cancel path<\/h3>\n
\n def handler(signum, frame):
\n monitor._cancelled = True
\n print(“nCancelling TensorRT build at next step boundary…”)<\/p>\n
\nnetwork = builder.create_network(
\n 1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
\n)<\/p>\n
\n parser.parse(f.read())<\/p>\n
\nconfig.progress_monitor = monitor<\/p>\n
\n if monitor._cancelled:
\n print(“Build cancelled cleanly.”)
\n else:
\n print(“Build failed.”)\n<\/p><\/div>\n4. The same pattern in C++<\/h3>\n
\n#include
\n#include
\n#include <\/p>\n
\npublic:
\n void phaseStart(char const* phaseName,
\n char const* parentPhase,
\n int32_t nbSteps) noexcept override {
\n std::lock_guard g(mu_);
\n phases_[phaseName] = {nbSteps, 0, parentPhase ? parentPhase : “”};
\n render();
\n }<\/p>\n
\n int32_t step) noexcept override {
\n std::lock_guard g(mu_);
\n auto it = phases_.discover(phaseName);
\n if (it != phases_.finish())
\n it->second.present = step;
\n render();
\n return !cancelled_.load();
\n }<\/p>\n
\n std::lock_guard g(mu_);
\n phases_.erase(phaseName);
\n render();
\n }<\/p>\n
\n cancelled_.retailer(true);
\n }<\/p>\n
\n struct Part {
\n int32_t nbSteps;
\n int32_t present;
\n std::string guardian;
\n };<\/p>\n
\n std::unordered_map phases_;
\n std::atomic cancelled_{false};<\/p>\n
\n};\n<\/p><\/div>\n
\n std::unique_ptr(
\n builder->createBuilderConfig());<\/p>\nThe place to wire it in actual methods<\/h2>\n
![\"Three-layer](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image-68.webp\")
<\/p>\n<\/figure>\nEdge circumstances to deal with<\/h2>\n
Get began<\/h2>\n