NVIDIA Holoscan is a platform for constructing real-time AI purposes on the edge, from medical imaging to robotics. HoloHub is its companion repository: a rising assortment of reference purposes and parts that display what’s potential.
We needed to discover how a general-purpose coding agent might use the identical examples, documentation, and growth instruments obtainable to an engineer in an precise growth activity.
On this submit, we stroll via constructing a real-time endoscopic software segmentation software utilizing an AI coding agent. HoloHub examples and documentation present implementation patterns, and the event expertise information the agent via the HoloHub growth course of.
The Holoscan CLI, invoked through ./holohub wrapper, gives the shared execution interface. The agent can uncover growth operations via the CLI, whereas the engineer can examine and repeat the identical instructions.
The event workflow proceeds in iterations:
The engineer defines a aim and constraints
The coding agent inspects related examples, implements the application-specific code, and makes use of CLI to run the required growth operations
The engineer evaluations the code, outputs, and assessments, then units the aim for the following iteration
The workflow is agent-agnostic; on this instance, we used Codex with GPT-5.6 sol max mode, and the agent processing occasions talked about had been approximate.
Setup and growth goal
The general goal is an end-to-end endoscopic software segmentation software: real-time inference with a reside visualization of the segmentation masks, together with statistical evaluation rendering.
We reused the present MONAI endoscopic software segmentation mannequin and a Holoscan pattern video, and confirmed that the present software monai_endoscopic_tool_seg labored regionally. We centered on a brand new software reusing the deep studying segmentation pipeline, and including complete visualization, runtime telemetry and repeatable benchmarking.
The brokers had been moreover supplied with:
Holoscan CLI with Bash execution permission
HoloHub repository with documentation in a progressive disclosure sample through the brokers.md
HoloHub growth expertise, together with holohub-app-lifecycle and holohub-debug-build-run
The subsequent sections present how the items work collectively in an engineer-guided, agentic growth workflow.
Iteration 0: Divide the aim
As a substitute of making an attempt to construct all the software with a single immediate, builders ought to decompose the last word goal into smaller, verifiable engineering iterations guided by uncertainty and proof. This method ensures that design selections are reviewed in a well timed method.
The great aim can thus be structured right into a sequence of reviewable subjects:
Is the event setting configured appropriately to run an analogous present software regionally?
Can the present mannequin and video run in a separate end-to-end software?
Does the visualization current significant data?
Can latency be measured repeatedly?
Can rendering throughput be improved with out characteristic regressions?
Every iteration produces reviewable code, outputs, and assessments, which inform the prompts and design selections for the following iteration.
Iteration 1: Create a minimal working software
The primary immediate outlined the result whereas constraining mannequin and knowledge reuse.
Developer immediate 1:
Use $holohub-app-lifecycle to create a separate new Python HoloHub software for displaying endoscopic software monitoring as mannequin outputs https://github.com/Undertaking-MONAI/model-zoo/tree/dev/fashions/endoscopic_tool_segmentation. Reuse the MONAI endoscopic software segmentation mannequin, pattern knowledge, preprocessing, and inference. Present the model-derived masks, protection and timeline, and helpful uncertainty measurements in a sophisticated HoloViz overlay. Don’t prepare or modify the mannequin weights. Make the pattern video work finish to finish.
That left implementation selections to the agent whereas holding mannequin reuse, visible proof, and weight integrity express.
The brokers collected data from the designated sources: learn the app lifecycle ability, close by HoloHub examples, challenge metadata, and CLI documentation.
Several types of actions had been taken as anticipated:
Inspected the related endoscopy, segmentation, HoloViz, recording, and testing patterns; monai_endoscopic_tool_seg, endoscopy_tool_tracking, surgical_scene_recon had been significantly helpful references
Dry-ran and invoked ./holohub create to generate and register the usual scaffold
Applied the appliance graph, execution modes, assessments and documentation utilizing present Holoscan operators and belongings
Constructed and ran the appliance through ./holohub run with the appliance metadata outlined by the CLI
The ensuing software linked video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz. Inference and masks postprocessing ran for each replayed body. The overlay reported frame-derived measurements.
The agentic processing time was 40 minutes. The developer might evaluation the reside app via the identical CLI utilized by the agent:
After reviewing the implementation, the visible outputs, and the check circumstances, we confirmed that the reused mannequin and pattern video labored within the new software. Visible evaluation additionally confirmed that the overlay wanted clearer measurements, so we proceed to outline the following iteration.
Iteration 2: Make future evaluations repeatable by implementing benchmarking
The second immediate turned the visible demonstration right into a repeatable growth artifact:
Developer immediate 2:
Revise the visible output, add extra significant statistics, software space, masks movement, temporal intersection-over-union as a stability indicator, edge entropy, FPS, bounding field place, and take away values that stay unchanged throughout replay. Add a benchmark mode that data precise latency and plots the ends in Python. Export the figures to the construct folder and likewise present them interactively when the setting helps it.
In response, the agent revised the dynamic measurements and screenshot readability, then made visible evaluation and benchmarking express software modes. The app was applied to have three named software modes:
With the Holoscan CLI and software lifecycle administration, the modes and assessments are discoverable and runnable with out remembering an in depth container and software script recipe:
./holohub run endoscopy_tool_segmentation_dashboard benchmark –language python
./holohub check endoscopy_tool_segmentation_dashboard –language python
Benchmark mode used Holoscan Knowledge Movement Monitoring for the configured path from the video replayer via preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink. It successfully reused the concepts offered within the present holoscan circulation benchmarking module. The agentic processing time was 20 minutes.
The revised benchmark offered extra repeatable measurements. With that baseline obtainable, the following iteration might examine efficiency with out counting on visible checks alone.
Iteration 3: Examine and enhance latency
As soon as the appliance was measurable, the developer issued a 3rd immediate:
Developer immediate 3:
Verify whether or not the deep-learning mannequin runs on each body. Examine methods to scale back latency, together with approaches that reap the benefits of comparable neighboring segmentations, and present the brand new benchmark outcomes.
In response, the agent confirmed that inference nonetheless ran on each body. It thought-about reusing masks throughout neighboring frames, which might keep away from some inference work however would require a coverage for deciding when stale output is appropriate. For this engineering iteration, it stored inference on each body and first eliminated lower-risk dashboard overhead:
Reused HoloViz enter specs and static coordinate tensors whereas refreshing textual content and dynamic geometry for each body.
Queued the present 10-value GPU-to-host telemetry copy asynchronously in two pinned buffers and rendered with the earlier accomplished values.
We in contrast the primary iteration’s implementation with the ultimate one on the identical check system. The optimized model was sooner in all 5 measured trials. The agentic processing time was half-hour.
Last handoff immediate and revalidation
After the three engineering iterations, the developer gave the agent a separate handoff immediate.
Developer immediate 4:
To finalize the progress, we reran the appliance and the assessments, verified the output figures and headless check outcomes. The implementation and benchmark proof had been retained together with the commit hash and dependency variations through ./holohub env-check and ./holohub env-info.
The event iterations present what the workflow produces. To higher perceive the impression of the CLI, expertise and documentation, we subsequent evaluate the identical growth activity beneath completely different combos of those sources.
Ablation examine
We in contrast useful resource utilization utilizing the identical coding agent and sandbox setting throughout an identical settings (notice that the examine was final performed on Aug. 1, utilizing Codex 0.146.0 with GPT-5.6 Sol at max reasoning effort). All evaluations had been based mostly on the only immediate used within the first iteration (with the ability title eliminated if unavailable) with the aim of making a brand new software.
CLI + expertise + docs/examples (this weblog)
The agentic processing time was 40 minutes, with a complete value of 11M tokens.
CLI + docs/examples
The brokers got brokers.md together with references to CLI utilization information and docs, in addition to the HoloHub codebase. No HoloHub growth expertise had been offered.
The agentic processing time was 65 minutes, with a complete value of 20M tokens. Whereas the workflow resulted in a whole software that achieved the implementation aim, the method was much less environment friendly. The brokers appropriately positioned comparable apps throughout the HoloHub repository, however tended to make use of generic Bash instruments that required extra trial-and-error probing of the event setting. For example, they typically utilized generic linting instruments earlier than CLI-based linting, or tried to put in Python dependencies and run inference scripts straight on the host (ought to have constantly labored in a container).
docs/examples
The brokers got the HoloHub codebase however with out the brokers.md and express steerage on the CLI utilization. No HoloHub growth expertise had been offered.
The agentic processing time was 40min, with a complete value of 15M tokens.
No CLI steerage was offered, the coding brokers nonetheless gained understanding of the CLI (from the general codebase examples) and used it as the primary dev tooling. Nonetheless the coding high quality is suboptimal in comparison with the opposite two settings:
The third get together mannequin config and code had been embedded within the software code incorrectly
A Dockerfile was created with out leveraging the present HoloHub base picture already has all of the required dependencies
Current optimized Holoscan operators corresponding to TensorRT inference and format converter had been ignored within the implementation, in consequence this model is 2.6x slower then the opposite two settings
Whereas follow-up prompts might probably tackle these points, the mix of CLI, expertise, and documentation/examples offered one of the best developer expertise with the bottom useful resource overhead.
A co-development loop for engineers and brokers
This submit follows a small, verifiable growth loop. The result’s an engineering prototype constructed on an present mannequin, pattern video, and Holoscan parts. It runs finish to finish, gives a number of software modes and automatic assessments, preserves the mannequin weights, and data reproducible benchmark proof.
The primary takeaway is the event loop shared by the developer and the brokers: ./holohub provides constant operations, the ability encodes project-specific sequences and checks, and examples and documentation present engineering context. The agent and the developer use the identical CLI instructions. The developer can deal with defining the targets, setting the constraints, evaluating trade-offs, and deciding whether or not the proof is ample.
