Positive-tune video and picture fashions at scale with NVIDIA NeMo Automodel and 🤗 Diffusers

A joint put up from NVIDIA and Hugging Face. Particular due to Sayak Paul from Hugging Face for his or her contributions to the combination work and for co-authoring this weblog.

Diffusion fashions energy a number of the most fun open-source releases of the final two years — resembling FLUX.1-dev for text-to-image and Wan 2.1 and HunyuanVideo for text-to-video. The 🤗 Diffusers library has develop into the de facto house for these fashions, giving researchers and builders a single, constant interface for inference, adaptation, and pipeline composition.

As well as, coaching and fine-tuning diffusion fashions are additionally on the rise, requiring utilities that supply memory-efficient sharding, latent caching, multiresolution bucketing, and configurations that scale gracefully from one GPU to tons of.

To cater to those technical calls for, we provide the NVIDIA NeMo Automodel open-source library. At the moment, we’re highlighting the collaboration between NVIDIA and Hugging Face that brings production-grade, distributed diffusion coaching to any Diffusers-format mannequin on the Hugging Face Hub — with no checkpoint conversion and no mannequin rewrites for any new mannequin. The mixing is documented within the Diffusers coaching information and is absolutely open supply beneath Apache 2.0.

Desk of contents

What’s NeMo Automodel?

NeMo Automodel is an open-source PyTorch DTensor-native coaching library, a part of the NVIDIA NeMo framework, constructed round two design rules that matter for the Diffusers ecosystem:

Hugging Face native. Level pretrained_model_name_or_path at any Diffusers mannequin ID on the Hub and begin coaching. NeMo Automodel makes use of Diffusers mannequin lessons (e.g. WanTransformer3DModel) for loading and Diffusers pipelines (WanPipeline) for technology. Checkpoints round-trip cleanly again into the Diffusers ecosystem.
One program, any scale. The recipes and coaching scripts will be simply modified to go well with coaching at any scale. Parallelism is a configuration selection, not a code rewrite — swap between FSDP2, tensor parallel, skilled parallel, context parallel, and pipeline parallel by declaring configurations, not rewriting fashions.

AutoModel at the moment helps flow-matching fashions solely. Underneath the hood, it makes use of circulate matching because the coaching goal, with latent-space coaching (through pre-encoded VAE outputs) and multiresolution bucketed dataloading to speed up throughput.

Supported diffusion fashions

NeMo Automodel integration ships with ready-to-use fine-tuning recipes for the open diffusion fashions beneath. The listing displays the recipes at the moment in examples/diffusion/finetune.

What this collaboration unlocks

For Diffusers customers, the sensible positive aspects break down into just a few concrete capabilities.

No checkpoint conversion. Pretrained weights from the Hub work out of the field. There is not any separate “coaching format” to transform to, then convert again. Your fine-tuned checkpoint hundreds straight right into a DiffusionPipeline for inference, or again to the Hub for sharing. Downstream instruments — quantization, compilation, LoRA adapters, customized samplers — all maintain working.

Quick path to new mannequin assist. When a brand new diffusion mannequin lands in Diffusers, enabling it in NeMo Automodel takes a small, contained code addition — a knowledge preprocessing handler and a mannequin adapter — reasonably than a full customized coaching script. The remainder of the recipe stack (FSDP2, bucketed dataloading, checkpointing, technology) carries over unchanged, and the identical YAML-driven workflow applies.

Full and parameter-efficient fine-tuning. Each full fine-tuning and LoRA-style PEFT are supported, so you may select between most high quality (full FT on a big cluster) or most effectivity (LoRA on a single node). The identical recipe construction handles each.

Scalable coaching that goes past what built-in scripts supply. NeMo Automodel provides sharding schemes resembling FSDP2, tensor, context, and pipeline parallelisms, multi-node orchestration (SLURM at the moment, Kubernetes coming), and multiresolution bucketing. These capabilities make coaching bigger fashions like FLUX.1-dev (12B) and HunyuanVideo (13B) potential.

A take a look at the fine-tuning workflow

On this part, we stroll by the standard workflow for fine-tuning any of the supported fashions. The advisable approach to set up Automodel is the NeMo Automodel Docker container (nvcr.io/nvidia/nemo-automodel:26.06), which ships with PyTorch, TransformerEngine, and different CUDA-compiled dependencies pre-built. Alternatively, set up with pip3 set up nemo-automodel or from supply (pip3 set up git+https://github.com/NVIDIA-NeMo/Automodel.git); see the set up information for all choices.

This information walks by a full-transformer fine-tune of FLUX.1-dev on the 78-card Rider–Waite tarot dataset, then producing from the ensuing checkpoint. It reuses the checked-in YAML configs and applies run-specific settings as command-line overrides, so no new config recordsdata are required.

1. Pre-encode the dataset

The diffusion recipe consumes cached VAE latents and textual content embeddings as an alternative of encoding supply photos throughout each coaching step. Stream the 78 Rider–Waite photos straight from Hugging Face and distribute preprocessing throughout all seen GPUs:

uv run –locked –no-default-groups
–extra diffusion
–extra diffusion-media
python -m instruments.diffusion.preprocessing_multiprocess picture
–dataset_name multimodalart/1920-raider-waite-tarot-public-domain
–dataset_media_column picture
–dataset_caption_column caption
–dataset_streaming
–max_images 78
–output_dir /cache/flux_tarot
–processor flux
–model_name black-forest-labs/FLUX.1-dev
–max_pixels 245760

The captions already comprise the trtcrd set off token. With this pixel funds and the dataset’s portrait side ratio, preprocessing assigns the samples to the 384×640 bucket utilized by the showcase run.

For picture coaching, preprocessing produces .pt cache recordsdata and sharded metadata:

/cache/flux_tarot/
├── 384×640/
│ ├── .pt
│ └── …
├── metadata_shard_0000.json
├── metadata.json
└── _hf_dataset/
└── photos/

2. Launch coaching with the present FLUX YAML

Use examples/diffusion/finetune/flux_t2i_flow.yaml straight. The YAML already selects FLUX.1-dev, full transformer fine-tuning, the FLUX flow-matching adapter, an efficient batch measurement of 32, and eight-way FSDP2.

Provide the tarot-specific paths and settings as command-line overrides:

uv run –locked –no-default-groups –extra diffusion
torchrun –nproc-per-node=8
examples/diffusion/finetune/finetune.py
-c examples/diffusion/finetune/flux_t2i_flow.yaml
–model.transformer_engine_fp8 false
–data.dataloader.cache_dir /cache/flux_tarot
–data.dataloader.base_resolution ‘[384,640]’
–lr_scheduler.lr_decay_style fixed
–lr_scheduler.lr_warmup_steps 20
–step_scheduler.max_steps 200
–step_scheduler.ckpt_every_steps 50
–checkpoint.checkpoint_dir /tmp/flux_tarot/checkpoints/full
–checkpoint.save_consolidated true
–seed 2026

The run produces checkpoints at steps 50, 100, 150, and 200. The ultimate checkpoint is labeled epoch_66_step_199; the label is zero-based although it represents the finished 2 hundredth optimizer step.

3. Generate from the fine-tuned checkpoint

Use the present FLUX technology YAML and level mannequin.checkpoint on the full coaching checkpoint:

uv run –locked –no-default-groups –extra diffusion
python examples/diffusion/generate/generate.py
-c examples/diffusion/generate/configs/generate_flux.yaml
–model.checkpoint /tmp/flux_tarot/checkpoints/full/epoch_66_step_199
–inference.top 640
–inference.width 384
–inference.prompts ‘[“a trtcrd of an astronaut tending a rose garden on Mars, “the gardener””]’
–output.output_dir /tmp/flux_tarot/generations/full/step_200
–seed 2026

Embody trtcrd to invoke the realized tarot type. For a management comparability, maintain the seed and scene fastened however omit the set off:

uv run –locked –no-default-groups –extra diffusion
python examples/diffusion/generate/generate.py
-c examples/diffusion/generate/configs/generate_flux.yaml
–model.checkpoint /tmp/flux_tarot/checkpoints/full/epoch_66_step_199
–inference.top 640
–inference.width 384
–inference.prompts ‘[“an astronaut tending a rose garden on Mars, “the gardener””]’
–output.output_dir /tmp/flux_tarot/generations/management
–seed 2026

Outcomes

At step 200, the triggered astronaut prompts retain their requested content material whereas buying a cream, pink, and black classic palette, heavy ink contours, flat coloration fields, aged-paper tones, and allegorical card composition. The untriggered astronaut stays photographic, demonstrating that the realized impact is considerably related to trtcrd reasonably than changing the bottom mannequin globally.

4. Efficiency

All measurements have been collected on one node with 8× NVIDIA H100 80GB GPUs. Outcomes are means ± pattern normal deviation over three steady-state 10-step home windows.

Textual content-to-image — 512×512(see if seconds will be aligned)

Textual content-to-video — 512×512×49 frames

Every pattern is one 49-frame video clip.

Measurement particulars

Different Finetuned/LoRA examples

The outcomes from fine-tuning and LoRA showcase the facility of NeMo Automodel for area specialization. As an example, fine-tuning the Wan 2.1 mannequin on a Ghibli video dataset efficiently tailored the output type, demonstrated by a noticeable change in a flower’s look in comparison with the baseline.

Baseline:

Finetuned on Ghibli’s movies:

We additionally noticed the distinct influence of utilizing LoRA, the place making use of the adapter to Wan 2.1 brought about the video to undertake a attribute Ghibli type, notably seen within the highlighting of characters’ eyes.

No LoRA:

LoRA:

These examples, together with these for FLUX.2, affirm that customers can obtain each most high quality through full fine-tuning and most effectivity through LoRA-style PEFT, tailoring the output to particular stylistic domains.

Strive it at the moment

Study extra in regards to the integration and discover extra fine-tuning examples within the NeMo Automodel documentation

Coming subsequent: Pythonic recipe APIs

YAML is a powerful match for reproducible configuration, particularly for groups that need recordsdata they’ll examine in, overview, and reuse, however many groups additionally want a programmatic interface.

In an upcoming NeMo Automodel launch, we plan to floor the diffusion recipes by a totally typed Pythonic API as effectively. Customers will be capable to compose the identical mannequin, information, optimizer, PEFT/LoRA, parallelism, checkpointing, and technology items straight from Python.

The Pythonic path is meant to make the recipes simpler to make use of from current coaching code, notebooks, and experiment workflows, and to supply a first-class Pythonic interface alongside the YAML quick-start path.

Sources