Diffusion Massive Language Fashions (dLLMs) have emerged as a promising various to purely autoregressive language fashions as a result of they’ll decode a number of tokens in parallel. Nevertheless, state-of-the-art block-wise dLLMs depend on a “remasking” mechanism that decodes solely probably the most assured tokens and discards the remaining, successfully losing computation. We show that recycling computation from the discarded tokens is helpful, as these tokens retain contextual info helpful for subsequent decoding iterations. In mild of this, we suggest Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them again for the following denoising step. RCD makes use of a decoupled two-stage coaching pipeline to bypass the reminiscence bottlenecks related to backpropagation. We validate our technique on each lengthy CoT reasoning (SDAR) and brief CoT instruction following (LLaDA) fashions. We show that a typical dLLM might be effectively transformed to the RCD paradigm with merely ∼1 billion tokens. RCD constantly improves frontier dLLMs by 5–10 factors in accuracy with minimal additional computation overhead throughout a variety of benchmarks. Notably, on probably the most difficult AIME duties, RCD almost doubles baseline accuracy and attains as much as 4–5x fewer denoising steps at equal accuracy ranges.
† College of California, Berkeley* Equal contribution‡ Equal advising
