[2505.17315] Longer Context, Deeper Pondering: Uncovering the Function of Lengthy-Context Skill in Reasoning

View a PDF of the paper titled Longer Context, Deeper Pondering: Uncovering the Function of Lengthy-Context Skill in Reasoning, by Wang Yang and 5 different authors

View PDF
HTML (experimental)

Summary:Current language fashions exhibit sturdy reasoning capabilities, but the affect of long-context capability on reasoning stays underexplored. On this work, we hypothesize that present limitations in reasoning stem, partly, from inadequate long-context capability, motivated by empirical observations similar to (1) increased context window size usually results in stronger reasoning efficiency, and (2) failed reasoning circumstances resemble failed long-context circumstances. To check this speculation, we look at whether or not enhancing a mannequin’s long-context skill earlier than Supervised Tremendous-Tuning (SFT) results in improved reasoning efficiency. Particularly, we in contrast fashions with equivalent architectures and fine-tuning information however various ranges of long-context capability. Our outcomes reveal a constant development: fashions with stronger long-context capability obtain considerably increased accuracy on reasoning benchmarks after SFT. Notably, these features persist even on duties with brief enter lengths, indicating that long-context coaching provides generalizable advantages for reasoning efficiency. These findings recommend that long-context modeling isn’t just important for processing prolonged inputs, but in addition serves as a vital basis for reasoning. We advocate for treating long-context capability as a first-class goal within the design of future language fashions.