Trendy Giant Language Fashions obtain spectacular reasoning capabilities with lengthy Chain of Ideas, however they incur substantial computational value throughout inference, and this motivates strategies to enhance the performance-cost ratio. Amongst these strategies, Speculative Decoding accelerates inference by using a quick however inaccurate draft mannequin to auto-regressively suggest tokens, that are then verified in parallel by a extra succesful goal mannequin. Nonetheless, as a consequence of pointless rejections attributable to token mismatches in semantically equal steps, conventional token-level Speculative Decoding struggles in reasoning duties. Though latest works have shifted to step-level semantic verification, which enhance effectivity by accepting or rejecting complete reasoning steps, current step-level strategies nonetheless regenerate many rejected steps with little enchancment, losing precious goal compute. To deal with this problem, we suggest ARBITRAGE, a novel step-level speculative technology framework that routes technology dynamically based mostly on the relative benefit between draft and goal fashions. As a substitute of making use of a set acceptance threshold, ARBITRAGE makes use of a light-weight router skilled to foretell when the goal mannequin is more likely to produce a meaningfully higher step. This routing approximates a really perfect ARBITRAGE ORACLE that at all times chooses the higher-quality step, reaching near-optimal effectivity–accuracy trade-offs. Throughout a number of mathematical reasoning benchmarks, ARBITRAGE constantly surpasses prior step-level SD baselines, decreasing inference latency by as much as ∼ 2× at matched accuracy.
† UC Berkeley‡ ICSI§ LBNL* Equal contribution
