[2505.03818] Program Semantic Inequivalence Recreation with Giant Language Fashions

View a PDF of the paper titled Program Semantic Inequivalence Recreation with Giant Language Fashions, by Antonio Valerio Miceli-Barone and a pair of different authors

View PDF
HTML (experimental)

Summary:Giant Language Fashions (LLMs) can obtain sturdy efficiency on on a regular basis coding duties, however they’ll fail on complicated duties that require non-trivial reasoning about program semantics. Discovering coaching examples to show LLMs to resolve these duties may be difficult.
On this work, we discover a technique to synthetically generate code reasoning coaching information primarily based on a semantic inequivalence sport (SInQ): a generator agent creates program variants which are semantically distinct, derived from a dataset of real-world programming duties, whereas an evaluator agent has to determine enter examples for which they behave in another way. The brokers prepare one another semi-adversarially, bettering their potential to grasp the underlying logic of code.
We evaluated our strategy on a number of code era and understanding benchmarks, together with cross-language vulnerability detection (Lu et al., 2021),, the place our methodology improves vulnerability detection in C/C++ code regardless of being educated completely on Python code, and the difficult Python builtin identifier swap benchmark (Miceli Barone et al., 2023),, displaying that whereas trendy LLMs nonetheless wrestle with this benchmark, our strategy yields substantial enhancements.
We launch the code wanted to copy the experiments, in addition to the generated artificial information, which can be utilized to fine-tune LLMs.