Kitten: A Simple Yet Effective Baseline for Evaluating LLM-Based Compiler Testing Techniques

Kitten: A Simple Yet Effective Baseline for Evaluating LLM-Based Compiler Testing Techniques

연구 분야: Verification

학회: ISSTA Companion '25: Proceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis

Compiler testing is critical and indispensable to improve the correctness of compilers. Spurred by recent advancements in Large Language Models (LLMs), LLM-based compiler testing techniques such as Fuzz4All, have demonstrated their potential in uncovering real bugs in diverse compilers and reducing the required engineering efforts in designing program generators. Given the continuous evolution of LLMs and the emergence of new LLM-based approaches, establishing robust baselines is crucial for rigorous evaluation and driving future advancements in this promising research direction. To this end, we introduce Kitten, a mutation-based, language-agnostic program generator. Kitten leverages a corpus of seed programs, analogous to the training set for LLMs, and utilizes the target language's syntax, akin to the knowledge learned by LLMs. Furthermore, Kitten's mutation operators can generate diverse test programs, demonstrating a behavior analogous to the ability of LLM inference to generate new code. Our evaluations demonstrate that, using existing compiler test suites as seed programs, Kitten outperforms Fuzz4All in terms of code coverage and bug detection capabilities. Within 24 hours, Kitten achieved 48.3%, 9.9%, and 33.8% higher coverage than Fuzz4All on GCC, LLVM, and Rustc, respectively, while identifying an average of 19.3, 20.3, and 15.7 bugs in these compilers across three runs. Over the course of nine months dedicated to Kitten's development and testing, we identified a total of 328 across the compilers GCC, LLVM, Rustc, Solc, JerryScript, scalac, and slang, of which 310 have been confirmed or fixed. We strongly believe that Kitten serves as an effective baseline, enabling the identification of limitations within existing LLM-based approaches and consequently driving advancements in this promising research direction.