Title:A Systematic Impact Study for Fuzzer-Found Compiler Bugs

Abstract:Despite much recent interest in compiler fuzzing, the practical impact of fuzzer-found miscompilations on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible impact of fuzzer-found compiler bugs. We follow a novel methodology where the impact of a miscompilation bug is evaluated based on (1) whether the bug appears to trigger during compilation; (2) the extent to which generated assembly code changes syntactically due to triggering of the bug; and (3) how likely such changes are to cause runtime divergences during execution. The study is conducted with respect to the compilation of more than 10 million lines of C/C++ code from 309 Debian packages, using 12% of the historical and now fixed miscompilation bugs found by four state-of-the-art fuzzers in the Clang/LLVM compiler, as well as 18 other bugs found by the Alive formal verification tool or human users. The results show that almost half of the fuzzer-found bugs propagate to the generated binaries for some applications, but barely affect their syntax and only cause two failures in total when running their regression test suites. Our manual analysis of a selection of bugs suggests that these bugs cannot trigger on the packages considered in the analysis, and that in general they affect only corner cases which have a low probability of occurring in practice. User-reported and Alive bugs do not exhibit a higher impact, with less frequently triggered bugs and one test failure.