RL-BIN++: Overcoming Binary Instrumentation Challenges in the Presence of Obfuscation Techniques and Problematic Features

RL-BIN++: Overcoming Binary Instrumentation Challenges in the Presence of Obfuscation Techniques and Problematic Features

연구 분야: Analysis

학회: ICSCA '21: Proceedings of the 2021 10th International Conference on Software and Computer Applications

This paper improves upon an earlier binary rewriter we designed called RL-Bin. Unlike static rewrites, which are inherently non-robust, RL-Bin uses a dynamic design and thus is more robust. However, although RL-Bin works for most compiled binaries, real-world features commonly found in obfuscated binaries are still not handled. The features include anti-disassembly, dynamically modified code, anti-rewriting, anti-debugging, and code convention violation. This paper presents RL-Bin++, an improved version of RL-Bin, that handles various problematic real-world features, thus correctly rewriting for nearly all benign binaries. We demonstrate that RL-Bin++ can efficiently instrument heavily obfuscated binaries (overhead averaging 2.76x, compared to 4.11x, and 5.31x overhead for DynamoRIO and Pin, which are comparable or lower overheads. However, the main achievement is that we achieved this while maintaining the low overhead of RL-Bin for unobfuscated binaries (only 1.05x). This makes RL-Bin++ the only robust binary instrumentation solution capable of being deployed in live systems since the overhead of DynamoRIO (1.16x), and Pin (1.29x) for unobfuscated binaries is too high for use in live systems.