Encrypted Malicious Traffic Detection Using Multi-instance Learning

Encrypted Malicious Traffic Detection Using Multi-instance Learning

연구 분야: Safety

학회: International Conference on Computational Science

The detection of malicious traffic remains a critical challenge in cybersecurity, particularly with the widespread adoption of encryption protocols, which obscure malicious activities within legitimate network traffic. Traditional detection methods typically rely on single-flow analysis and fail to capture the multi-flow interactions present in malicious traffic, resulting in poor detection performance in scenarios with mixed benign and malicious flows. In this paper, we propose a novel approach that leverages multi-instance learning (MIL) to address the challenge of mixed traffic by aggregating flows into bags and employing attention mechanisms to prioritize critical instances. Our framework processes encrypted traffic by first segmenting bursts to capture traffic patterns, followed by CNN-based feature extraction to identify relevant characteristics. The attention pooling mechanism then prioritizes significant instances, effectively filtering out irrelevant flows and emphasizing multi-flow interactions that are indicative of attacks. Experimental results on real-world datasets demonstrate significant improvements in both robustness and precision, highlighting the framework’s effectiveness in detecting encrypted malicious traffic in complex network environments.