CMT-YARN: an efficient security framework for yarn based on an improved merkle tree

CMT-YARN: an efficient security framework for yarn based on an improved merkle tree

연구 분야: Verification

학회: The Journal of Supercomputing

Hadoop is one of the most popular platforms for distributed big data processing, and YARN is at the core of Hadoop 2.0. During data processing, malicious nodes may compromise results by sharing incorrect outputs in the public cloud. To address this issue, this paper introduces an efficient YARN security framework, termed CMT-YARN, which incorporates an enhanced convolutional Merkle tree (CMHT) to ensure the reliability of task execution results. In a hybrid cloud environment, CMT-YARN leverages the improved CMHT technique for dual verification of intermediate and final results, ensuring data integrity and reliability. Compared to the conventional Merkle tree, CMHT reduces computational overhead by more than 37% during verification. Qualitative and quantitative analyzes indicate that with a data sampling rate of 6.8%, CMT-YARN can ensure that the number of malicious acts undetected by the system does not exceed 5; when the sampling rate exceeds 26.9%, the framework can guarantee the detection of all malicious activities. Experimental results on a real Hadoop cluster demonstrate that CMT-YARN significantly enhances computational and storage performance compared to traditional solutions.