A data lake-based security transmission and storage scheme for streaming big data

A data lake-based security transmission and storage scheme for streaming big data

연구 분야: Safety

학회: Cluster Computing

Streaming big data presents unique security challenges due to its real-time generation and distributed transmission methods, making it vulnerable to security threats such as data leakage and tampering. To address these challenges, we propose a secure transmission and storage scheme for streaming big data based on a data lake architecture. Our approach leverages an ECC lightweight encryption algorithm in a streaming data encryption interceptor to filter and encrypt key information at the data source. We also introduce the SSL secure transmission protocol to ensure secure data transmission over a multi-source streaming data transmission channel constructed using Flume and Kafka. Furthermore, we design a data partition scheme based on LZO compression and implement a data lake storage system using Hadoop. Our proposed scheme can efficiently and securely transfer streaming data from multiple data sources to the data lake, while providing high data query efficiency. Experimental results show that our stream data encryption interceptor reduces memory load by 18% and filters and encrypts key information at a faster speed compared to similar schemes. In addition, our data lake storage scheme demonstrates lower data write latency and space occupancy, making it well-suited for large-scale streaming data applications.