Research on the Framework of Network Security Threat Intelligence Crawler

Research on the Framework of Network Security Threat Intelligence Crawler

연구 분야: Safety

학회: SECA '25: Proceedings of the 2025 International Conference on Software Engineering and Computer Applications

In light of the global proliferation of advanced persistent threats (APTs) and ransomware attacks, cyber threat intelligence (CTI) has emerged as a pivotal component of proactive defence strategies. However, existing cyber crawling frameworks face three challenges: high technical costs, performance bottlenecks and scalability issues, and ecological sustainability risks. This paper therefore proposes an artificial intelligence-driven, distributed web crawling framework based on cybersecurity threat intelligence. This framework overcomes the limitations of existing open-source tools with regard to dynamic rendering, high technical costs, low scalability, anti-crawling measures and intelligent analysis. Experiments demonstrate that the proposed framework enhances crawling efficiency by 70% and reduces resource utilisation by 50% compared to the Scrapy approach. This research provides a scalable paradigm for building an autonomous, controllable, cybersecurity-intelligent infrastructure, offering a new direction for cyber-crawling research. Its applications can be extended to key areas such as financial risk control and public opinion monitoring.