A Comprehensive User Study on Medical Device Threat Modeling Methodologies and CyberLlama2 - MEDICALHARM Threat Modeling Assistant.

A Comprehensive User Study on Medical Device Threat Modeling Methodologies and CyberLlama2 - MEDICALHARM Threat Modeling Assistant.

연구 분야: Safety

학회: International Journal of Information Security

Threat modeling is essential for identifying cyber threats to Modern Medical Devices (MMDs). However, no existing study investigates how MMD manufacturers utilize and perceive existing threat modeling methodologies. Although more industries are adopting the advancement of Large Language Models (LLM) to understand and solve domain-specific challenges, so far, little research has evaluated the effectiveness of LLM in solving threat modeling challenges. To address these gaps, we surveyed 81 participants from MMD manufacturing companies to uncover potential challenges when repurposing threat modeling methodologies, originally conceptualized for IT systems, for MMDs. We also developed a threat modeling-assisted LLM called CyberLlama2 to help identify cybersecurity, privacy, and safety threats in MMDs. The survey analysis revealed that STRIDE is the most widely used methodology. Participants’ satisfaction with their threat modeling methodologies depended on factors like identifying in-depth and third-party component threats, the difficulty of the terminology to non-security roles, and high false-positive rates. Our study also revealed that resource intensiveness, threat prioritization, and inability to identify relevant threats were not statistically significant factors when participants evaluated their satisfaction with their methodology. Our evaluation of CyberLlama2 to help in threat modeling showed a significant improvement in performance over the baseline and other cybersecurity models.