M2-Conformer: Multi-modal CNN- Transformer for Driving Behavior Detection

M2-Conformer: Multi-modal CNN- Transformer for Driving Behavior Detection

연구 분야: Software Development

학회: 2022 5th International Symposium on Autonomous Systems (ISAS)

As playing a key role in the development of Advanced Driver Assistance System (ADAS), driving behavior detection has attracted more and more attention over recent years. In this paper, a novel hybrid deep learning framework, Multi-Modal CNN-Transformer (M2-Conformer) is proposed for driving behavior detection from video frames and multivariate vehicle signals. The M2-Conformer is constructed with both Transformer and CNN architecture in paralleled branch to extract driving scene features and vehicle dynamics features respectively. Specially, a dynamic token sparsification is utilized for Transformer branch to prune redundant tokens hierarchically, which makes the framework easy to obtain speed-up. To enhance the performance, a custom-built Feature Aggregation Module (FAM) is designed for M2-Conformer to generate aggregated features from different branches with higher-quality. Experiments carried out on a naturalistic driving data set indicate that the proposed M2-Conformer achieves a superior complexity/accuracy trade-off for driving behavior detection compared to other state-of-the-art architectures.