Dual prior guided depth image super-resolution with multi-scale transformer fusion network

Dual prior guided depth image super-resolution with multi-scale transformer fusion network

연구 분야: Cryptography

학회: The Visual Computer

Guided depth super-resolution (GDSR) is a challenging task that aims to restore a high-resolution depth map from a low-resolution one, using a high-resolution RGB image of the same scene as guidance. Recently, most existing GDSR methods have been limited by insufficient prior knowledge and inappropriate guidance, leading to several challenges, such as serious boundary discontinuities, incomplete structures, and unsatisfactory colors in the reconstructed high-resolution depth maps. To overcome these limitations, we propose a dual prior guided depth image super-resolution method using a multi-scale transformer fusion network. Firstly, we design a dual prior mechanism by employing a color branch to extract color priors from RGB images and an edge branch to extract edge priors from depth images. This approach provides effective prior knowledge for high-resolution depth images, resulting in complete structures and satisfactory colors. Meanwhile, we introduce the self-attention mechanism of the Transformer to the guided depth map super-resolution task to extract global features through a transformer block that utilizes feature mapping from a semi-coupled convolutional block. In addition, we introduce a multi-scale feature fusion module to maintain the boundary continuity of the reconstructed depth image by enhancing the depth image super-resolution guided by RGB images from multiple scales. Extensive quantitative and qualitative experiments on multiple datasets demonstrate that the proposed method has significant advantages over other state-of-the-art GDSR methods. In particular, our method achieves superior super-resolution performance on depth images with noise.