3DGENie: Synthetic point clouds for semantic segmentation in realistic virtual environments

3DGENie: Synthetic point clouds for semantic segmentation in realistic virtual environments

연구 분야: Strategies

학회: Multimedia Tools and Applications

A main R&D pillar in the modern car manufacturing industry revolves around investigating the use of digital assets to train semantic segmentation models in the absence of sufficient real-world data. This study hypothesizes that the procedural generation capabilities of 3DGENie, combined with extensive randomization and realistic sensor simulation, will significantly enhance the effectiveness of synthetic point clouds in training robust semantic segmentation models. To test this, we propose a novel synthetic data generation pipeline called 3DGENie, designed to generate flexible and extensible 3D point clouds tailored for various scenarios and sensor configurations. Leveraging state-of-the-art procedural layout generation, 3DGENie produces region layout trees and applies 3D scene construction and asset randomization to create realistic virtual environments. Synthetic sensors are then used to simulate diverse data capture scenarios, such as colored point clouds and point clouds with reflectance, enabling a comprehensive study of environmental variables’ influence on model performance. Our experiments demonstrate that datasets generated by 3DGENie consistently enhance mean Intersection over Union (mIoU) and mean Accuracy (mAcc) for semantic segmentation models, particularly when real labeled data is limited. In addition to its performance, 3DGENie standouts out compared with existing 3D data generation methods is its flexibility and extensibility to different application scenarios: It is not strictly tied to any specific use case, and can be seamlessly primed to handle different scenarios according to the existing reference data and to the user’s needs. This is emphasized in this study through 3DGENie’s application to two real-world application scenarios using large-scale auto industry data: car assembly lines and autonomous driving. Built on Nvidia Omniverse and Pixar’s Universal Scene Description (USD), the pipeline ensures seamless interoperability across platforms. These results underscore the potential of 3DGENie to address data scarcity challenges, advancing applications from industrial automation to autonomous systems and beyond.