A Novel Pipeline for Data Management and Analysis that Integrates Data Lakehouse Architecture into the Aeronautics Industry

A Novel Pipeline for Data Management and Analysis that Integrates Data Lakehouse Architecture into the Aeronautics Industry

연구 분야: Databases

학회: International Conference on Innovative Intelligent Industrial Production and Logistics

Currently, the majority of processes or systems aim to benefit from the data produced by their own or other relevant systems, with the objective of increasing efficiency. This is especially true in the field of industrial systems, where a multitude of devices attempt to publish their metrics and data into the system, often resulting in characteristics that can be classified as big data. However, companies often struggle with the correct and useful utilization of this harvested data. Therefore, this paper focuses on a use case of a data pipeline system with a data lakehouse in an airplane parts factory. The developed architecture shows that with some adjustments to the classic data lakehouse architecture, it is possible to achieve higher parallelism in order to simultaneously store data in the data lake and data warehouse. Additionally, a visualization tool was developed to highlight how metric calculation and outlier detection can be automated or facilitated with the utilization of data, as opposed to manual labor.