Weight-ETL: A Multi-Stage Cluster Scheduling Method for Medical Data Governance

Weight-ETL: A Multi-Stage Cluster Scheduling Method for Medical Data Governance

연구 분야: Databases

학회: 2024 3rd International Conference on Big Data, Information and Computer Network (BDICN)

ETL cluster scheduling technology plays a crucial role, which unveils the latent value of medical data and improve the medical digitization. However, in real-world scheduling scenarios, variations in the performance configurations of cluster nodes can significantly impact scheduling times. In addition, most mainstream ETL tools often neglect hospital business requirements in the governance process, leading to meaningless time consumption for important tasks.To solve these problems, we propose the Weight-ETL, a multi-stage cluster scheduling algorithm specifically tailored for medical data governance. This algorithm takes into account the task importance and node performance comprehensively throughout the ETL governance process, thus enhancing data governance efficiency and aligning with hospital requirements. Quantitative experiments demonstrate that our proposed algorithm effectively prioritizes the tasks with high demand relevance and improves the efficiency of cluster scheduling compared with previous methods.