Addressing cost and resource variability for big data task scheduling in heterogeneous cloud environments

Addressing cost and resource variability for big data task scheduling in heterogeneous cloud environments

연구 분야: Databases

학회: The Journal of Supercomputing

Big data consists of large and complex datasets that exist on a very large scale and complexity which challenge methods of data management, processing, and analysis. Given the ever-increasing demands from users and companies, and considering the massive nature of big data, there is a critical need for powerful scheduling algorithms. Previous studies typically treat tasks and cloud components equivalently without taking costs into account. In contrast, the current paper formulates the scheduling issue as an integer linear programming challenge for scheduling big data tasks in heterogeneous cloud environments, optimizing costs and resource use via server diversity, dynamic pricing, and a novel scheme change rescheduling mechanism. This method makes the created model more relevant to real-life situations, and linear models like simplex and searching state-space trees are used to find a solution. The evaluation findings, when comparing the suggested method to earlier approaches, demonstrate a decrease in costs and an improvement in the uptake of the proposed methodology.