Auto-scaling of Scientific Workflows in Kubernetes

Auto-scaling of Scientific Workflows in Kubernetes

연구 분야: Software Development

학회: International Conference on Computational Science

Kubernetes has gained extreme popularity as a cloud-native platform for distributed applications. However, scientific computations which typically consist of a large number of jobs – such as scientific workflows – are not typical workloads for which Kubernetes was designed. In this paper, we investigate the problem of autoscaling, i.e. adjusting the computing infrastructure to the current resource demands. We propose a solution for auto-scaling that takes advantage of the known workflow structure to improve scaling decisions by predicting resource demands for the near future. Such a predictive autoscaling policy is experimentally evaluated and compared to a regular reactive policy where only the current demand is taken into account. The experimental evaluation is done using the HyperFlow workflow management systems running five simultaneous instances of the Montage workflow on a Kubernetes cluster deployed in the Google Cloud Platform. The results indicate that the predictive policy allows achieving better elasticity and execution time, while reducing monetary cost.