Reproducibility Challenges of External Computational Experiments in Scientific Workflow Management Systems

Reproducibility Challenges of External Computational Experiments in Scientific Workflow Management Systems

연구 분야: Software Development

학회: International Conference on Human-Computer Interaction

Reproducibility is essential in scientific experiments to ensure that results can be consistently obtained using new data and methods across studies to answer the same scientific question. It evaluates the validity and robustness of the results. A scientific workflow management system (SWfMS) offers specialized tools for data analysis, modelling, and visualization, streamlining complex analyses and accelerating discoveries. It facilitates reproducibility by providing a standardized environment for executing scientific experiments. However, external scientific experiments often utilize tools developed independently by researchers, employing diverse programming languages and operating systems. Reproducing these experiments within an SWfMS requires installation, configuration, and incorporation of the tools into the SWfMS environment for composition, execution, and analysis of workflows. Managing a multi-user SWfMS is challenging, especially when integrating and configuring a diverse array of scientific tools. Many tools require elevated privileges either during installation or execution, creating roadblocks for efficient onboarding and potentially compromising security. Dissimilarities between the SWfMS and individual tool environments can render straightforward installation impossible, requiring time-consuming workarounds. These challenges burden administrators and impede research progress by hindering the seamless reproduction of critical scientific experiments. Our research addressed the critical challenge of reproducing external scientific experiments within SWfMSs. The complex process of integrating diverse software tools and executing them within the SWfMS runtime infrastructure often hinders reproducibility. We conducted a user study to investigate researchers’ challenges during experiment reproduction in SWfMSs. Based on the identified pain points, we propose a solution that empowers domain experts to package their tools within self-contained, isolated Docker containers. These containers can then be seamlessly uploaded to the system via a user-friendly graphical interface, eliminating installation obstacles and facilitating integration with the SWfMS runtime environment. Furthermore, we conducted another user study to evaluate the effectiveness of our proposed method, ensuring its practical utility for researchers in real-world scenarios.