On the need to monitor continuous integration practices

On the need to monitor continuous integration practices

연구 분야: Software Development

학회: Empirical Software Engineering

One of the crucial activities in software development is monitoring. It plays a vital role in verifying the proper implementation of processes, the identification of errors, and the discovery of opportunities for improvement. Continuous Integration (CI) encompasses a set of widely adopted practices that enhance software development. However, there are indications that developers may not adequately monitor CI practices. Hence, this paper explores developers’ perceptions regarding monitoring CI practices. To achieve this, we first defined metrics associated with CI practices. Then, we conducted a series of analyses to evaluate how these metrics are perceived in projects or by their developers. As a starting point, we perform a Document Analysis to assess developers’ expressed need for practice monitoring in pull request comments generated by developers during the development process. After that, we conduct a survey of 28 developers from 121 open-source projects to understand the perception of the significance of monitoring seven CI practices in their projects. Finally, we triangulate the emergent themes from our survey by performing a second Document Analysis to understand the extent of monitoring features supported by existing CI services. Our key findings indicate that: 1) the most frequently mentioned CI practice during the development process is “Write automated developer tests”, which is associated with the metric “Coverage” (> 80%), while “Don’t commit broken code” and “Fix Broken Builds Immediately’ present notable opportunities for monitoring CI practices; 2) developers do not adequately monitor all CI practices and express interest in monitoring additional practices; and 3) the most popular CI services currently offer limited native support for monitoring CI practices, requiring the use of third-party tools. Our results lead us to conclude that monitoring CI practices is often overlooked by both CI services and developers. Using third-party tools in conjunction with CI services is challenging, they monitor some redundant practices, and they still fall short of fully supporting CI practices monitoring. Therefore, CI services should implement CI practice monitoring, which would facilitate and encourage developers to monitor them.