SQL Autograder: Web-based LLM-powered Autograder for Assessment of SQL Queries

SQL Autograder: Web-based LLM-powered Autograder for Assessment of SQL Queries

연구 분야: Databases

학회: International Journal of Artificial Intelligence in Education

Structured query language (SQL) queries are an important aspect of database concepts in the information technology (IT) domain. Evaluation of SQL queries ensures that the learners can understand and apply various SQL concepts correctly. However, this can be a laborious task when carried out manually by course instructors at universities, which often does not scale well. To address these limitations, this study proposes a web-based application, SQL autograder, which can be used by instructors of a university course to evaluate assessments and enhance the quality of education and learning outcomes. We propose a framework that makes use of large language models (LLMs) to assess the correctness of SQL queries submitted by students. We train a variety of open-source LLMs of varying sizes on a diverse dataset of SQL queries, with queries ranging from simple ones that include a single JOIN statement to more complex ones involving multiple SQL features. We implemented and tested our LLM-based framework in real-world educational settings for a university course, which shows promising results in enhancing the learning experience for students by providing instant feedback on areas needing improvement. We tested our application on 88 participants and found that the autograder is 180x faster than the instructor, with an average accuracy of 96.77%. After taking the qualitative feedback from the participants, 97% of them found it to be useful. The proposed framework reduces the workload of instructors by offering a more scalable and consistent evaluation process that enhances the performance of students.