Finetuning LLMs for Text-to-SQL with Two-Stage Progressive Learning

Finetuning LLMs for Text-to-SQL with Two-Stage Progressive Learning

연구 분야: Databases

학회: CCF International Conference on Natural Language Processing and Chinese Computing

With the widespread usage of large language model (LLMs), LLM-based method has become the mainstream approach for Text-to-SQL tasks, achieving leading performance on Text-to-SQL leaderboards. However, generating complex SQL queries correctly has always been a main challenge. Current LLM-based models primarily utilize prompting-based methods on large scale closed-source LLMs (e.g., GPT-4 and ChatGPT), which may cause concerns of usage costs and data privacy. For fine-tuning based methods, it is difficult to generate complex SQL accurately in only one fine-tuning step. Focusing on this, we propose TSP-SQL, a Two-Stage Progressive learning method for Text-to-SQL. TSP-SQL decomposes Text-to-SQL task into two stages: SQL elements generation auxiliary task, and SQL query generation main task. The two tasks are progressively fine-tuned on a single model, effectively reducing the difficulty of SQL generation and improving accuracy. TSP-SQL achieves state-of-the-art performance among open-source fine-tuning based methods on Spider dev set, and surpasses most of the methods based on large scale closed-source LLMs.