CPSDbench: a large language model evaluation benchmark and baseline for Chinese public security domain

CPSDbench: a large language model evaluation benchmark and baseline for Chinese public security domain

연구 분야: Verification

학회: International Journal of Data Science and Analytics

Large language models (LLMs) have demonstrated significant potential and effectiveness across multiple domains. To evaluate the performance of mainstream LLMs in public security tasks, this study aims to construct a specialized evaluation benchmark tailored to the Chinese public security domain—CPSDbench. CPSDbench integrates datasets related to public security collected from real-world scenarios, supporting a comprehensive evaluation of LLMs across four key dimensions: text classification, information extraction, question answering, and text generation. Furthermore, this study introduces a set of innovative evaluation metrics designed to more precisely quantify the efficacy of LLMs in executing tasks related to public security. Through the in-depth analysis and evaluation conducted in this research, we not only enhance our understanding of the performance strengths and limitations of existing models in addressing public security issues but also provide references for the future development of more accurate and customized LLM models targeted at applications in this field.