Who dominates generative AI? Analyzing user feedback to identify common use cases and areas for improvement in ChatGPT, Copilot and Gemini

Who dominates generative AI? Analyzing user feedback to identify common use cases and areas for improvement in ChatGPT, Copilot and Gemini

연구 분야: Artificial Intelligence

학회: Knowledge and Information Systems

Individuals are increasingly engaging with Generative AI applications for assistance in personal, educational and professional contexts. Previous research has primarily focused on such human–AI interaction from the perspectives of design principles, ethical considerations and societal impact. However, this emphasis may have overlooked the importance of evaluating these applications in terms of service quality and effectiveness. For instance, several key questions remain unaddressed: what are the contexts in which users most frequently interact with these applications, how effective are these applications in providing satisfying responses across different use cases and what interaction contexts require improvements to better meet user expectations. To address these questions, this research proposes a data analytical approach to analyze textual reviews of leading generative AI applications, including ChatGPT, Copilot and Gemini. The findings reveal that the common interaction contexts, in order of prevalence, are Education, Profession, Writing, Visual Content, Planning and Entertainment. Furthermore, a Machine Learning-inspired approach conducts a comparative analysis of application-specific use cases to assess which interaction contexts effectively satisfy users and which use cases are correlated with lower user satisfaction, highlighting potential areas for improvement. Overall, ChatGPT excels in satisfying users, whereas Copilot remains competitive but delivers services that are marginally less satisfying. Conversely, Gemini significantly lags behind in effectively meeting user exceptions. Validation of the results in the majority of interaction contexts yielded an accuracy exceeding 80%. With a focus on results understandability, this research provides valuable insights to application developers, ultimately facilitating the delivery of more effective AI-assisted services.