Assessment of the quality, readability, and usefulness of ChatGPT generated medical information for ten common cancer types

Assessment of the quality, readability, and usefulness of ChatGPT generated medical information for ten common cancer types

연구 분야: Verification

학회: Universal Access in the Information Society

Chat Generative Pre-trained Transformer (Chat GPT) is an open platform to access different types of information in various areas including medicine. In this study we aimed to assess the quality, readability and usefulness of ChatGPT - generated medical information for the ten common cancer types. Seven questions for ten common types of cancer were asked to Chat GPT, and the medical information provided was evaluated with the Global Quality Scale (GQS). The information provided is classified as “useful” or "misleading”. The readability of the answers generated by Chat GPT was assessed using the Flesch–Kincaid Grade Level (FKGL) test and the Gunning Frequency of Gobbledygook (Gunning Fog) formula. Among all 70 answers the mean scores were 4.03 ± 0.48 for GQS, 15.11 ± 7.36 for FKGL and 18.16 ± 7.40 for Gunning Fog. Across all questions 4.3% of the total content found as misleading. Answers for questions about colorectal cancer exhibited the highest readability score (median FKGL = 21.5, min = 10.0–max = 25.7, median Gunning Fog = 23.7, min = 10.8–max = 29.4). Answers for skin cancer questions had the lowest readability (median FKGL = 11.4, min = 8.1–max = 46.9, median Gunning Fog = 14.2, min = 11.2–max = 49.8). The readability was assessed as “difficult”. The ChatGPT platform can represent a source of information as it provides highly accurate data. The ChatGPT platform can represent a source of information as it provides highly accurate data. However, readability was “difficult”. Although we think that the platform may be useful for medical students and others in the medical field seeking health information, it may not be suitable for the public for the same purpose.