Clinical Decision Support via ChatGPT: An Evidence-Based Perspective in Pediatric Occupational Therapy

Abstract

Aim: This study aims to evaluate the accuracy of ChatGPT’s responses to clinical questions related to Cerebral Palsy (CP) and Autism Spectrum Disorder (ASD) in the field of occupational therapy, as well as the reliability of the references it provides.Method: Ten clinical questions were formulated, and answers were prepared based on clinical guidelines and reviewed by two expert clinicians. The same questions were presented to ChatGPT, and its responses and references were rated by two independent evaluators using a four-point Likert scale. Weighted Kappa Coefficients were calculated to assess inter-rater agreement.Results: ChatGPT’s responses demonstrated high accuracy, with strong inter-rater agreement (Weighted Kappa: 0.72–0.75). However, significant deficiencies were identified in reference reliability, with moderate to high inter-rater agreement (Weighted Kappa: 0.75–0.78). Additionally, fictitious reference rates ranged from 40% to 100% for CP-related questions and from 25% to 100% for ASD-related questions.Conclusion: While ChatGPT shows potential as a clinical decision support tool in occupational therapy, the limitations in reference accuracy restrict its reliability in healthcare applications. Future studies should focus on improving artificial intelligence models’ reference accuracy to enhance their applicability in healthcare settings.