Medical News Observer

• GPT-3.5 and GPT-4 analyzed 23,035 surgical questions, achieving 53.3% and 64.4% accuracy respectively.
• Performance varied across surgical specialties, with strengths in anatomy and weaknesses in orthopedics and neurosurgery.
• Findings stress the importance of custom training for AI in medical subdomains, promising enhanced future healthcare assistance.

Researchers explore the use of Large Language Models (LLMs) like GPT-3.5 and GPT-4 in medical education, mainly focusing on their effectiveness in answering surgical questions relevant to clinical practice. The study utilized the MedMCQA dataset, which contains many multi-choice clinical questions. Of these, researchers selected 23,035 surgical questions and posed them to GPT-3.5 and GPT-4.

Study Results

The findings revealed that both GPT-3.5 and GPT-4 showed a significant difference in accuracy, with GPT-3.5 achieving 53.3% and GPT-4 achieving 64.4% accuracy in answering surgical questions. Interestingly, the performance of each model varied when compared with their overall performance on the full MedMCQA dataset. GPT-4 was less accurate in the surgical domain than in general medicine, while GPT-3.5 showed the opposite trend. Moreover, the accuracy of these models differed significantly across various surgical specialties. They performed well in anatomy, vascular, and pediatric surgery but were less accurate in orthopedics, ENT (Ear, Nose, and Throat), and neurosurgery.

Implications

This study highlights the potential of Large Language Models (LLMs) like GPT-3.5 and GPT-4 as supportive tools in the medical field. It also emphasizes the need for specialized training of LLMs for specific medical domains, ensuring more reliable and precise information for healthcare professionals. This study demonstrates the rapid evolution of LLM capabilities, suggesting that future models could be even more effective in assisting medical practitioners. It also emphasizes the importance of domain-specific evaluation of AI tools for their safe and effective integration into healthcare practices.

Reference

Ref: Murphy Lonergan, Rebecca, Jake Curry, Kallpana Dhas, and Benno I. Simmons. 2023. “Stratified Evaluation of GPT’s Question Answering in Surgery Reveals Artificial Intelligence (AI) Knowledge Gaps.” Cureus 15 (11): e48788. https://doi.org/10.7759/cureus.48788