Medical News Observer

 

• GPT-4 scored higher than residents, particularly in ‘obstetrics and gynecology,’ and ‘internal medicine.’
• Demonstrated superior ability in answering difficult questions, where resident correct response rates were lower.
• Despite its proficiency in medical knowledge, GPT-4 showed limitations in areas requiring patient interaction and professionalism.

Researchers investigate the capabilities of GPT-4 in the context of clinical reasoning and medical knowledge, particularly in the Japanese language. The researcher’s objective was to assess and compare the clinical competencies of GPT-4 with those of Japanese medical residents using the General Medicine In-Training Examination (GM-ITE). 

The methodology uses GPT-4 to answer single-choice exam questions from the 2020, 2021, and 2022 GM-ITE, focusing on general theory, symptomatology, clinical reasoning, physical examinations, and specific diseases. Questions were further categorized into different specialties and difficulty levels based on the residents’ correct response rates.

The results reveal that GPT-4 significantly outperformed the medical residents in the GM-ITE, scoring higher in most categories, especially in ‘specific diseases,’ ‘obstetrics and gynecology,’ and ‘internal medicine.’ GPT-4 showed a higher aptitude for difficult questions, with lower correct response rates among residents. 

The conclusion highlights GPT-4’s strengths in comprehensive disease understanding and its limitations in areas requiring patient interaction and professionalism. This study underscores the potential and limits of AI in medical education and practice, emphasizing its role in complex problem-solving and its inadequacy in interpersonal aspects of medicine.

Reference:

Watari, Takashi, Soshi Takagi, Kota Sakaguchi, Yuji Nishizaki, Taro Shimizu, Yu Yamamoto, and Yasuharu Tokuda. 2023. “Performance Comparison of ChatGPT-4 and Japanese Medical Residents in the General Medicine In-Training Examination: Comparison Study.” JMIR Medical Education 9 (December): e52202. https://doi.org/10.2196/52202.