Al to Improve the Diagnosis of Rare Rheumatic Diseases

Advanced artificial intelligence (AI) technologies, particularly large language models such as OpenAI's ChatGPT, hold significant potential for enhancing medical decision-making. While ChatGPT was not specifically designed for medical applications, it has shown utility in various healthcare scenarios, including answering patient inquiries, drafting medical documentation, and aiding consultations. Despite these advancements, its role in supporting diagnostic reasoning-especially among less experienced medical students-and for complex rare diseases remains underexplored.

Diagnostic reasoning is a multifaceted process that combines pattern recognition, knowledge synthesis, and probabilistic thinking. Tools like ChatGPT could potentially alleviate cognitive burden, enhance diagnostic accuracy, and ultimately accelerate the diagnosis for rare diseases. However, ChatGPT is not tailored for diagnostic reasoning and lacks comprehensive validation in this domain. Additionally, it is susceptible to generating misinformation or plausible-sounding but inaccurate responses, which may hinder rather than support clinical decision-making. Therefore, understanding how medical students utilize such AI tools is essential before they are integrated into educational or clinical workflows. This study will also assess a standardized prompt to facilitate ChatGPT usage and will give students direct access to enable a realistic scenario.

This study will investigate the impact of ChatGPT on the diagnostic accuracy of medical students when tackling cases of rare rheumatic diseases. Participants will be randomized into two groups: one with access to ChatGPT and one using conventional diagnostic tools. Each participant will analyze diagnostic cases by providing up to 5 differential diagnoses and and rating the diagnostic confidence. Independent reviewers, blinded to group allocation, will evaluate the accuracy and quality of their responses. This study hence aims to provide insights into the potential benefits and limitations of integrating AI tools like ChatGPT.

• ChatGPT (Other): OpenAI's ChatGPT language model with chat interface.

Primary outcomes

• Diagnostic accuracy of top diagnosis [during evaluation]: Participants in each group will make at least one disease suggestion (top diagnosis) and up to a total of a maximum of 5 suggestions. Percentage of exact matches of the top suggestion with the actual diagnosis will be analyzed

Secondary outcomes

• Diagnostic accuracy of top 5 suggestions [during evaluation]: Participants in each group will make at least one disease suggestion (top diagnosis) and up to a total of a maximum of 5 suggestions. Percentage of exact matches with the actual diagnosis included in the top 5 suggestions will be analyzed
• Diagnostic reasoning [during evaluation]: For each case, participants will receive 1 point for each plausible diagnosis and 2 points for a completely correct response. The total scores will be compared between the randomized groups.
• Diagnostic confidence [during evaluation]: For each case participants will be asked for their diagnostic confidence (VAS 0-10). The mean score will be compared between groups.
• Time spent for diagnosis [during evaluation]: We will compare how much time (in seconds) participants spend per case between the two study arms.

Sex: All, ages: 18 Years to any, accepts healthy volunteers

Inclusion Criteria:

* Medical students having started with clinical subjects (Internal medicine)

Exclusion Criteria:

* Not being a medical student

• Results publication: Roemer A, Schlicker N, Kernder A, Albe B, Hack J, Hirsch M, Mayr A, Kuhn S, Knitza J. Large language models enhance diagnostic reasoning of medical students in rheumatology: a randomized controlled trial. BMC Med Educ. 2026 Mar 25;26(1):579. doi: 10.1186/s12909-026-09079-w. PMID 41877128