Large Language Models to Aid Gynecological Oncology Treatment

Advanced artificial intelligence (AI) technologies, particularly large language models such as OpenAI's ChatGPT, hold significant potential for enhancing medical decision-making. While ChatGPT was not specifically designed for medical applications, it has shown utility in various healthcare scenarios, including answering patient inquiries, drafting medical documentation, and aiding clinical consultations. Despite these advancements, its role in supporting treatment decision-making-particularly in complex oncological cases-remains underexplored.

Treatment decision-making in gynecological oncology is a multifaceted process that integrates evidence-based guidelines, tumor biology, patient-specific factors, and clinical expertise. AI tools like ChatGPT could potentially assist in synthesizing relevant guideline-based recommendations, improving decision accuracy, and facilitating more efficient clinical workflows. However, ChatGPT is not specifically tailored for oncological treatment decisions and lacks comprehensive validation in this domain. Additionally, it may generate misinformation or plausible-sounding but inaccurate recommendations, which could impact clinical judgment. Therefore, understanding how medical professionals, including students and early-career physicians, interact with such AI tools is essential before broader integration into clinical practice. Locally deployable models, such as Llama, enable secure, on-premise usage while retrieval-augmented generation ensures guideline-compliant recommendations.

This study will investigate the impact of language models on treatment decision support for medical students managing gynecological oncology cases. This is a crossover study, where participants will be randomized into two groups. All participants begin with access to ChatGPT for two vignettes. They then proceed with two cases using either a locally deployed language model, followed by two cases relying on guideline PDFs, or vice versa.

Each participant will analyze clinical cases, propose treatment plans, and rate their confidence in their decisions and decision support system usability. This study aims to provide insights into the potential benefits and limitations of integrating AI tools like ChatGPT into oncological treatment decision-making.

• Local language model (Other): Group will be given access to local language model first after using ChatGPT and then will get access to pdf file
• Guideline pdf (Other): Group will be given access to pdf file after ChatGPT and then to a local language model

Primary outcomes

• Treatment concordance with tumor board decisions [directly (within 10 minutes) after Intervention]: Participants in each group select treatment modalities for case vignettes

Secondary outcomes

• Treatment confidence [directly (within 10 minutes) after Intervention]: For each case participants will be asked for their treatment confidence (VAS 0-10). The mean score will be compared between decision support groups.
• Time spent for treatment decision [directly (within 10 minutes) after Intervention]: Time (in seconds) participants spend per case between the decision support groups will be compared.

Sex: All, ages: 18 Years to any, accepts healthy volunteers

Inclusion Criteria:

\- Medical students having started with clinical subjects

Exclusion Criteria:

\- Not being a medical student

• Sebastian Griewing, MD PhD, Principal investigator, Philipps University Marburg