Effectiveness of Original vs. AI-Generated Plain Language Summaries of Systematic Reviews

Studies show that Plain Language Summaries (PLS), which are intended to improve accessibility to health information for laypeople, are lacking in terms of comprehension, readability, and user engagement. Further empirical research is needed to evaluate the comparative effectiveness of AI-generated summaries versus original Plain Language Summaries. This study, conducted at the Department of Oncology and Radiotherapy at the University Hospital of Split aims to explore whether large language models (LLMs), such as ChatGPT, can be leveraged to provide summaries of complex medical content, as those found in systematic reviews, to be more streamlined and adapted for laypeople. The prospective, randomized, single-center, parallel group, double-blind, controlled trial focuses specifically on patients diagnosed with non-metastatic breast cancer, histologically confirmed, and aims to help them make more evidence-based medical decisions regarding their health and recovery. The study's primary objective is to analyze patients' level of intention for changing their behavior and start applying the interventions described in the summary. The level of intention to change behavior will be measured using a visual analog scale ranging from 0 to 100 (0 - do not agree at all and 100 - fully agree). The secondary outcomes are patients' level of understanding calculated by the score received on a five-question knowledge survey (5 as a maximum score), the level of trust in (range 0-100) and clarity of (range 0-100) the information presented in the PLS, and their confidence in their decision to change behavior (range 0-100). The survey is based on the information presented in the PLS and the questions focus on patients' understanding of the benefits and risks of the intervention as well as the quality of evidence described in the systematic reviews. Recruitment for this study will take place over 6 months. Patients will be invited to participate in the study before or after their routine oncology appointment. The participants will read two summaries from two respective Cochrane reviews: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer. They will then take the brief survey that tests the impact and likelihood of applying the recommendations described in the summary and their overall understanding of the medical information discussed in the reviews. The study includes female adult oncology patients diagnosed with non-metastatic breast cancer, histologically confirmed, following a modality of the treatment, which includes one or more of the following: neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant radiotherapy (including local and locoregional radiation treatment, with possible radiotherapy boost where indicated). Since participants in this study are limited to reading material and completing a survey, the study is non-invasive in nature and poses low-risk. Data will be analyzed using descriptive statistics to summarize participant characteristics and assess baseline comparability between the two study arms.

• AI-generated PLS (Behavioral): Reading AI-generated Plain Language Summaries of two Cochrane systematic reviews covering the same two evidence-based topics: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer.
• Original PLS (Behavioral): Reading original Plain Language Summaries of two Cochrane systematic reviews covering the same two evidence-based topics: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer.

Primary outcomes

• Patients' level of intention for changing their behavior [Immediately after reading individual PLS]: Patients' level of intention for changing their behavior and start applying the interventions described in the summary, measured on a visual analogue scale (VAS) ranging from 0 to 100 (do not agree at all to 100 fully agree).

Secondary outcomes

• Patients' comprehension of the information presented in the summary [Immediately after reading individual PLS]: Patients' level of understanding of the information presented in the summaries calculated by the score received on a five-question knowledge survey (score range 0-5).
• Patients' level of trust in the information presented [Immediately after reading individual PLS]: Patients' level of trust in the information presented in the summaries, measured on a VAS (range 0-100).
• Patients' level of perceived clarity in the information [Immediately after reading individual PLS]: Patients' level of perceived clarity of the information presented in the summaries, measured on a VAS (range 0-100).
• Patient's level of confidence in their decision to change behavior [Immediately after reading individual PLS]: Patients' level of confidence in the decision to change behavior, measured on a VAS (range 0-100)

Sex: Female, ages: 18 Years to any

Inclusion Criteria:

* female patients with histologically confirmed non-metastatic breast cancer currently undergoing chemotherapy or radiotherapy:

  1. neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
  2. adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
  3. adjuvant radiotherapy (including local and locoregional radiation treatment with possible radiotherapy boost where indicated).
* patients need to be able to read and understand Croatian.

Exclusion Criteria:

* female patients with metastatic breast cancer.
* female patients receiving only hormonal treatment.

• Hasannejadasl H, Roumen C, Smit Y, Dekker A, Fijten R. Health Literacy and eHealth: Challenges and Strategies. JCO Clin Cancer Inform. 2022 Sep;6:e2200005. doi: 10.1200/CCI.22.00005. PMID 36194843
• Seiler A, Jenewein J. Resilience in Cancer Patients. Front Psychiatry. 2019 Apr 5;10:208. doi: 10.3389/fpsyt.2019.00208. eCollection 2019. PMID 31024362
• Goerling U, Faller H, Hornemann B, Honig K, Bergelt C, Maatouk I, Stein B, Teufel M, Erim Y, Geiser F, Niecke A, Senf B, Wickert M, Buttner-Teleaga A, Weis J. Information needs in cancer patients across the disease trajectory. A prospective study. Patient Educ Couns. 2020 Jan;103(1):120-126. doi: 10.1016/j.pec.2019.08.011. Epub 2019 Aug 12. PMID 31474389
• Diefenbach M, Turner G, Carpenter KM, Sheldon LK, Mustian KM, Gerend MA, Rini C, von Wagner C, Gritz ER, McQueen A, Prayor-Patterson HM, Miller SM. Cancer and patient-physician communication. J Health Commun. 2009;14 Suppl 1(Suppl 1):57-65. doi: 10.1080/10810730902814079. PMID 19449269
• Rosa WE, Levoy K, Doyon K, McDarby M, Ferrell BR, Parker PA, Sanders JJ, Epstein AS, Sullivan DR, Rosenberg AR. Integrating evidence-based communication principles into routine cancer care. Support Care Cancer. 2023 Sep 8;31(10):566. doi: 10.1007/s00520-023-08020-x. PMID 37682354
• Buljan I, Tokalic R, Roguljic M, Zakarija-Grkovic I, Vrdoljak D, Milic P, Puljak L, Marusic A. Framing the numerical findings of Cochrane plain language summaries: two randomized controlled trials. BMC Med Res Methodol. 2020 May 6;20(1):101. doi: 10.1186/s12874-020-00990-4. PMID 32375659
• Gainey K, Smith J, McCaffery K, Clifford S, Muscat D. Are plain language summaries published in health journals written according to instructions and health literacy principles? A systematic environmental scan. BMJ Open. 2024 Nov 27;14(11):e086464. doi: 10.1136/bmjopen-2024-086464. PMID 39608994
• Banic A, Fidahic M, Suto J, Roje R, Vuka I, Puljak L, Buljan I. Conclusiveness, linguistic characteristics and readability of Cochrane plain language summaries of intervention reviews: a cross-sectional study. BMC Med Res Methodol. 2022 Sep 10;22(1):240. doi: 10.1186/s12874-022-01721-7. PMID 36088293
• Zaretsky J, Kim JM, Baskharoun S, Zhao Y, Austrian J, Aphinyanaphongs Y, Gupta R, Blecker SB, Feldman J. Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and Format. JAMA Netw Open. 2024 Mar 4;7(3):e240357. doi: 10.1001/jamanetworkopen.2024.0357. PMID 38466307
• Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, Vignat J, Gralow JR, Cardoso F, Siesling S, Soerjomataram I. Current and future burden of breast cancer: Global statistics for 2020 and 2040. Breast. 2022 Dec;66:15-23. doi: 10.1016/j.breast.2022.08.010. Epub 2022 Sep 2. PMID 36084384
• Soldato D, Arecco L, Agostinetto E, Franzoi MA, Mariamidze E, Begijanashvili S, Brunetti N, Spinaci S, Solinas C, Vaz-Luis I, Di Meglio A, Lambertini M. The Future of Breast Cancer Research in the Survivorship Field. Oncol Ther. 2023 Jun;11(2):199-229. doi: 10.1007/s40487-023-00225-8. Epub 2023 Apr 1. PMID 37005952
• Nardin S, Mora E, Varughese FM, D'Avanzo F, Vachanaram AR, Rossi V, Saggia C, Rubinelli S, Gennari A. Breast Cancer Survivorship, Quality of Life, and Late Toxicities. Front Oncol. 2020 Jun 16;10:864. doi: 10.3389/fonc.2020.00864. eCollection 2020. PMID 32612947
• Shaikh H, Bradhurst P, Ma LX, Tan SYC, Egger SJ, Vardy JL. Body weight management in overweight and obese breast cancer survivors. Cochrane Database Syst Rev. 2020 Dec 11;12(12):CD012110. doi: 10.1002/14651858.CD012110.pub2. PMID 33305350
• Suto Pavicic J, Marusic A, Buljan I. Using ChatGPT to Improve the Presentation of Plain Language Summaries of Cochrane Systematic Reviews About Oncology Interventions: Cross-Sectional Study. JMIR Cancer. 2025 Mar 19;11:e63347. doi: 10.2196/63347. PMID 40106236
• Suto J, Marusic A, Buljan I. Linguistic analysis of plain language summaries and corresponding scientific summaries of Cochrane systematic reviews about oncology interventions. Cancer Med. 2023 May;12(9):10950-10960. doi: 10.1002/cam4.5825. Epub 2023 Mar 23. PMID 36951519
• Stephan D, Bertsch AS, Schumacher S, Puladi B, Burwinkel M, Al-Nawas B, Kammerer PW, Thiem DG. Improving Patient Communication by Simplifying AI-Generated Dental Radiology Reports With ChatGPT: Comparative Study. J Med Internet Res. 2025 Jun 9;27:e73337. doi: 10.2196/73337. PMID 40489773
• Furmaniak AC, Menig M, Markes MH. Exercise for women receiving adjuvant therapy for breast cancer. Cochrane Database Syst Rev. 2016 Sep 21;9(9):CD005001. doi: 10.1002/14651858.CD005001.pub3. PMID 27650122
• Jassim GA, Whitford DL, Hickey A, Carter B. Psychological interventions for women with non-metastatic breast cancer. Cochrane Database Syst Rev. 2015 May 28;(5):CD008729. doi: 10.1002/14651858.CD008729.pub2. PMID 26017383