ChatGPT 3.5 Better Improves Comprehensibility of English, than Spanish, Generated Responses to Osteosarcoma Questions

TY - JOUR

T1 - ChatGPT 3.5 Better Improves Comprehensibility of English, than Spanish, Generated Responses to Osteosarcoma Questions

AU - Dias, Rosamaria

AU - Castan, Ashley

AU - Gotoff, Katie

AU - Kadkoy, Yazan

AU - Ippolito, Joseph

AU - Beebe, Kathleen

AU - Benevenia, Joseph

N1 - Publisher Copyright: © 2025 The Author(s). Journal of Surgical Oncology published by Wiley Periodicals LLC.

PY - 2025/6

Y1 - 2025/6

N2 - Background: Despite adequate discussion and counseling in the office, inadequate health literacy or language barriers may make it difficult to follow instructions from a physician and access necessary resources. This may negatively impact survival outcomes. Most healthcare materials are written at a 10th grade level, while many patients read at an 8th grade level. Hispanic Americans comprise about 25% of the US patient population, while only 6% of physicians identify as bilingual. Questions/Purpose: (1) Does ChatGPT 3.5 provide appropriate responses to frequently asked patient questions that are sufficient for clinical practice and accurate in English and Spanish? (2) What is the comprehensibility of the responses provided by ChatGPT 3.5 and are these modifiable?. Methods: Twenty frequently asked osteosarcoma patient questions, evaluated by two fellowship-trained musculoskeletal oncologists were input into ChatGPT 3.5. Responses were evaluated by two independent reviewers to assess appropriateness for clinical practice, and accuracy. Responses were graded using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level test (FKGL). The responses were then input into ChatGPT 3.5 for a second time with the following command “Make text easier to understand”. The same method was done in Spanish. Results: All responses generated were appropriate for a patient-facing informational platform. There was no difference in the Flesch Reading Ease Score between English and Spanish responses before the modification (p = 0.307) and with the Flesch-Kincaid grade level (p = 0.294). After modification, there was a statistically significant difference in comprehensibility between English and Spanish responses (p = 0.003 and p = 0.011). Conclusion: In both English and Spanish, none of the ChatGPT generated responses were found to be factually inaccurate. ChatGPT was able to modify responses upon follow-up with a simplified command. However, it was shown to be better at improving English responses than equivalent Spanish responses.

AB - Background: Despite adequate discussion and counseling in the office, inadequate health literacy or language barriers may make it difficult to follow instructions from a physician and access necessary resources. This may negatively impact survival outcomes. Most healthcare materials are written at a 10th grade level, while many patients read at an 8th grade level. Hispanic Americans comprise about 25% of the US patient population, while only 6% of physicians identify as bilingual. Questions/Purpose: (1) Does ChatGPT 3.5 provide appropriate responses to frequently asked patient questions that are sufficient for clinical practice and accurate in English and Spanish? (2) What is the comprehensibility of the responses provided by ChatGPT 3.5 and are these modifiable?. Methods: Twenty frequently asked osteosarcoma patient questions, evaluated by two fellowship-trained musculoskeletal oncologists were input into ChatGPT 3.5. Responses were evaluated by two independent reviewers to assess appropriateness for clinical practice, and accuracy. Responses were graded using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level test (FKGL). The responses were then input into ChatGPT 3.5 for a second time with the following command “Make text easier to understand”. The same method was done in Spanish. Results: All responses generated were appropriate for a patient-facing informational platform. There was no difference in the Flesch Reading Ease Score between English and Spanish responses before the modification (p = 0.307) and with the Flesch-Kincaid grade level (p = 0.294). After modification, there was a statistically significant difference in comprehensibility between English and Spanish responses (p = 0.003 and p = 0.011). Conclusion: In both English and Spanish, none of the ChatGPT generated responses were found to be factually inaccurate. ChatGPT was able to modify responses upon follow-up with a simplified command. However, it was shown to be better at improving English responses than equivalent Spanish responses.

UR - https://www.scopus.com/pages/publications/85216759972

UR - https://www.scopus.com/pages/publications/85216759972#tab=citedBy

U2 - 10.1002/jso.28109

DO - 10.1002/jso.28109

M3 - Article

C2 - 39898783

SN - 0022-4790

VL - 131

SP - 1692

EP - 1695

JO - Journal of Surgical Oncology

JF - Journal of Surgical Oncology

IS - 8

ER -