Evaluating ChatGPTs Multilingual Performance in Clinical Nutrition Advice Using Synthetic Medical Text: Insights from Central Asia
Adilmetova G. Nassyrov R. Meyerbekova A. Karabay A. Varol H.A. Chan M.-Y.
March 2025Elsevier B.V.
Journal of Nutrition
2025#155Issue 3729 - 735 pp.
Background: Although large language models like ChatGPT-4 have demonstrated competency in English, their performance for minority groups speaking underrepresented languages, as well as their ability to adapt to specific sociocultural nuances and regional cuisines, such as those in Central Asia (for example, Kazakhstan), still requires further investigation. Objectives: To evaluate and compare the effectiveness of the ChatGPT-4 system in providing personalized, evidence-based nutritional recommendations in English, Kazakh, and Russian in Central Asia. Methods: This study was conducted from 15 May to 31 August, 2023. On the basis of 50 mock patient profiles, ChatGPT-4 generated dietary advice, and responses were evaluated for personalization, consistency, and practicality using a 5-point Likert scale. To identify significant differences between the 3 languages, the Kruskal–Wallis test was conducted. Additional pairwise comparisons for each language were carried out using the post hoc Dunns test. Results: ChatGPT-4 showed a moderate level of performance in each category for English and Russian languages, whereas in Kazakh language, outputs were unsuitable for evaluation. The scores for English, Russian, and Kazakh were as follows: for personalization, 3.32 ± 0.46, 3.18 ± 0.38, and 1.01 ± 0.06; for consistency, 3.48 ± 0.43, 3.38 ± 0.39, and 1.09 ± 0.18; and for practicality, 3.25 ± 0.41, 3.37 ± 0.38, and 1.07 ± 0.15, respectively. The Kruskal–Wallis test indicated statistically significant differences in ChatGPT-4s performance across the 3 languages (P < 0.001). Subsequent post hoc analysis using Dunns test showed that the performance in both English and Russian was significantly different from that in Kazakh. Conclusions: Our findings show that, despite using identical prompts across 3 distinct languages, the ChatGPT-4s capability to produce sensible outputs is limited by the lack of training data in non-English languages. Thus, a customized large language model should be developed to perform better in underrepresented languages and to take into account specific local diets and practices.
AI , Chatbots , LLMs , Medical Natural Language Processing (NLP) , personalized diet , precision nutrition
Text of the article Перейти на текст статьи
Department of Biomedical Sciences, School of Medicine, Nazarbayev University, Astana, Kazakhstan
Department of Medicine, School of Medicine, Nazarbayev University, Astana, Kazakhstan
Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, Astana, Kazakhstan
Department of Biomedical Sciences
Department of Medicine
Institute of Smart Systems and Artificial Intelligence
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026