Automatic feedback through natural language processing using a chatbot-based simulated patient (PEPE) for the training of mental health professionals

Journal of Artificial Intelligence and Computing Applications (2025) - Special Issue - 3(2): 33–37 lipsum subcaption Expanded abstract Automatic feedback through natural language processing using a chatbot-based simulated patient (PEPE) for the training of mental health professionals Ricardo Cruz 1,*, Violeta Felix Romero 2, Marcela Rosas 2, Diana Patricia Tzek 2, and Ivan Meza 1 1Instituto de Investigaciones en Matem´aticas Aplicadas y en Sistemas, UNAM 2Facultad de Psicolog´ıa, UNAM ABSTRACT Mental, neurological, and substance use disorders are highly prevalent worldwide; however, it is estimated that between 75% and 95% of affected individuals lack access to treatment. Therefore, it is essential to promote the development of effective strategies for the training and evaluation of mental health professionals. There is evidence supporting the effectiveness of using standardized simulated patients to train healthcare professionals. One of the main challenges is the extent to which these simulations can authentically and validly represent real patients. The use of Artificial Intelligence has recently been explored to enhance the ecological validity of simulations through Natural Language Processing (NLP). NLP enables various applications, including chatbots and virtual assistants that can engage in natural, human-like conversations. In healthcare, chatbots can serve as valuable tools for training and providing feedback to professionals on a wide range of topics. This project aims to evaluate the impact of automated feedback—delivered through a simulated patient chatbot named PEPE—on the training of healthcare professionals at the Addiction Prevention Center, Faculty of Psychology, UNAM, in areas such as depression, anxiety, and substance abuse. Keywords: LLMs, chatbots, mental health professionals 1. Introduction The use of simulations in the assessment and training of health professionals implies the strengthening of competencies for the management of mental health risk indicators in the population [1]. Simulations provide participants with immediate and detailed feedback on their performance, allowing practice in a controlled environment, reducing the likelihood of making mistakes with real patients. Training health professionals to address issues such as substance abuse, depression, and anxiety at the primary level is crucial to ensure the implementation of effective interventions. The use of technological resources based on natural language processing ensures that the simulation is as close as possible to the reality faced daily by mental health professionals. There are examples of chatbots that are used to diagnose potential mental health problems based on the person’s profile and symptoms, offering help in areas such as stress, depression, sleep, and self-esteem [2, 3, 4]. However, very few focus on being used as a virtual patient for the training of health professionals, and even E-mail address: carlosricardo[email protected] https://doi.org/10.5281/zenodo.17227862 ©2025 The Author(s). Published by Maikron. This is an open access article under the CC BY license. This article is part of the Special Issue on ICAIMH 2025. ISSN: 3061-8843 Journal of Artificial Intelligence and Computing Applications (2025) 3(2) Figure 1. Main Page of the Chatbot Simulated Patient (PEPE) fewer in the Spanish language [5]. Therefore, one of the main contributions of this work will be the development of a chatbot in Spanish that provides automatic feedback to mental health professionals on depression, anxiety, and substance use. This will allow professionals in training to identify areas for improvement, facilitating continuous learning and the refinement of clinical skills. 2. Methodology With the aim of training mental health specialists from the Addiction Prevention Center at the Faculty of Psychology of UNAM in providing counseling on depression, anxiety, and substance abuse, a chatbot named PEPE (Spanish acronym for Programa de Entrenamiento con Pacientes Estandarizados) was developed. PEPE integrates various technologies, such as an API for querying ChatGPT-4o, as well as a web system developed at the Institute of Research in Applied Mathematics and Systems at UNAM. This system uses Python as the programming language, along with a socket-based architecture for thread management, database handling, dialogue engine, security, authentication, and graphical interface as showed in Figure 1. Additionally, an administration web page was created to store the data and training sessions of the mental healthcare professionals as shown in Figure 2. ChatGPT-4o was pre-trained using transcriptions of simulated patient-student session conversations on topics of substance use, depression, and anxiety, creating a script for the fine-tuning of the Large Language Model (LLM). The simulated cases as shown in Fig 3 were developed based on the analysis of videos and files of real cases, obtaining a sample of the representative sociodemographic and clinical characteristics of typical cases who are treated at a primary care training center. Once the case was developed, the indicators and dialogues were validated by expert judges, and finally, PEPE was trained to represent these patients in simulated interactions, providing standardized information and presenting all the stimuli of the case [6]. The functionalities currently available in PEPE are as follows: First use case for evaluating a healthcare professional. The use case includes elements such as signs, symptoms, and strategies to reduce alcohol consumption, as mentioned in the Mental Health Gap Action Programme (mhGAP) Intervention Guide [7]. In this use case, PEPE simulates a 30-year-old male patient who presents with concerns related to excessive alcohol consumption (See Figure 4). 3. Results and Discussion Currently, PEPE can engage in dialogue with a mental healthcare professional; presenting general information about the case such as personal details, the reason for consultation, and triggers and consequences associated with alcohol use. However, the following objectives are yet to be achieved: 34 Journal of Artificial Intelligence and Computing Applications (2025) 3(2) Figure 2. Webpage for registering mental health professionals in PEPE’s administrative interface. Figure 3. Example of a simulated alcohol abuse case that includes incorrect, incomplete, and nonpertinent answers. 35 Journal of Artificial Intelligence and Computing Applications (2025) 3(2) Figure 4. Screenshot showing a dialogue in Spanish between the chatbot PEPE, simulating a 30-year-old male patient expressing concerns about alcohol abuse, and a trainee mental healthcare professional. •Automatic Feedback for Mental Health Professionals. For each question posed by PEPE, the healthcare professional will provide a response, which will be immediately evaluated by PEPE, receiving instant feedback and an assessment. •Improvements to the conversation model to make it more adaptive. •Improvements to the historical storage of conversations to contextualize the responses. •Evaluation of improvements at the conversation flow level. •Evaluation of the system with end users. •Creation of a results log. •Adding the use cases for depression and anxiety. 4. Conclusion The research highlights the development and application of a chatbot-based simulated patient, PEPE, to train mental health professionals in addressing issues such as depression, anxiety, and substance abuse. The implementation of automatic feedback via Natural Language Processing allows for real-time assessment and continuous learning. Future work will focus on refining the conversation model to enhance its adaptability, evaluating healthcare professionals’ responses, and improving the storage of conversation histories for better context. Furthermore, end-user evaluations will be conducted, and a results log will be created to track progress. The integration of AI technologies into mental health training, while highly promising, also presents important ethical considerations. A central concern is the protection of personal data generated during simulated interactions. In this project, data privacy is safeguarded by ensuring that all training information is provided anonymously by healthcare professionals from the Addiction Prevention Center at UNAM. This approach protects sensitive information, promotes ethical data handling, and minimizes the risk of compromising individual privacy in educational and research contexts. Furthermore, the implementation of AI systems like PEPE demands transparency regarding the accuracy and limitations of the feedback they provide. It is essential that users understand that AI-generated evaluations are based on algorithmic models rather than clinical judgment, and should be interpreted as supportive tools to guide learning—not as definitive assessments. Despite these limitations, PEPE demonstrates strong potential to enhance the training and skill development of mental health professionals, ultimately contributing to more effective and accessible mental healthcare delivery. However, future evaluations are necessary to assess the system’s effectiveness and its actual impact on educational outcomes. Ethics Statement This study did not involve human participants or animals and therefore did not require ethical approval. 36 Journal of Artificial Intelligence and Computing Applications (2025) 3(2) CRediT authorship contribution statement Ricardo Cruz: Conceptualization, Software, Writing – original draft. Violeta Felix Romero: Investigation, Writing – review editing. Marcela Rosas: Investigation, Writing – review editing. Diana Patricia Tzek: Investigation, Writing – review editing. Iv´an Meza: Conceptualization, Software, Writing – review editing. Declaration of Generative AI and AIassisted technologies in the writing process The authors utilized Grammarly and ChatGPT to refine sentence structure and enhance readability. No content was generated by AI; all scientific insights and original ideas are the authors’ own. Declaration of competing interest The authors declare no competing interests. Acknowledgements The authors would like to thank UNAM, especially for the support received from the Support Program for Research and Technological Innovation Projects (PAPIIT) TA300124. References [1] C. Osborn and R. Cash, “Effects of interview training with simulated patients on suicide, threat, and abuse assessment,” Athens Journal of Social Sciences, vol. 8, no. 4, pp. 245–258, oct 2021. doi: 10.30958/ajss.8-4-3 . [2] G. Cameron, D. Cameron, G. Megaw, R. Bond, M. Mulvenna, S. O’Neill, C. Armour, and M. McTear, “Assessing the usability of a chatbot for mental health care,” in Internet Science, S. S. Bodrunova, O. Koltsova, A. Følstad, H. Halpin, P. Kolozaridi, L. Yuldashev, A. Smoliarova, and H. Niedermayer, Eds. Cham: Springer International Publishing, 2019, pp. 121–132. doi: 10.1007/978-3-030-17705-811. [3] A. F. Ur Rahman Khilji, S. R. Laskar, P. Pakray, R. A. Kadir, M. S. Lydia, and S. Bandyopadhyay, “Healfavor: Dataset and a prototype system for healthcare chatbot,” in 2020 International Conference on Data Science, Artificial Intelligence, and Business Analytics (DATABIA), 2020, pp. 1–4. doi: 10.1109/DATABIA50434.2020.9190281 . [4] C. Gudmundsen Høiland, A. Følstad, and A. Karahasanovic, “Hi, can i help? exploring how to design a mental health chatbot for youths,” Human Technology, vol. 16, no. 2, pp. 139–169, Aug 2020. doi: 10.23987/ht.88715 . [Online]. Available: https://ht.csr-pub.eu/index.php/ht/article/view/5 [5] A. Su´arez, A. Adanero, V. D´ıaz-Flores Garc´ıa, Y. Freire, and J. Algar, “Using a virtual patient via an artificial intelligence chatbot to develop dental students’ diagnostic skills,” International Journal of Environmental Research and Public Health, vol. 19, no. 14, p. 8735, 2022. doi: 10.3390/ijerph19148735 . [Online]. Available: https://www.mdpi.com/1660-4601/19/14/8735 [6] V. F´elix Romero, D. Ortiz G´omez, S. Morales Chain´e, and C. Uriarte Rojo, “Caso simulado estandarizado: Evaluaci´on conductual en profesionales de la salud en adicciones,” Acta de Investigaci´on Psicol´ogica, vol. 11, no. 3, pp. 87–98, dic 2021. doi: 10.22201/fpsi.20074719e.2021.3.395 . [Online]. Available: https://www.revista-psicologia.unam.mx/revista aip/ index.php/aip/article/view/395 [7] World Health Organization, mhGAP Intervention Guide for Mental, Neurological and Substance Use Disorders in Non-specialized Health Settings: Version 2.0, World Health Organization, Geneva, 2016. [Online]. Available: https://www.who.int/publications/i/item/9789241549790 37