GN5-1 AI Chatbot Incubator Project: Closure Report

© GÉANT Association on behalf of the GN5-1 project. The research leading to these results has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101100680 (GN5-1). Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. 10-12-2024 AI Chatbot Incubator Project: Closure Report Grant Agreement No.: 101100680 Work Package: 6 Task Item: 4 Dissemination Level: PU (Public) Lead Partner: PSNC Document ID: GN5-1-24-RB4CLA Authors: Marcin Wolski (PSNC), Branko Marović (UoB), Maciej Łabędzki (PSNC), Kjiroski Kiril (UKIM), Marcin Rochowiak (PSNC), Dom Mayerl (GÉANT), Ivana Golub (PSNC) Abstract The project focuses on developing an AI chatbot for the GÉANT Network Automation eAcademy, part of its online learning platform. The goal is to enhance the learning experience by providing a digital assistant capable of conducting natural and engaging user interactions. Running from June to December 2024, the six-month initiative successfully developed and integrated the chatbot into the eAcademy platform. The AI chatbot leverages advanced AI technologies to deliver personalised, interactive support, transforming how users engage with the platform. AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA ii Table of Contents Executive Summary 1 1 Introduction 2 2 Project Results 3 2.1 Software Solution 3 2.2 Technological Framework 4 3 Next Steps and Future Work 5 4 Conclusions 6 References 7 Glossary 8 Table of Figures Figure 2.1: AI chatbot on the GÉANT eAcademy main page 3 Figure 2.2: Chatbot components within the GÉANT eAcademy environment 4 AI Chatbot Incubator Project: Closure Report Document ID: <Doc Property: Keywords> 1 Executive Summary The GN5-1 NETDEV Incubator [NETDEV_Inc] project developed an AI chatbot to enhance the GN Network Automation eAcademy, part of GÉANT’s online learning platform. Conducted from June to December 2024, this project aimed to improve user engagement and accessibility in e-learning through advanced AI-driven interactions. It was a collaborative effort involving PSNC, AMRES, UKIM, and the GÉANT Association. The AI chatbot functions as a digital learning assistant, providing personalised guidance, interactive navigation, and proactive assistance. Key features include tailored recommendations, direct content access, and periodic engagement through quizzes. The solution was deployed as a proof-of-concept on a test instance of the Moodlebased eAcademy platform and evaluated by IT students. The project delivered two primary results: (1) a software solution demonstrating the chatbot’s capabilities and (2) a technological framework combining large language models (LLMs) and retrieval-augmented generation (RAG). These technologies ensure accurate, context-aware responses by integrating pre-trained AI capabilities with real-time information retrieval. Evaluations highlighted the chatbot’s effectiveness in addressing both specific and general queries while identifying areas for refinement, such as improved contextual understanding and safety measures. This project aligns with GÉANT’s educational mission, demonstrating potential for broader applications across the eAcademy. Future iterations may incorporate expanded content recognition, integration with additional modules, and support for content optimisation. The AI chatbot’s architecture, based on open models like Meta’s Llama, avoids vendor lock-in and ensures GDPR compliance. The successful implementation and evaluation of the AI chatbot affirm its value as an innovative tool in digital education. It lays a groundwork for future development within GÉANT’s initiatives, promising to enhance learning experiences and accessibility across its expanding platform. AI Chatbot Incubator Project: Closure Report Document ID: <Doc Property: Keywords> 2 1 Introduction The landscape of education is undergoing a significant transformation, driven by technological advancements that are reshaping the way we learn and interact with knowledge. Traditional methods of information dissemination, often characterised by static content and passive learning, are giving way to more dynamic and engaging approaches. Innovative tools are essential for enhancing communication between service functions and their users. In response to this revolution, the concept of an AI chatbot for the GÉANT Network Automation eAcademy [NeA] was sketched and then implemented as a NETDEV Incubator project, a part of the Network Development Work Package (WP6) [NETDEV] of the GN5-1 project. Unlike conventional search functions, the AI chatbot serves as a digital learning assistant, guiding students through the complexities of the platform, stimulating their curiosity, and providing timely and accurate answers to their queries. By engaging in natural and coherent conversations, the chatbot fosters a more interactive and enjoyable learning environment, making education accessible to a wider audience. The AI chatbot project started in June 2024 and ended in December 2024. The project was a joint effort by several NRENs, namely PSNC, AMRES, and UKIM and the GÉANT Association. The project team consisted of PSNC technical staff responsible for software delivery and operation, and Subject Matter Experts (SMEs) from supporting NRENs focusing on requirements specification and evaluation. AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA 3 2 Project Results The project resulted in two main deliverables: (1) a software solution that is a proof-of-concept for a learning bot for the GÉANT eAcademy [GeA] and (2) a technological framework for an AI chatbot. 2.1 Software Solution The AI chatbot software solution is a proof-of-concept of a learning bot for the GÉANT eAcademy, deployed on a test instance of the eAcademy Moodle platform and evaluated by a selected group of students [AI_Chatbot]. Figure 2.1: AI chatbot on the GÉANT eAcademy main page The AI chatbot assists students with the learning path of the Network Automation courses and provides recommendations on how to start, what to do next after completing a selected course, or directly presents specific parts of the course containing the relevant content. Moreover, the bot proactively assists students by providing short quizzes after periods of inactivity. Evaluation The evaluation of the AI chatbot for GÉANT eAcademy materials highlighted its strengths in handling both domain-specific and vague queries, providing intuitive navigation and ensuring safe and ethical interactions. While the chatbot demonstrated clear potential, a few areas requiring continuous attention were also identified. User tests, which included theme-specific questions, conversational interactions, and attempts to confuse the chatbot, revealed a need for enhanced contextualisation, navigability, and safety mechanisms in case of AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA 4 transitioning such a chatbot to a public-facing system. The chatbot significantly supports GÉANT’s and GLAD’s educational missions by broadening its applicability to interactive delivery mechanism and possibility to expand its educational scope by blending several existing and possibly new knowledge domains. With expanded knowledge coverage and lessons learned from prototype testing, the chatbot could evolve into a robust publicfacing system, offering a versatile AI-assisted learning environment and ensuring a seamless and secure user experience. 2.2 Technological Framework The produced AI chatbot is not just a tool but a complete technological framework delivering AI-powered interactive solutions. It is built upon two leading technologies, namely Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG). LLMs are the core technology behind next-generation AI solutions, sometimes called genAI apps. They are trained on massive text datasets, such as books and articles, to learn patterns and relationships between words. RAG functions as a fact-checker for a Small Large Language Model (sLLM). It enables the AI model to consult real-time information from reliable sources before crafting responses and enhancing accuracy and trustworthiness. Figure 2.2: Chatbot components within the GÉANT eAcademy environment The diagram illustrates a simplified architecture of the AI chatbot system with the following components: • eAcademy Website: An HTML page (container) serving as main interface integrated with the e-learning platform’s front end. It is loaded from the e-academy.geant.org server, powered by the Moodle platform. • Chatbot IFrame: An embedded frame within the HTML page displaying the chatbot interface. • Moodle Application Server: A suite of Moodle components, acting as the GÉANT eAcademy platform, serving pages and course content. • Chatbot Application Server: A developed suite of specialised software components (including AI models and NLP modules) delivering the AI chatbot functions in an SaaS model. AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA 5 3 Next Steps and Future Work The knowledge and experience gained with LLMs and RAG in this project can be utilised in future GÉANT NETDEV Incubator projects. The architectural patterns and underlying technologies of the chatbot support the addition of new educational resources and learning tracks for the GÉANT eAcademy. The eAcademy is growing at a rapid pace. There are 66 individual courses over the whole Academy in 2024 and 49 more in development. This provides great opportunities for future growth and deployment of the AI chatbot once it reaches the production-grade maturity. The developed chatbot solution could evolve into a robust public-facing system, supporting a seamless and secure AI-assisted learning environment and user experience. We hope that future iterations of the project will see the implementation of chatbot functionality across the entire eAcademy. A future chatbot should not only integrate all existing modules but also integrate new content into its knowledge base. Additionally, support guides for authors should also be developed to optimise course materials and course design for better absorption by the chatbot. This can also enhance human digestibility of course materials. PSNC plans to leverage the AI chatbot framework to deliver new interactive applications for various communities. AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA 6 4 Conclusions The AI chatbot for the GÉANT eAcademy is a unique and innovative solution leveraging the latest technologies and methodologies. Delivered as a complete and fully-functional Proof of Concept (PoC), it includes a widget and backend services with an AI model tailored to GÉANT Network Automation eAcademy courses. The architectural patterns, underlying technologies of the chatbot, such as LLMs and RAG, as well as developed testing approach, support the addition of new educational resources and learning tracks, and the utilisation of content stored outside the platform, that could be used in course materials as reference documents. The AI chatbot has been integrated with Moodle on the test instance of the GÉANT Moodle platform and is currently operated on the PSNC infrastructure. By using an open foundation model of Meta’s Llama, there is no risk of commercial vendor lock-in. Additionally, all input data is managed internally, ensuring GÉANT GDPR compliance. The evaluation demonstrated the AI chatbot’s adequate interpretation and use of GÉANT eAcademy materials, which would make it a valuable tool for the GÉANT community. It is able to assist users effectively while clearly explaining its scope and limitations. Testing indicated that transitioning to a public-facing system requires enhanced contextualisation, navigability, and safety mechanisms. However, it also proved the chatbot’s ability to support GÉANT’s educational missions and expand its applicability. AI Chatbot Incubator Project: Closure Report Document ID: GN5-1-24-RB4CLA 7 References [NETDEV] https://wiki.geant.org/display/NETDEV [NETDEV_Inc] https://wiki.geant.org/display/NETDEV/NETDEV+Incubator [AI_Chatbot] https://uat-academy.geant.org/moodle/ [GeA] https://e-academy.geant.org/moodle/ [NeA] https://wiki.geant.org/display/NETDEV/Network+eAcademy