Post-hoc recommendation explanations in e-learning: An empirical study using fuzzy tools

Post-hoc recommendation explanations in e-learning: An empirical study using fuzzy tools Ouahiba Remadnia1[0000−0002−2122−0876], Raciel Yera2[0000−0001−9759−261X], Rosa M. Rodríguez2[0000−0002−1736−8915], and Faiz Maazouzi1[0000−0003−4652−9121] 1Department of Computer Science, University of Souk Ahras, Algeria {w.remadnia,f.maazouzi}@univ-soukahras.dz 2Department of Computer Science, University of Jaén, Spain {ryera, rmrodrig}@ujaen.es Abstract. The increasing reliance on artificial intelligence (AI) in education has highlighted the necessity for transparent and trustworthy recommender systems. This paper presents a framework that combines collaborative filtering, specifically Singular Value Decomposition (SVD), with post-hoc, model-agnostic explanation techniques to generate interpretable recommendations in e-learning environments. To achieve this, we implement both crisp and fuzzy association rule mining approaches for explaining recommendations based on learners’ historical behaviors. The fuzzy model leverages membership degrees to account for interaction intensity, while the crisp model uses binarized user–item transactions. We generate explanations by mapping recommended items to association rules, and explanation quality is assessed using the Average Fidelity metric. Experimental evaluations on a real-world MOOC dataset demonstrate the effectiveness of the framework in delivering recommendations with interpretable justifications. The results validate the feasibility of integrating fuzzy-based explainability into recommendation pipelines, as the proposed approach enhances the crisp-based explanation approach, taking into account fidelity-based measures. Keywords: Explainable Recommender Systems ·E-Learning ·Fuzzy Association Rules ·Crisp explanation approach·Collaborative Filtering ·Singular Value Decomposition (SVD) · 1 Introduction AI has grown in popularity in recent years, leading to widespread adoption across a range of services. One of its key applications is in education [13], where recommender systems (RS) have significantly enhanced its overall impact. These systems have become an essential part of daily life, serving as effective tools to help people access online information that best fits their preferences and needs. Typically, RSs provide a list of suggestions based on the user’s profile and behavior, which may include interactions with available options, item attributes, and other relevant data. This preprint has not undergone peer review (when applicable) or any post-submission improvements or corrections. The Version of Record of this contribution is published in Proceedings of IDEAL 2025. LNCS 16239, and is available online at https:// doi.org/10.1007/978-3-032-10489-2_8 2 O. Remadnia et al. Recently, successful recommendation approaches have focused on computational intelligence techniques such as matrix factorization and deep learningbased methods to predict users’ unknown preferences and generate recommendation lists [16, 23]. However, these strategies suffer from a major drawback: their black-box nature reduces transparency and undermines credibility. Research in recommender systems (RS) has primarily emphasized improving the accuracy of recommendation algorithms [18]. At the same time, governments, society, and several studies—including [20] stress the importance of explainability in recommendation outcomes, alongside accuracy improvements. Explainable recommendation has emerged as an important dimension across various domains, including e-health and e-business, where it enhances decisionmaking by providing transparency and trust. Its relevance has also extended to highly active sectors such as e-commerce and e-learning [26], underscoring its role in supporting informed and confident user decisions. This contribution is motivated by the need to enhance trust and transparency in RSs for the e-learning domain. As learning recommendation continues to emerge as a primary research area [10], it is increasingly evident that current educational recommendation frameworks often rely on complex, black-box algorithms [8]. Although prior studies have explored explainability in recommendation systems [26], few have focused specifically on the e-learning context [26]. In addition, evaluating the impact of fuzzy tools in this task remains necessary [23]. Our work addresses these gaps by making the following contributions: –The developing of a post-hoc recommendation explanation framework to be used on an e-learning domain. –The evaluation of the impact of uncertainty management using fuzzy tools, as opposed to information processing through crisp models. The paper is organized as follows. Section 2 introduces a brief overview on recommendation explanation in e-learning. Section 3 details the proposed framework, respectively considering fuzzy and crisp information processing. Section 4 develops a set of experiments for measuring the discussed approaches, including dataset analysis, evaluation protocol, and experimental results. Section 5 concludes the paper, pointing out future directions. 2 The Explainable E-Learning Recommender System Domain Explainable e-learning RS represent a growing yet relatively underexplored area within the broader landscape of RS. Traditionally, e-learning RS focused on improving personalization and learner engagement through tailored content. However, the dimension of explainability has only recently gained relevance, driven by the need to increase transparency, user trust, and pedagogical alignment within educational environments [26]. In this context, surveys such as that of Adadi and Berrada [1] mapped the landscape of explainable AI in RS, highlighting both the potential and Post-hoc recommendation explanations in e-learning 3 the challenges of incorporating explainability into e-learning. Researchers have integrated traditional collaborative filtering (CF) and content-based (CB) approaches with explainability techniques, including memory-based methods [12], matrix factorization approaches with explicit factors [27], and post-hoc modelagnostic methods such as LIME and SHAP to complement black-box recommendations [17]. Moreover, explainable e-learning RS increasingly leverage heterogeneous data sources, such as learner performance metrics, course attributes, semantic content features, and engagement signals extracted from interactions. These diverse data sources offer rich contexts to generate recommendations that are not only personalized but also interpretable and aligned with learner goals. Recent studies have focused on enhancing transparency, trust, and engagement in personalized learning environments. For example, researchers have examined how explanations affect trust in e-learning platforms [14], user satisfaction and participation in learning projects [4], and student engagement through course recommendations [25]. Other works proposed ontology-driven frameworks for generating structured, learner-centered explanations using knowledge graphs [2] and applied Bayesian models to improve transparency in quiz recommendations [19]. These approaches rely on diverse datasets, ranging from small-scale user studies to large-scale educational repositories, and employ methods including hybrid CF, matrix factorization, CB filtering, knowledge graphs, and Bayesian Knowledge Tracing. However, the literature still shows limited exploration of post-hoc and model-agnostic explanation methods within e-learning recommender systems. Our contribution aims to fill this gap. 3 Exploring post-hoc explanation generation in e-learning recommendations The current section presents the framework used to generate post-hoc explanations for the recommendations provided by an e-learning platform, specifically a MOOC platform. Figure 1 illustrates the overall framework, which builds on ExplanationMining [15], one of the pioneering approaches for post-hoc recommendation generation. This method has recently been applied in domains such as e-health [22] and group recommendation [24]. The framework first preprocesses the raw interaction data of the system, converting it into a format suitable for subsequent stages. It also filters out users or items with very limited activity on the platform, since these could introduce noise and negatively affect later steps. Next, the framework develops two parallel processes. On one side, it trains a recommendation model and then generates the top-nrecommendations. On the other side, it discovers frequent items from the filtered dataset, leading to rule generation and pruning to avoid redundancy. Finally, the system uses these rules to generate recommendation 4 O. Remadnia et al. explanations, which it categorizes according to model fidelity. This framework produces explanations of the form: “We recommend ibecause you engaged with X.” The next subsections will present two implementations of this framework, respectively using fuzzy (Section 3.1) and crisp approaches (Section 3.2). Fig. 1. The general framework used for generating post-hoc explanations 3.1 Explanations generation through a fuzzy approach As pointed out in the Introduction section, the use of fuzzy approaches has not been sufficiently explored in the explanation generation tasks in RS. Nevertheless, it is important to highlight that in some scenarios, such as e-learning related, it could have a remarkable role given the continuous nature of some of the underlying data (e.g. watching time, as will be discussed below), where the use of techniques for managing the uncertainty could lead to better performance. Based on these premises, our proposal is driven by the following stages: Interaction processing and filtering. We begin by filtering the raw dataset of user-item interactions to retain only the most active users and the most popular items. The goal of this step is to increase the density of the interaction matrix and improving the statistical reliability of the discovered knowledge. Alternatively, we could opt for keeping the full dataset to be used in the next stages of the proposal. The experimental stage of the current contribution details how this filtering process was specifically done at this scenario. In any case, the filtered interactions are sorted chronologically per user. Recommendation model training and top-n recommendations. We train a CF model based on matrix factorization (SVD) using the rating values from the dataset. Particularly, we will use the popular FunkSVD approach [7], a matrix factorization technique used for rating prediction in RS. Rather than performing a full singular value decomposition, it learns low-dimensional latent factors for users and items by minimizing the regularized squared error between predicted and actual ratings using stochastic gradient descent (Equation 1). min b∗,p∗,q∗X (u,i)∈K (rui −ˆrui)2+λb2 u+b2 i+∥pu∥2+∥qi∥2(1) Post-hoc recommendation explanations in e-learning 5 The predicted rating ˆrui (Equation 2) for a user-item pair is computed as the sum of the global average rating (µ), user and item bias terms (buand bi), and the dot product of the user and item latent vectors (q⊤ ipu). ˆrui =µ+bu+bi+q⊤ ipu(2) This allows the model to capture both general tendencies (via biases) and nuanced interactions (via latent factors), making it well-suited for collaborative filtering tasks. We use this model to predict unknown ratings for user-item pairs, and generate top-N recommendations for each user. Frequent itemset discovery, rule generation, and redundancy pruning. In parallel, from the obtained dataset we construct fuzzy transactions, where each item is associated with a degree of membership corresponding to the preference degree of the item (between 0 and 1). These fuzzy transactions are used to mine frequent itemsets using a fuzzy version of the Apriori algorithm [9]. In this version, the support of an itemset is computed as the average of the mean membership degrees across all transactions in which the itemset appears. Let T={T1, T2, . . . , TN}be a set of fuzzy transactions, where each transaction Tiis a mapping Ti:I → [0,1] assigning to each item j∈ I a degree of membership Ti(j). For an itemset X⊆ I, the fuzzy support with mean aggregation is defined as: suppmean(X) = 1 |T | X T∈T   1 |X|X j∈X T(j) (3) Herein, we have defined the fuzzy support based on the arithmetic mean, taken into account that it is less restrictive than the largely-used minimum function, and therefore could be more adequate to e-learning scenarios where interaction levels varies across the different users. Based on these itemsets, we extract fuzzy association rules of the form A=⇒B, where Aand Bare item subsets. Rules are filtered by a minimum confidence threshold, and then pruned to eliminate redundant rules—those whose antecedents and consequents are subsets of stronger rules with similar confidence. Explanation generation. In this phase, a recommended item is considered explainable if there exists a fuzzy rule A=⇒Bsuch that the user’s training history contains all elements of Aand the recommended item belongs to B. Finally, we compute the model fidelity as the ratio of recommended items that are explainable in this way. Fidelity can be calculated globally or at the user level, allowing for fine-grained evaluation of explainability effectiveness. Algorithm 1 illustrates this approach. Herein, for each recommendation list (Line 4), first it is extracted the user preference history (Line 5). Subsequently, for each item in the recommendation list (Line 6), it is verified whether for some previously extracted rule, the items in the antecedent are in the user preference history, and the consequent matches with the current (Line 9). In such case, the rule is 6 O. Remadnia et al. registered as the explanation (Line 10). Finally, the model fidelity is calculated as the proportion of explainable items (Line 13). Algorithm 1 Post-hoc Explanation and Fidelity Calculation Require: Top-N recommendations R, user histories H, fuzzy rules F Ensure: Explanations E, global fidelity F 1: E ← ∅ 2: explained ←0 3: total ←0 4: for all (u, rec_list)∈ R do 5: Hu← H[u] 6: for all i∈rec_list do 7: total ←total + 1 8: for all (A→B)∈ F do 9: if A⊆Huand i∈Bthen 10: E[u]← E[u]∪ {(i, A →B)} 11: explained ←explained + 1 12: break 13: F←explained/total 14: return (E, F) 3.2 Explanations generation through a crisp approach This section is focused on formalizing an alternative crisp approach for implementing the framework screened at Figure 1. With this aim in mind, we will provide formalization and implementation details at the distinctive stage in relation to the previous fuzzy approach. In this alternative, transactions are built from the training set by binarizing interactions. An item is included in a user’s transaction if the user has either rated it or watched at least some portion of it. These binary transactions are also processed with the standard Apriori algorithm [3] to discover frequent itemsets above a given minimum support. Crisp association rules A=⇒Bare then extracted from these itemsets, using standard confidence thresholds. To reduce redundancy and in a similar way to the former approach, we apply a pruning step that removes rules whose antecedents and consequents are subsets of higherconfidence rules. Herein, the classic support is defined as follows [3]. Let T={T1, T2, . . . , TN} be a set of transactions, where each transaction Ti⊆ I is a subset of items. Given an itemset X⊆ I, its crisp support is defined as: suppcrisp(X) = 1 |T | X T∈T δ(X⊆T)(4) where δ(X⊆T)=1if X⊆T, and 0otherwise. Post-hoc recommendation explanations in e-learning 7 The remaining stages of this approach are performed in a similar way to those presented in the previous section. 4 Experiments In this section, we present the experimental validation of our proposed approach. We detail the dataset used, the evaluation protocol followed, and discuss the obtained results. 4.1 Dataset For our work, we used the E-learning Recommender System Dataset [5], a realworld dataset provided by Mandarine Academy, an Ed-Tech company specializing in corporate training through MOOCs, web conferences, and other digital learning methods. The dataset is publicly available and is designed to facilitate research in RS, particularly within the e-learning domain. The dataset contains user interaction data collected from the MOOC platform mooc.office365-training.com, spanning from early 2016 to late 2021. It captures user behavior in both explicit and implicit formats. Explicit ratings include interactions such as likes, social shares, and bookmarks, whereas implicit feedback is derived from watch time and page views. Given the low frequency of explicit interactions, the dataset primarily emphasizes implicit feedback, enabling the analysis of user engagement without requiring direct user input. In total, the dataset comprises a large number of explicit and implicit ratings provided by anonymized users over learning items that include associated metadata such as content descriptions and duration. Specifically, the ratings reflect user engagement with learning materials via explicit feedback (e.g., ratings) and implicit feedback (e.g., views or clicks). For the purpose of our study, we focus on the rating data. Two language-based versions of the dataset are available: (1) a French content dataset comprising 85,339 ratings, and (2) an English content dataset comprising 3,659 ratings. Each interaction in the dataset includes the following fields: user_id,item_id, watch_percentage,created_at, and rating. Here, the user ratings are derived by normalizing the watch percentage to a 1–10 scale. Further information on these dataset can be checked at Hafsa et al. [5]. 4.2 Evaluation protocol This subsection is focused on discussing the evaluation protocol and some implementation details associated to the developed experiments. –Interaction processing and filtering. At this stage we will consider two experimental scenarios, by using the whole dataset (full), or just a dataset composed of the top 500 more active users, and their evaluations over the top 500 popular items (top-500). 90% of each user data were using from training, leaving the remaining 10% for building the test set in each case. 8 O. Remadnia et al. –Recommendation model training and top-n recommendations. We will use the SVD approach implemented inside the Surprise library [6], which is a popular toolkit for deploying experiments in RS. The default parameters for this approach, defined in Surprise, will be used across our experiments. Top 10 recommendations will be only considered in the current contribution, due to the limitation in space. –Frequent itemsets discovery, rule generation, and pruning. In this stage, it will be used a fixed value of minconf = 0.2and several values of minimum support, tailored to each experimental scenario and dataset. Furthermore, for the case of the fuzzy support, it is necessary the definition of the function T(). Figure 2 illustrates the membership function that will be considered in this work, that receives as input the rating values in the range [1;10]. –Explanation generation. The explanation generation is performed across Algorithm 1, which receives as input the top n recommendations, user profiles and extracted rules, and does not depend on any specific parameter in the version provided in the current contribution. As pointed out, to quantify the alignment between the recommendations and the generated explanations, we employ the Average Fidelity metric, defined as: Fidelity =Number of explained recommendations Total number of recommendations (5) Higher fidelity indicates better interpretability of the recommendations without sacrificing personalization. Fig. 2. Membership function T() used for calculating the fuzzy support. 4.3 Results and discussion We present a comparative evaluation of our fuzzy and crisp explanation approaches across multiple configurations, using model fidelity as the main evaluation metric. Fidelity captures the proportion of recommended items that can Post-hoc recommendation explanations in e-learning 9 be post-hoc explained by mined association rules. Across all experimental conditions, the fuzzy explanation approach consistently outperforms the crisp alternative. This difference is especially notable at lower support thresholds, where the use of fuzzy logic enables partial evidence accumulation, resulting in higher explanatory coverage. For example, in the English dataset limited to the top 500 users and items, the fuzzy approach achieves a fidelity of 0.183 at a support of 0.03, which approximately duplicated the value obtained by the crisp explanation method under the same conditions. This regularity is repeated across the French dataset and the full datasets, suggesting that fuzzy rules are more effective at capturing latent, approximate relationships among items that crisp rules might overlook. The flexibility associated to the management of the fuzzy membership grades seems to be an important component for mitigating the effects of data sparsity and interaction variability, especially common in e-learning contexts. Table 1. English dataset. Top 500 active users and items. Minconf=0.2 Approach-Support values 0.03 0.04 0.05 0.06 0.07 0.08 Fuzzy explanation approach 0.183 0.091 0.082 0.082 0.064 0.008 Crisp explanation approach 0.097 0.071 0.046 0.046 - - Table 2. French dataset. Top 500 active users and items. Minconf=0.2 Approach-Support values 0.2 0.3 0.4 Fuzzy explanation approach 0.094 0.013 0.004 Crisp explanation approach 0.027 0.008 - As expected, increasing the minimum support threshold has a negative impact on fidelity, as fewer rules are mined and fewer recommendations can be explained. This decline is more relevant in the crisp setting, which quickly reaches zero fidelity at moderate support values, such as 0.07 or 0.08. In contrast, the fuzzy approach maintains non-zero fidelity across a wider range of thresholds, although it also declines as support increases. For instance, in the English full dataset, fuzzy fidelity decreases from 0.057 to 0.005 as the support increases from 0.04 to 0.08. The differences between experiments using the full dataset and those filtered to include only the top 500 most active users and most popular items further highlight the impact of data sparsity. Higher fidelity scores are obtained in the filtered case, as the denser interaction patterns lead to more frequent item cooccurrences and better rule coverage. In contrast, the full datasets include many