Task S6_RT1.3 – Deliverable DS6_RT1.3.4 - Report on novel methodologies for optimal recommendation by empowering Machine Learning algorithms and Learning-to-Rank models

1 1 Task S6_RT1.3 – Deliverable DS6_RT1.3.4 - Report on novel methodologies for optimal recommendation by empowering Machine Learning algorithms and Learning-to-Rank models 1 INTRODUCTION This document presents the research and results developed during my research grant (Assegno di Ricerca) for the iNEST project, specifically within the scope of DS6_RT1.3.4, titled Report on novel methodologies for optimal recommendation by empowering Machine Learning algorithms and Learning-to-Rank models. The core of the research focused on the design and development of vertical search engines dedicated to tourist destinations, accommodation facilities, and territorial information. These systems were required to incorporate aspects of fairness and/or explainability. In the context of search engines, and more broadly Information Retrieval (IR), fairness refers to the balanced treatment of items in rankings, avoiding discrimination against entities belonging to sensitive or minority groups. Fairness is closely related to the datasets used to train machine learning (ML) models, which may contain sampling biases or reflect social prejudices. This can lead models to generate rankings that unfairly disadvantage certain user groups (e.g., based on gender) or categories of tourist services (e.g., hotels versus apartments). Explainability in IR refers to the ability to explain how decisions are made by the ML-based ranking models. The aim is to provide a simple and intuitive link between the input features and the model’s relevance predictions, highlighting which aspects of the query, user profile, or returned item influenced the ranking. In summary, fairness helps build trust among stakeholders, such as hospitality providers who want assurance that their offerings are ranked fairly. Explainability, on the other hand, can improve the experience for tourists and help businesses understand which features they may need to improve in order to gain more visibility. During the research grant, I extensively analyzed fairness in Learning-to-Rank (LTR), which led to the publication of two project milestones: LambdaRank Gradients are Incoherent [16, 19] and LambdaFair: a Fair and Effective LambdaMART [20,21]. 2 STATE OF THE ART Current search engine technologies rely on machine learning algorithms, particularly LTR models [3,4,28], to re-rank results presented to users. These systems leverage complex features related to the query and user context, as well as characteristics of the items being ranked (e.g., documents, points of interest), to identify the most relevant results that fulfill users’ information needs. 2 In terms of fairness in LTR models, the literature primarily categorizes existing approaches into three broad groups: pre-processing,in-processing, and postprocessing methods [34,35]. Pre-processing methods aim to mitigate bias in the training data prior to model learning. These techniques are generally model-agnostic and focus on enhancing individual fairness without modifying the underlying ML algorithms. For instance, [15] proposes transforming feature vectors into more equitable representations to improve individual fairness. Similarly, [36] introduces a strategy that balances the dual objectives of maintaining an accurate representation of the data and hiding sensitive group membership information to enhance fairness. In-processing methods incorporate fairness constraints directly into the learning process, typically by modifying the loss function or adding regularization terms that encourage fairness. A notable example is DELTR [32], which reduces grouplevel disparities in item exposure within rankings while maintaining relevance. Similarly, Fair-PG-Rank [26] addresses fairness by modeling exposure as expected attention, operating under a merit-based constraint that ensures items receive exposure proportional to their relevance, thus balancing fairness and effectiveness. Recent research has identified stochastic Plackett-Luce (PL) ranking models [18,24] as a powerful in-processing approach for jointly optimizing effectiveness and fairness metrics. Unlike deterministic algorithms that depend on heuristic optimization, PL models are fully differentiable and can be optimized via stochastic gradient descent directly on ranking metrics. However, estimating gradients in practice is challenging because it involves summing over all possible permutations of items. To tackle this, Oosterhuis [22] proposed PL-Rank, an efficient method to estimate gradients of PL models with respect to both fairness and effectiveness metrics by leveraging specific properties of ranking metrics and PL distributions. Further advancing this line of work, the PL-Rank-3 algorithm [23] introduced by Oosterhuis achieves unbiased gradient estimates with computational efficiency comparable to state-of-theart sorting algorithms. Building upon these advances, Gorantla et al. [11] proposed Group-Fair-PL, which optimizes group fairness through a novel objective that computes expected ranking utility over only those rankings satisfying explicit group representation constraints. Group-Fair-PL enforces either equal or proportional representation of protected items within the top-𝑘ranks. Equal representation requires the same number of items from each group in the top-𝑘, whereas proportional representation reflects the relative group proportions in the dataset. Post-processing methods modify the output rankings after model inference to satisfy fairness criteria. FA*IR [31] adjusts the ranking to ensure that the proportion of protected candidates meets minimum thresholds at different ranking depths. The CFAΘalgorithm [33] provides a mechanism to continuously trade off between individual and group fairness, enabling practitioners to fine-tune the fairness criteria applied to the final ranking output. 3 3 BACKGROUND Learning-to-rank methods are a cornerstone of information retrieval systems, used to re-order a set of documents in response to a query. Given a query 𝑞and a candidate document set 𝐷 = {𝑑1,…,𝑑𝑛}, an LTR model learns a scoring function that assigns each document 𝑑𝑖∈𝐷a score 𝑠𝑖. The final ranking 𝜋is obtained by sorting documents in descending order of 𝑠𝑖. Formally, 𝜋is a permutation of {1,…,𝑛}such that 𝜋[𝑟]=𝑖means 𝑑𝑖occupies the 𝑟-th position, where 𝑟∈ℕ+and 1≤𝑟≤𝑛. For any 𝑟ℓ≤𝑟𝑢, we write 𝜋[𝑟ℓ,𝑟𝑢] to denote the subsequence of documents ranked from position 𝑟ℓto 𝑟𝑢. In particular, the prefix of length 𝑟𝑢is 𝜋[1,𝑟𝑢]. Finally, we use 𝑑𝑖≺𝜋𝑑𝑗 to indicate that 𝑑𝑖appears before 𝑑𝑗in the ranking 𝜋. 3.1 EFFECTIVENESS METRIC We evaluate both effectiveness and fairness. For effectiveness, we use the Normalized Discounted Cumulative Gain (NDCG) metric [13]. Let 𝜋be a ranking of 𝐷of size 𝑛, and let each document 𝑑𝑖∈𝐷have a relevance label 𝑦𝑖. The Discounted Cumulative Gain part of NDCG is defined as: DCG(𝜋)= 𝑛 ∑ 𝑟=1 2𝑦𝜋[𝑟] −1 log2(𝑟+1) and IDCG(𝜋)=max 𝜋′DCG(𝜋′). Then, the NDCG metric is defined as: NDCG(𝜋)= DCG(𝜋) IDCG(𝜋), which yields a value in [0,1], with higher values indicating better alignment with the true relevance ordering. To reflect typical user behavior, where only the top results are examined, NDCG is computed at cutoff 𝑘, denoted NDCG@𝑘. 3.2 FAIRNESS METRIC In [20, 21], the fairness metric we employed is Normalized Discounted Difference (rND) [30], which captures group fairness as statistical parity [34]. Given a ranking 𝜋of 𝐷with |𝒢+|protected items, rND evaluates whether, for various cutoffs 𝑟, the top-𝑟positions contain a fraction of protected items equal to |𝒢+|/𝑛. Here 𝒢+⊆𝐷 denotes the protected subset. 4 rND divides the ranking into overlapping prefixes of lengths 𝑟=𝑏,2𝑏,3𝑏,…, where 𝑏>1is the bin size. For each prefix length 𝑟, it computes the absolute deviation from the ideal proportion and discounts it by log2(𝑟). The rND metric is defined as follows: rND(𝜋)= 1 rDmax 𝑛 ∑ 𝑟=𝑏,2𝑏,… 1 log2(𝑟)∣|𝒢+ 𝜋[1,𝑟]| 𝑟−|𝒢+ 𝜋[1,𝑛]| 𝑛∣ , where 𝒢+ 𝜋[1,𝑟] is the set of protected items in the top-𝑟, and rDmax is the worst-case sum obtained by ranking all elements of the group with smaller cardinality first. The metric lies in [0,1], with lower values indicating greater fairness. As with NDCG, the rND metric can be computed up to a certain cutoff 𝑘, i.e., rND@𝑘. 3.3 DATASETS We performed an extensive evaluation on publicly available datasets. In LambdaRank Gradients are Incoherent [16,19], we used the following three datasets. The Istella-X [17] dataset has the highest average number of documents per query and the largest number of non-relevant documents. Yahoo! Learning to Rank Challenge Set 1 [6] dataset is the smallest dataset, with about 700,000documents and an average of 23.73documents per query. The MSLR Web30K Fold 1 [25] dataset consists of feature vectors extracted from query-document pairs. It is the most balanced dataset, with about half of the documents labeled as relevant. All datasets have graded 5-level relevance labels ranging from 0(non-relevant) to 4(highly relevant). For LambdaFair: a Fair and Effective LambdaMART [20, 21] instead, we considered three additional datasets widely used in fairness-aware learning-to-rank research: Statlog (German Credit Data) [12], and Home Mortgage Disclosure Act (Connecticut) [8]. The German Credit Data dataset contains binary relevance labels related to creditworthiness, with 1,000individuals per query sampled to create 100,000 queries. We derived two variants of this dataset by splitting protected and unprotected groups based on: •Sex: females as protected, males as unprotected [14,26,29], •Age: individuals under 35as protected, those 35and older as unprotected [1]. The Home Mortgage Disclosure Act (Connecticut) dataset includes home mortgage loan records across US states since 2007. We focused on Connecticut, using data from 2013–2015 for training, 2016 for validation, and 2017 for testing [11]. Similar to German Credit Data, we sampled 50individuals per query with a 4:1ratio of non-approved to approved loans, for a total of 100,000queries. The protected group consists of females, and the unprotected group of males. 5 Since MSLR-30K does not provide protected group labels, we followed previous works [1,14, 27,29] and used the QualityScore2 feature (QS2, feature ID 133) as a discriminatory attribute. Following Vardasbi et al. [27], documents with QS2 values below 10are assigned to the protected group, and those with QS2 values equal or above 10to the unprotected group. All datasets are partitioned into training, validation, and test sets following a 60%- 20%-20% split. 3.4 LAMBDAMART LambdaMART [3] is a learning algorithm widely used in IR to train effective ranking models by directly optimizing a target metric. It addresses the non-differentiability and flat regions of ranking metrics [2, 4, 9, 19] by leveraging a smooth gradient approximation. Formally, let 𝑞be a query, 𝐷 = {𝑑1,…,𝑑𝑛}the candidate documents, and 𝑌 = {𝑦1,…,𝑦𝑛}their relevance labels. LambdaMART constructs a ground-truth pairwise preference set 𝑃where (𝑖,𝑗) ∈ 𝑃if and only if 𝑦𝑖> 𝑦𝑗. For each document 𝑑𝑖, LambdaMART computes an approximated gradient 𝜆𝑍 𝑖for the ranking metric 𝑍(e.g., NDCG@𝑘) as: 𝜆𝑍 𝑖= ∑ 𝑗 ∶ (𝑖,𝑗)∈𝑃 𝜆𝑍 𝑖𝑗 − ∑ 𝑘 ∶ (𝑘,𝑖)∈𝑃 𝜆𝑍 𝑘𝑖 ,(1) where each partial approximated gradient 𝜆Z 𝑖𝑗 is defined as: 𝜆𝑍 𝑖𝑗 =𝜕𝐶(𝑠𝑖−𝑠𝑗) 𝜕𝑠𝑖=−𝜎 1+𝑒𝜎(𝑠𝑖−𝑠𝑗)∣Δ𝑍𝑖𝑗∣. (2) Here, 𝑠𝑖and 𝑠𝑗are the scores predicted for 𝑑𝑖and 𝑑𝑗,𝐶is the RankNet cost function [5], and Δ𝑍𝑖𝑗 =𝑍(𝜋𝑗𝑖)−𝑍(𝜋𝑖𝑗) denotes the change in metric 𝑍when swapping 𝑑𝑖and 𝑑𝑗in the current ranking. The scalar 𝜎controls the sigmoid’s steepness. These 𝜆𝑍 𝑖values serve as pseudo-gradients in iterative optimization algorithms such as neural networks (e.g., LambdaRank [4]) or gradient-boosted trees (e.g., LambdaMART [3]). 4 CONTRIBUTION The primary contribution of the research conducted during the grant period was centered on the development of LTR algorithms with an emphasis on fairness, particularly through in-processing methods. This line of work addresses a critical issue 6 in the deployment of modern search and recommendation systems: the risk of amplifying biases and producing systematically unfair rankings. The research aimed to integrate fairness considerations directly into the optimization process of LTR models, rather than relying on pre-processing (data transformation) or post-processing (re-ranking) techniques. This in-processing approach is both theoretically and practically significant, as it allows for fairness constraints or objectives to be embedded within the model’s learning dynamics, potentially leading to more balanced and equitable outputs without sacrificing effectiveness. Two main contributions were achieved during the grant period, each resulting in peerreviewed publications: •LambdaRank Gradients are Incoherent [16, 19]: This work revisits the foundational LambdaRank framework, uncovering a mathematical incoherence in the gradient formulation used for training that can unfairly misrank equally relevant items. By analyzing the gradient dynamics, the research demonstrates that the implicit assumptions made by LambdaRank can lead to unintended optimization behaviors. The paper proposes a refined understanding of ranking gradients, paving the way for more principled and fair LTR methods. •LambdaFair: a Fair and Effective LambdaMART [20, 21]: Building on the insights above, this paper introduces LambdaFair, an extension of LambdaMART that incorporates fairness constraints directly into the learning process. LambdaFair is designed to mitigate exposure disparity among groups while maintaining competitive ranking performance. Extensive experimental evaluation on publicly available datasets confirms that LambdaFair achieves a better trade-off between fairness and effectiveness compared to existing baselines. Overall, the research contributes both theoretical insights and practical algorithms that advance the state of the art in fair information retrieval. It demonstrates that it is possible to reconcile the goals of effectiveness and fairness in ranking, and it provides tools that can be adopted or extended in real-world systems. 4.1 LAMBDARANK GRADIENTS ARE INCOHERENT Many LTR algorithms still rely on gradient-based optimization, either by approximating the ranking metric or by constructing heuristic gradients, as these methods have proven to be highly effective in practice. As mentioned above, LambdaMART optimizes a non-differentiable objective by generating ad hoc gradients for each document. These gradients are based on heuristic assumptions about each document’s contribution to the overall ranking and its interactions with other documents. Consequently, the gradients produced by LambdaMART are inherently approximate. 7 In [16, 19], we demonstrate that LambdaMART and its related methods, such as LambdaRank [4] and the loss functions introduced by [28], exhibit an intrinsic issue stemming from their heuristic foundations: the generation of incoherent gradients. Our analysis revealed three critical and previously undocumented behaviors in LambdaMART and its derivatives: •i) We found that LambdaMART suffers from gradient incoherencies that hinder the learning process. Specifically, it can assign stronger downward gradient forces to documents with higher relevance than to those with lower relevance. This leads the model to mislearn the correct ranking order. •ii) We observed that optimizing truncated IR metrics exacerbates these incoherencies, further degrading model performance. Although truncated metrics (e.g., top-𝑘) are useful for focusing learning on the top ranks and reducing training time, they also increase the risk of incorrect gradient estimation. •iii) Finally, we discovered that optimizing truncated metrics introduces unfair treatment of equally relevant documents. In particular, when two or more documents have the same relevance, those ranked lower in the list receive weaker upward gradient signals than those in higher positions. This puts lower-ranked yet equally relevant documents at a disadvantage, since they require a stronger push to improve their rank. These findings reveal that the widely used LambdaMART algorithm and its derivatives can exhibit unfair behavior, especially in scenarios where fairness across equally relevant items is essential. 4.1.1 GRADIENT INCOHERENCY AND UNFAIR DOCUMENT COMPARISON Gradient-based learning algorithms, such as artificial neural networks or gradientboosted decision trees, run iterative updates to build a ranker that minimizes a given cost function 𝐶. For instance, gradient-boosted decision trees iteratively learn a new tree that approximates 𝜕𝐶/𝜕𝑠𝑖for each document 𝑑𝑖in the training set 𝐷and its score 𝑠𝑖. Unfortunately, most IR metrics are rank-based: they depend on ranking 𝜋rather than on 𝑠𝑖. This makes the cost function either flat or non-differentiable. Note that 𝜋is the ranking over the documents 𝑑𝑖∈𝐷sorted in decreasing order of scores 𝑠𝑖predicted by the ranker, and 𝜋[𝑖]denotes the position of document 𝑑𝑖in the ranking. LambdaRank’s cost function defined by [4] is one of the most relevant approaches used to tackle this problem, and it stems from the RankNet cost proposed by [5], which is enhanced by considering the impact on the IR metric. The gradient is computed on the basis of pair-wise lambdas 𝜆𝑖𝑗 as defined in Equation 1, where 𝑃is the set of ordered documents pairs (𝑖,𝑗)such that 𝑦𝑖> 𝑦𝑗, i.e., 𝑃 = {(𝑖,𝑗) ∣ 𝑑𝑖,𝑑𝑗∈ 𝐷 ∧ 𝑦𝑖> 𝑦𝑗}. The value of 𝜆𝑖𝑗 estimates the change on the 8 Table 1: Detailed computation of LambdaMART gradients. 𝑑𝑖𝜋[𝑖] 𝑦𝑖𝑠𝑖𝜆𝑖 𝑑11 4 0.02 𝜆1=𝜆12 +𝜆13 ≈0.176+0.221≈0.397 𝑑22 0 0.01 𝜆2=−𝜆12 −𝜆32 ≈−0.176−0.004≈−0.180 𝑑33 1 0.00 𝜆3=−𝜆13 +𝜆32 ≈−0.221+0.004≈−0.217 cost function 𝐶when the difference between the two scores 𝑠𝑖and 𝑠𝑗increases or decreases. To manage user behavior and increase training efficiency, real-world applications of information retrieval systems mostly try to optimize the effectiveness only for the first 𝑘results. IR metrics naturally provide a truncated version, i.e. NDCG@𝑘is computed by considering only the contribution of the top-𝑘ranked documents. By training the model to optimize a truncated metric 𝑍to a certain truncation level 𝜏, pairs of documents ranked beyond 𝜏are not considered since the corresponding contribution to the metric is equal to 0. Thus, in order to reduce the training time, the number of document pairs in 𝑃is limited while computing the gradients 𝜆𝑖in Equation 1 by replacing the set 𝑃with 𝐼𝜏={(𝑖,𝑗)|𝑑𝑖,𝑑𝑗∈𝐷∧𝑦𝑖>𝑦𝑗∧min(𝜋[𝑖],𝜋[𝑗])≤𝜏}. It is important to note that, although closely related, the truncation level 𝜏is different from the metric cutoff 𝑘. The former affects the number of document pairs to process, and the latter affects the evaluation of the metric. Moreover, they may not be equal, i.e. 𝜏may be slightly larger than 𝑘to process more pairs during the training phase. Table 1 shows an example of LambdaMART gradients when maximizing NDCG. The query has only three documents with their ranks 𝜋[𝑖]and scores 𝑠𝑖predicted by the model, and relevance label 𝑦𝑖. The top-ranked document with relevance equal to 4 and is correctly pushed up by the gradient 𝜆1. Interestingly enough, the second and third documents are misranked with labels 0 and 1 respectively. The LambdaMART gradient is negative for both documents, but the document with the larger label is pushed down with greater strength. We may conclude that such gradients are not going to improve the ranking but rather increase the gap between the two misranked documents. We call this phenomenon gradients incoherency. To explain in detail the reason for such behavior, in Table 1 we report the computation of the document gradients 𝜆𝑖as a function of the pair-wise 𝜆𝑖𝑗 according to Equation 1 in case of the NDCG metric. Document 𝑑1has a positive gradient 𝜆1 as it is ranked higher than documents with smaller relevance labels. Document 𝑑2 is the least relevant and receives a negative gradient contribution from both the other documents. Unexpectedly, document 𝑑3receives the strongest downward push even if it has a higher label than 𝑑2. The reason is that swapping document 𝑑1with 𝑑3has a larger impact on the NDCG than swapping 𝑑1with 𝑑2, resulting in 𝜆13 >𝜆12. LambdaMART prefers avoiding the risk of moving 𝑑1to the third position rather than pushing 𝑑3up to the second place. Indeed, this comes from the discount factor of NDCG metric that demotes documents’ contributions in the lower ranks. 15 minimal number of inter-bin swaps is applied between equally relevant documents belonging to different groups to balance protected and unprotected items across ranking bins and approximate |𝒢+|/𝑛. Since these documents have equal relevance, such swaps represent ties that do not affect NDCG but serve to reduce rND by improving group balance. The second stage proceeds in the same manner as used for generating 𝜋rND+. 4.2.2 RESUSLTS Figure 3 illustrates the trade-off between effectiveness and fairness, where fairness is measured as (1−rND)%(higher values indicate better fairness). Specifically, we present NDCG and rND values evaluated at cutoff 𝑘 = 15on the test sets of each dataset. With the exception of the MSLR-30K dataset, where ΔrND performs best, rND+emerges as the overall top-performing variant, achieving higher fairness with only a slight reduction in effectiveness compared to LambdaMART and the other LambdaFair variants. Compared to the fair baselines, LambdaFair consistently achieved higher effectiveness with a slight decrease in fairness. 48.0 50.0 52.0 54.0 Effectiveness 73.6 74.4 75.2 76.0 76.8 Fairness MSLR-30K 90.0 92.5 95.0 97.5 100.0 Effectiveness 72.0 74.0 76.0 78.0 80.0 Statlog (Age) 92.0 94.0 96.0 98.0 100.0 Effectiveness 79.6 80.0 80.4 80.8 Statlog (Sex) 88.0 90.0 92.0 94.0 96.0 Effectiveness 80.9 81.0 81.0 81.1 HMDA-CT LambdaMART NDCG+ ∆rND rND+ PL-Rank-3 Group-Fair-PL Figure 3: Effectiveness and fairness trade-off. Fairness = (1−rND)%. Results for models trained and evaluated with cutoff 𝑘=15. 5 CONCLUSIONS During this research grant, I developed two novel algorithms aimed at creating fair and effective search engines: Lambda-eX [16,19] and LambdaFair [20,21]. Both algorithms employ an in-processing approach, integrating fairness constraints directly into the learning-to-rank optimization process. These strategies allow the model to simultaneously maximize ranking effectiveness while actively mitigating discrimi- 16 natory biases that may arise among equally relevant items or groups requiring protection. More specifically, these algorithms address two critical aspects of fairness in ranking systems. First, they ensure individual fairness by treating items with comparable relevance scores equitably during the training phase, thus reducing unwarranted discrimination between similarly qualified entities. Second, they enforce group fairness by promoting statistically fair exposure for protected groups across different ranking positions. This dual focus supports the development of ranking models that balance user relevance needs with fairness objectives, avoiding unfair underrepresentation of protected or disadvantaged groups. In practical terms, these advancements have significant implications for domains such as tourism, where search results influence user decisions and economic opportunities. For instance, these algorithms can ensure that tourism-related entities, such as accommodations, attractions, or activities, that are equally relevant to a user’s query receive equal consideration, preventing biases that might favor popular or centrally located options unfairly. Additionally, protected entities, such as activities located in less frequented or marginalized areas, are guaranteed proportional exposure in the ranking results. This leads to more equitable visibility and can help support diverse and sustainable tourism development. Overall, the development and evaluation of Lambda-eX and LambdaFair contribute to advancing fairness-aware learning-to-rank research by demonstrating that fairness constraints can be effectively incorporated into ranking optimization without significantly compromising relevance. Future work will focus on extending these methods to handle multiple protected attributes simultaneously, exploring adaptive fairness constraints based on user preferences, and applying the algorithms to other domains where fairness in ranking is paramount. REFERENCES [1] A. Bower, H. Eftekhari, M. Yurochkin, and Y. Sun, “Individually fair rankings,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=71zCSP_HuBN [2] S. Bruch, “An alternative cross entropy loss for learning-to-rank,” in WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, J. Leskovec, M. Grobelnik, M. Najork, J. Tang, and L. Zia, Eds. ACM / IW3C2, 2021, pp. 118–126. [Online]. Available: https://doi.org/10.1145/3442381.3449794 [3] C. J. C. Burges, “From ranknet to lambdarank to lambdamart: An overview,” 2010. [4] C. J. C. Burges, R. Ragno, and Q. V. Le, “Learning to rank with nonsmooth cost functions,” in Advances in Neural Information Processing Systems 19, Pro- 17 ceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, B. Schölkopf, J. C. Platt, and T. Hofmann, Eds. MIT Press, 2006, pp. 193–200. [5] C. J. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. N. Hullender, “Learning to rank using gradient descent,” in Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, ser. ACM International Conference Proceeding Series, L. D. Raedt and S. Wrobel, Eds., vol. 119. ACM, 2005, pp. 89–96. [6] O. Chapelle and Y. Chang, “Yahoo! learning to rank challenge overview,” in Proceedings of the Yahoo! Learning to Rank Challenge, held at ICML 2010, Haifa, Israel, June 25, 2010, ser. JMLR Proceedings, O. Chapelle, Y. Chang, and T. Liu, Eds., vol. 14. JMLR.org, 2011, pp. 1–24. [Online]. Available: http://proceedings.mlr.press/v14/chapelle11a.html [7] M. Corporation, LightGBM Release 3.3.3.99, 2023. [8] F. F. I. E. Council, “HMDA Data Publication,” 2017, released due to the Home Mortgage Disclosure Act. [Online]. Available: https://www.consumerfinance. gov/data-research/hmda/historic-data/ [9] P. Donmez, K. M. Svore, and C. J. C. Burges, “On the local optimality of lambdarank,” in Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, J. Allan, J. A. Aslam, M. Sanderson, C. Zhai, and J. Zobel, Eds. ACM, 2009, pp. 460–467. [Online]. Available: https://doi.org/10.1145/1571941.1572021 [10] R. Fisher, The design of experiments. 1935. Edinburgh: Oliver and Boyd, 1935. [11] S. Gorantla, E. Bhansali, A. Deshpande, and A. Louis, “Optimizing learning-torank models for ex-post fair relevance,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, G. H. Yang, H. Wang, S. Han, C. Hauff, G. Zuccon, and Y. Zhang, Eds. ACM, 2024, pp. 1525–1534. [Online]. Available: https://doi.org/10.1145/3626772.3657751 [12] H. Hofmann, “Statlog (German Credit Data),” UCI Machine Learning Repository, 1994, DOI: https://doi.org/10.24432/C5NC77. [13] K. Järvelin and J. Kekäläinen, “Cumulated gain-based evaluation of IR techniques,” ACM Trans. Inf. Syst., vol. 20, no. 4, pp. 422–446, 2002. [Online]. Available: http://doi.acm.org/10.1145/582415.582418 18 [14] J. Kotary, F. Fioretto, P. V. Hentenryck, and Z. Zhu, “End-to-end learning for fair ranking systems,” in WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, F. Laforest, R. Troncy, E. Simperl, D. Agarwal, A. Gionis, I. Herman, and L. Médini, Eds. ACM, 2022, pp. 3520–3530. [Online]. Available: https://doi.org/10.1145/3485447.3512247 [15] P. Lahoti, K. P. Gummadi, and G. Weikum, “ifair: Learning individually fair data representations for algorithmic decision making,” in 35th IEEE International Conference on Data Engineering, ICDE 2019, Macao, China, April 8-11, 2019. IEEE, 2019, pp. 1334–1345. [Online]. Available: https://doi.org/10.1109/ICDE.2019. 00121 [16] C. Lucchese, F. Marcuzzi, and S. Orlando, “Does lambdamart do what you expect?” in Proceedings of the 13th Italian Information Retrieval Workshop (IIR 2023), Pisa, Italy, June 8-9, 2023, ser. CEUR Workshop Proceedings, F. M. Nardini, N. Tonellotto, G. Faggioli, and A. Ferrara, Eds., vol. 3448. CEUR-WS.org, 2023, p. 72. [Online]. Available: https://ceur-ws.org/Vol-3448/paper-16.pdf [17] C. Lucchese, F. M. Nardini, R. Perego, S. Orlando, and S. Trani, “Selective gradient boosting for effective learning to rank,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, K. Collins-Thompson, Q. Mei, B. D. Davison, Y. Liu, and E. Yilmaz, Eds. ACM, 2018, pp. 155–164. [Online]. Available: https://doi.org/10.1145/3209978.3210048 [18] R. D. Luce, Individual Choice Behavior: A Theoretical analysis. New York, NY, USA: Wiley, 1959. [19] F. Marcuzzi, C. Lucchese, and S. Orlando, “Lambdarank gradients are incoherent,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, I. Frommholz, F. Hopfgartner, M. Lee, M. Oakes, M. Lalmas, M. Zhang, and R. L. T. Santos, Eds. ACM, 2023, pp. 1777–1786. [Online]. Available: https://doi.org/10.1145/3583780.3614948 [20] ——, “Lambdafair: A fair and effective lambdamart,” in Proceedings of the 14th Italian Information Retrieval Workshop, IIR 2024, Udine, Italy, September 5-6, 2024, E. Maddalena, S. Mizzaro, K. Roitero, and M. Viviani, Eds., 2024. [21] ——, “Lambdafair for fair and effective ranking,” in Advances in Information Retrieval - 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6-10, 2025, Proceedings, Part IV, ser. Lecture Notes in Computer Science, C. Hauff, C. Macdonald, D. Jannach, G. Kazai, F. M. Nardini, F. Pinelli, F. Silvestri, and N. Tonellotto, Eds., vol. 15575. Springer, 2025, pp. 197–213. [Online]. Available: https://doi.org/10.1007/978-3-031-88717-8_15 19 [22] H. Oosterhuis, “Computationally efficient optimization of plackett-luce ranking models for relevance and fairness,” in SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, and T. Sakai, Eds. ACM, 2021, pp. 1023–1032. [Online]. Available: https://doi.org/10.1145/3404835.3462830 [23] ——, “Learning-to-rank at the speed of sampling: Plackett-luce gradient estimation with minimal computational complexity,” in SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, E. Amigó, P. Castells, J. Gonzalo, B. Carterette, J. S. Culpepper, and G. Kazai, Eds. ACM, 2022, pp. 2266–2271. [Online]. Available: https://doi.org/10.1145/3477495.3531842 [24] R. L. Plackett, “The analysis of permutations,” Journal of the Royal Statistical Society. Series C (Applied Statistics), vol. 24, no. 2, pp. 193–202, 1975. [Online]. Available: http://www.jstor.org/stable/2346567 [25] T. Qin and T. Liu, “Introducing LETOR 4.0 datasets,” CoRR, vol. abs/1306.2597, 2013. [Online]. Available: http://arxiv.org/abs/1306.2597 [26] A. Singh and T. Joachims, “Policy learning for fairness in ranking,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 5427–5437. [Online]. Available: https://dl.acm.org/doi/10.5555/3454287.3454774 [27] A. Vardasbi, F. Sarvi, and M. de Rijke, “Probabilistic permutation graph search: Black-box optimization for fairness in ranking,” in SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, E. Amigó, P. Castells, J. Gonzalo, B. Carterette, J. S. Culpepper, and G. Kazai, Eds. ACM, 2022, pp. 715–725. [Online]. Available: https://doi.org/10.1145/3477495.3532045 [28] X. Wang, C. Li, N. Golbandi, M. Bendersky, and M. Najork, “The lambdaloss framework for ranking metric optimization,” in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Z. Broder, M. J. Zaki, K. S. Candan, A. Labrinidis, A. Schuster, and H. Wang, Eds. ACM, 2018, pp. 1313–1322. [29] H. Yadav, Z. Du, and T. Joachims, “Policy-gradient training of fair and unbiased ranking functions,” in SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 20 Virtual Event, Canada, July 11-15, 2021, F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, and T. Sakai, Eds. ACM, 2021, pp. 1044–1053. [Online]. Available: https://doi.org/10.1145/3404835.3462953 [30] K. Yang and J. Stoyanovich, “Measuring fairness in ranked outputs,” in Proceedings of the 29th International Conference on Scientific and Statistical Database Management, Chicago, IL, USA, June 27-29, 2017. ACM, 2017, pp. 22:1–22:6. [Online]. Available: https://doi.org/10.1145/3085504.3085526 [31] M. Zehlike, F. Bonchi, C. Castillo, S. Hajian, M. Megahed, and R. Baeza-Yates, “Fa*ir: A fair top-k ranking algorithm,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 - 10, 2017, E. Lim, M. Winslett, M. Sanderson, A. W. Fu, J. Sun, J. S. Culpepper, E. Lo, J. C. Ho, D. Donato, R. Agrawal, Y. Zheng, C. Castillo, A. Sun, V. S. Tseng, and C. Li, Eds. ACM, 2017, pp. 1569–1578. [Online]. Available: https://doi.org/10.1145/3132847.3132938 [32] M. Zehlike and C. Castillo, “Reducing disparate exposure in ranking: A learning to rank approach,” in WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, Y. Huang, I. King, T. Liu, and M. van Steen, Eds. ACM / IW3C2, 2020, pp. 2849–2855. [Online]. Available: https://doi.org/10.1145/3366424.3380048 [33] M. Zehlike, P. Hacker, and E. Wiedemann, “Matching code and law: achieving algorithmic fairness with optimal transport,” Data Min. Knowl. Discov., vol. 34, no. 1, pp. 163–200, 2020. [Online]. Available: https://doi.org/10.1007/s10618-019-00658-8 [34] M. Zehlike, K. Yang, and J. Stoyanovich, “Fairness in ranking, part I: score-based ranking,” ACM Comput. Surv., vol. 55, no. 6, pp. 118:1–118:36, 2023. [Online]. Available: https://doi.org/10.1145/3533379 [35] ——, “Fairness in ranking, part II: learning-to-rank and recommender systems,” ACM Comput. Surv., vol. 55, no. 6, pp. 117:1–117:41, 2023. [Online]. Available: https://doi.org/10.1145/3533380 [36] R. S. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning fair representations,” in Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, ser. JMLR Workshop and Conference Proceedings, vol. 28. JMLR.org, 2013, pp. 325–333. [Online]. Available: http://proceedings.mlr.press/v28/zemel13.html