Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity

. . Latest updates: hps://dl.acm.org/doi/10.1145/3757887.3763007 . . RESEARCH-ARTICLE Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity BENEDIKT HÖLTGEN, University of Tübingen, Tubingen, Baden-Wuremberg, Germany . NURIA OLIVER, ELLIS Alicante, Alicante, Alicante, Spain . . . Open Access Support provided by: . ELLIS Alicante . University of Tübingen . PDF Download 3757887.3763007.pdf 22 December 2025 Total Citations: 0 Total Downloads: 234 . . Published: 05 November 2025 . . Citation in BibTeX format . . EAAMO '25: Equity and Access in Algorithms, Mechanisms, and Optimization November 5 - 7, 2025 Pittsburgh, USA . . Conference Sponsors: SIGECOM SIGAI EAAMO '25: Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (November 2025) hps://doi.org/10.1145/3757887.3763007 ISBN: 9798400721403 . Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity Benedikt Höltgen University of Tübingen Tübingen, Germany [email protected] Nuria Oliver ELLIS Alicante Alicante, Spain [email protected] Abstract Fairness through Unawareness (FtU) describes the idea that discrimination against demographic groups can be avoided by not considering group membership in the decisions or predictions. This idea has long been criticized in the machine learning literature as not being sufficient to ensure fairness. In addition, the use of additional features is typically thought to increase the accuracy of the predictions for all groups, so that FtU is sometimes thought to be detrimental to all groups. In this paper, we show both theoretically and empirically that FtU can reduce algorithmic discrimination without necessarily reducing accuracy. We connect this insight with the literature on Model Multiplicity, to which we contribute with novel theoretical and empirical results. Furthermore, we illustrate how, in a real-life application, FtU can contribute to the deployment of more equitable policies without losing efficacy. Our findings suggest that FtU is worth considering in practical applications, particularly in high-risk scenarios, and that the use of protected attributes such as gender in predictive models should be accompanied by a clear and well-founded justification. CCS Concepts •Theory of computation → Theory and algorithms for application domains;•Computing methodologies → Learning linear models;Supervised learning;Philosophical/theoretical foundations of artificial intelligence. Keywords Machine Learning, Model Multiplicity, Rashomon, Fairness, Logistic Regression, Disparate Impact ACM Reference Format: Benedikt Höltgen and Nuria Oliver. 2025. Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity. In Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’25), November 05–07, 2025, Pittsburgh, PA, USA. ACM, New York, NY, USA, 15 pages. https: //doi.org/10.1145/3757887.3763007 1 Introduction Nurtured by the ever-increasing availability of data and compute, as well as a widespread fondness of quantification in general [ 37 ] and Big Data in particular [ 8 ], machine learning-based systems This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. EAAMO ’25, Pittsburgh, PA, USA ©2025 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2140-3/25/11 https://doi.org/10.1145/3757887.3763007 are increasingly used to assist decision making in a variety of areas of society, including in high-stakes domains such as healthcare, finance, law enforcement, and the provision of social services. Leveraging data is often seen not only as a way to improve decisions but also as a way to avoid human bias and enhance objectivity and fairness. However, there is mounting evidence of a variety of problems that can arise in different contexts as a result of this trend [ 45 ]. One such problem is unintended algorithmic discrimination, which can arise for different reasons, including biased training data, flawed model assumptions, unequal representation of different demographic groups, or misuse of the models. As a result, algorithmic decisions may systematically disadvantage certain individuals or communities grouped by their protected attributes, such as gender, race, age, or religion. These concerns have prompted a substantial body of research focused on understanding, detecting, and mitigating algorithmic biases [ 3 ]. A seemingly straightforward approach to preventing algorithmic discrimination across groups is the exclusion of protected attributes from the set of features that the model can “see". This idea is known as Fairness through Unawareness or FtU [ 3 ]. However, the algorithmic fairness literature has shown that avoiding the use of protected attributes in the models is generally not sufficient to prevent discrimination [ 10 , 34 ]: other variables can encode group membership information and thus their use could lead to discrimination. Furthermore, FtU is often criticized not only for being an unreliable approach to achieve fairness, but also for potentially reducing the predictive performance of the model for all groups [13].1 Such results align with a common intuition in the Big Data era: that more data (and attributes) lead to better, fairer, and more objective decisions [ 8 ]. A recent practical example of this way of thinking can be found in Austria, where gender was used as an input in an algorithm designed to allocate job training programs to unemployed individuals. Officials dismissed criticisms regarding potential discriminatory outcomes by arguing that the algorithm’s predictions reflect the actual chances individuals face in the labour market (as cited in Allhutter et al . [1] ). More generally, many arguments in favour of including protected attributes rely on the idea that machine learning models aim to estimate “true" probabilities from data, which is a questionable assumption [ 26 ]. As a result of this assumption, modelling choices may be accepted with limited scrutiny or justification, despite their significant implications [ 33 ]. However, there is growing evidence that policy outcomes are highly sensitive to modelling decisions, including the choice of thresholds, input variables, and specific design parameters [ 2 , 27 ]. In parallel, emerging research on model multiplicity highlights that different models trained on the same data can perform similarly in terms 1Such a reduction in accuracy need not lead to a reduction in utility [12]. 241 EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Benedikt Höltgen and Nuria Oliver of accuracy, yet differ substantially in their behaviour, particularly with respect to fairness [ 7 ]. This perspective highlights that there is no clear hierarchy between models in terms of accuracy and that modelling choices need to be checked against alternative choices. In light of these concerns, we revisit the commonly held scepticism toward Fairness through Unawareness (FtU). While we do not suggest that FtU offers a simple solution to achieve algorithmic fairness, we demonstrate that, contrary to prevailing assumptions, omitting protected attributes does not necessarily harm predictive performance and can contribute to more equitable outcomes. To substantiate these claims, we analyse the relationship between data, model class, and fairness, both from a theoretical and from an empirical perspective. While we derive general insights, we place particular emphasis on logistic regression because of its broad and extensive use in practice to build models from tabular data. Regarding fairness, we focus on disparate impact (DI) or statistical/demographic parity, which compares the positive classification rates for different groups according to their protected attribute. Our findings suggest that including protected attributes can be especially problematic in the context of logistic regression, as their use by such models can easily exacerbate disparities between protected groups. By linking the concepts of FtU and model multiplicity, we argue for a shift in perspective: fairness should not be seen as a constraint or a trade-off, but as a deliberate modelling choice that can lead to more equitable outcomes without necessarily sacrificing predictive performance. In sum, the main contributions of this work are as follows: • We analyse the relationship between disparate impact and base rate difference in the data both for any classifier and for logistic regression. In the case of logistic regression, we demonstrate the risk of exacerbated unfairness depending on the classification thresholds. • We contribute to the understanding of model multiplicity and algorithmic fairness by providing mathematical results that characterise the potential for multiplicity and connect it to fairness: (1) general upper bounds on model multiplicity and changes in disparate impact which are tight for unconstrained models; and (2) lower bounds on achievable reductions in disparate impact based on FtU for logistic regression. • We empirically demonstrate how unaware models can yield similar levels of accuracy than aware models with significant improvements in fairness in multiple settings. • We highlight the practical implications on our work by discussing a real-world example related to employment programs in a European country. The structure of this article is as follows. In Section 2, we provide an overview of related work on fairness, less discriminatory alternatives, and model multiplicity. In Section 3, we highlight the importance of modelling choices by demonstrating that there is no straightforward connection between disparate impact and differences in base rates, even assuming well-calibrated models. We also show that disparate impact can often be higher than the base rate difference for aware models (i.e. models using protected attributes), especially for models based on logistic regression. In Section 4, we frame our work from the perspective of model multiplicity. We provide new theoretical upper bounds and relate them to disparate impact and to fairness through unawareness. In the case of logistic regression, we also provide theoretical lower bounds for reducing disparate impact. We illustrate our insights empirically in Section 5, showing that FtU can provide fairer models with almost equal accuracy across multiple datasets, protected attributes, and model classes. Section 6 illustrates the impact of our work in a real-world scenario where an aware logistic regression model was used to assign unemployed individuals to job training programs. We show how in this case an unaware model would be equally performative but fairer. Finally, the main conclusions and lines of future work are described in Section 7. 2 Related work 2.1 Algorithmic fairness and disparate impact Under the influence of legal, economic, and philosophical notions of discrimination as well as several high-profile investigations into existing software, an extensive body of work on algorithmic fairness has introduced a variety of metrics to quantify and address discrimination in machine learning systems [ 3 ]. One line of work focuses on error rate disparities, i.e., the differences in false positive or false negative rates across demographic groups. However, some scholars have argued that error rates should serve as diagnostic tools rather than direct targets for fairness interventions [ 3 ]. Another influential line of work draws on legal and policy traditions, particularly anti-discrimination law in the United States, introducing the concept of disparate impact—also referred to in the machine learning literature as demographic or statistical parity [ 4 , 21 ]. Disparate Impact focuses on algorithmic outcomes rather than errors: it compares the rates at which individuals from different demographic groups receive positive classifications (e.g., job offers, university admissions, loan approvals). If one group consistently receives favourable outcomes at a lower rate than other groups, this may be evidence of systemic bias. 2 While disparate impact has the advantage of aligning closely with legal standards and public perceptions, it also faces criticism, particularly in machine learning contexts. Enforcing strict parity in classification outcomes may lead to a reduction in accuracy and thus overall “utility” [ 24 ], especially when base rates differ across groups. This trade-off can result in decisions that, while statistically balanced, may harm individuals or reduce efficiency without delivering clear benefits [ 13 ]. Moreover, socio-technical critiques point out that measures such as disparate impact that focus on classification rates, just like error-focused measures, only consider model properties and may fail to result in just outcomes when actually deployed [ 28 , 40 ]. Nonetheless, disparate impact has a more straightforward connection to the real-world consequences of models than error rates. In this work, we assume that reducing disparate impact is often desirable as long as it does not imply a notable reduction in accuracy, broadly in line with calls for non-ideal approaches to fairness [ 20 ] and comparative (rather than transcendental) approaches to justice [ 42 ]. Indeed, this perspective reflects the original motivation behind the notion of disparate impact; in contrast, its criticism in algorithmic fairness often 2 These outcome disparities are sometimes quantified by means of the disparate impact ratio and evaluated using heuristics like the “80% rule"—which, however, is only a “crude test" [38]. 242 Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA concerns the full elimination of disparate impact, which reflects its common treatment as a mathematical formula that is exchangeable with other metrics. 2.2 Less discriminatory alternatives and model multiplicity In legal doctrine under U.S. anti-discrimination law, the concept of disparate impact is closely related to the idea of a Less Discriminatory Alternative: a policy or decision-making process that leads to disparate impact can be justifiable as long as there exists no alternative that meets the same legitimate objectives while producing less discriminatory outcomes. This principle plays a central role in Title VII of the Civil Rights Act and has informed a wide range of employment and civil rights cases [ 4 ]. In the context of machine learning, less discriminatory alternatives can be operationalized as a fairness-aware criterion for model selection: if two models perform similarly regarding predictive accuracy but one model has lower disparate impact than the other, the fairer model could be considered as the less discriminatory alternative and thus would be both ethically preferable and potentially legally required in regulated environments [4]. Although central to disparate impact, the concept of less discriminatory alternatives has only recently gained attention in the ML literature, particularly in the context of model multiplicity (MM) [ 6 , 23 ]. MM describes the phenomenon where multiple different models can achieve comparable accuracy on the same task. While the underlying idea—sometimes called Rashomon effect—has been known for decades [ 9 ], only in recent years has it become a focus of systematic investigation [ 31 ]. Despite promising recent work [41], this area of research is still in its early stages. An intriguing aspect of MM is that models with similar accuracy can exhibit very different behaviours regarding robustness, fairness, or interpretability [ 7 , 15 , 39 ]. By virtue of MM, it is often possible to select fairer models without losing accuracy, i.e. less discriminatory algorithms (LDAs) [ 5 ]. In such cases, the existence of accurate and fairer alternatives underscores the need to examine the modelling pipeline beyond aggregate performance metrics. From a fairness perspective, MM implies that selecting one model over another is not only a neutral performance-based decision, as it can have a differential real-world impact across demographic groups. In this paper, we investigate how Fairness through Unawareness (FtU) can, in some cases, provide a less discriminatory alternative without requiring explicit fairness optimisation, which has been investigated in other works [ 15 , 23 ]. We examine whether models that exclude protected attributes from their input can still maintain competitive predictive performance while reducing disparate impact, particularly in the context of logistic regression, a widely used model in real-world scenarios. 3 Calibrated aware models can exacerbate inequality “Predictive models trained with supervised learning methods are often good at calibration [...] But calibration also means that by default, we should expect our models to faithfully reflect disparities found in the input data." [3, p. 21] Reflected in this quote from the most prominent textbook on algorithmic fairness is the frequent assumption that disparities present in the data will be carried over into calibrated machine learning models by default. While this assumption may hold true for probabilistic models, it does not apply to classifiers derived from these models. In this section, we demonstrate that, perhaps counterintuitively, decisions based on calibrated probabilities can actually worsen the inequalities reflected in the base rates of labels. We also show that this effect is especially pronounced in policies relying on logistic regression models. We assume an input space 𝒳 and a, possibly empirical, joint probability distribution 𝑃 over random variables (𝑋,𝑌) which can take binary values in 𝒳× { 0 , 1 } . Take a predictor 𝑓 : 𝒳→ [ 0 , 1 ] and a corresponding classifier 𝐹:𝒳→ {0,1}, 𝑥 ↦→ 1[𝑓(𝑥) ≥ 0.5].(1) Assume two groups that form a partition {𝒜,ℬ} of 𝒳 . 3 The purpose of this section is to analyse how the disparate impact (DI) DI(𝐹):=E𝑃[𝐹(𝑋)|𝑋∈𝒜]−E𝑃[𝐹(𝑋)|𝑋∈ℬ](2) relates to the difference in base rates (DBR) DBR :=E𝑃[𝑌|𝑋∈𝒜]−E𝑃[𝑌|𝑋∈ℬ].(3) In this paper, we assume that ℬ is the disadvantaged group, i.e., 𝐷𝐵𝑅 ≥0. 3.1 General case The first observation is that for calibrated predictors, the label rates in the data are reflected in the means of the predictors. In the standard sense of Chouldechova [11] , a predictor 𝑓 is calibrated on both groups if ∀𝑣∈ [0,1],∗ ∈ {𝒜,ℬ}:E𝑃[𝑌|𝑓(𝑋)=𝑣, 𝑋 ∈ ∗]=𝑣. (4) Now, if 𝑓 is calibrated in this sense, then the average prediction of 𝑓 on either group trivially equals the average outcome for that group (simply by integrating over 𝑣): ∀∗ ∈ {𝒜,ℬ}:E𝑃[𝑓(𝑋) | 𝑋∈ ∗]=E𝑃[𝑌|𝑋∈ ∗].(5) While all predictors that are calibrated on a given dataset will have the same mean, equal to the base rate, these different predictors can lead to different classification rates. We illustrate this with a simple toy example. Example 1. Take 𝒳={𝑥1, 𝑥2, 𝑥3, 𝑥4} with 𝑃 defined via ∀𝑥∈ 𝒳 : 𝑃(𝑥)= 0 . 25 and 𝑃(𝑌= 1 |𝑋=𝑥)= 0 . 4 , 0 . 4 , 0 . 55 , 0 . 7for 𝑥=𝑥1, 𝑥2, 𝑥3, 𝑥4 , respectively. Then the predictors 𝑓1, 𝑓2, 𝑓3 as defined in Table 1a are all calibrated in the standard sense (without groups) while leading to very different classifications (Table 1b). We turn next to the quantities of interest: disparate impact and difference in base rates. First, we consider the most general case, with arbitrary 𝑓 , 𝛼 , and 𝑃 . It is important to note that disparate impact depends on the ratio of predictions above/below the decision threshold, whereas the base rate difference depends on the means of the predictions (assuming calibration). We can formalise this in 3 This is the case when the input features contain a protected attribute encoding group membership, i.e., for group-aware models. 243 EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Benedikt Höltgen and Nuria Oliver a𝑥1𝑥2𝑥3𝑥4 𝑝(𝑥)0.4 0.4 0.55 0.7 𝑓1(𝑥)0.45 0.45 0.45 0.7 𝑓2(𝑥)0.475 0.55 0.475 0.55 𝑓3(𝑥)0.5125 0.5125 0.5125 0.5125 b𝑥1𝑥2𝑥3𝑥4 𝐵(𝑥)0 0 1 1 𝐹1(𝑥)0 0 0 1 𝐹2(𝑥)0 1 0 1 𝐹3(𝑥)1 1 1 1 Table 1: Top (table a): Calibrated predictors for Example 1, where 𝑝(𝑥)=𝑃(𝑌= 1 |𝑋=𝑥) is the true probability. 𝑓1 is given by joining 𝑥3 with 𝑥1 and 𝑥2 ; 𝑓2 given by joining 𝑥1 with 𝑥3 and joining 𝑥2 with 𝑥4 ; 𝑓3 is given by the constant base rate predictor. All of these predictors are calibrated. Bottom (table b): Classifiers derived by the models in table a, where 𝐵is the Bayes optimal classifier. terms of the distributions of the predictions for each group, 𝑃𝒜 and 𝑃ℬ, where 𝑃∗(𝑣):=𝑃(𝑓(𝑋)=𝑣|𝑋∈ ∗) for 𝑣∈ [0,1],∗ ∈ {𝒜,ℬ}.(6) Then, for random variables 𝑉𝑎,𝑉𝑏∼𝑃𝒜, 𝑃ℬ describing the groupwise predictions, the disparate impact is given by DI(𝐹)=𝑃(𝐹(𝑋)=1|𝑋∈𝒜) − 𝑃(𝐹(𝑋)=1|𝑋∈ℬ) =𝑃𝒜({𝑉𝑎≥0.5}) − 𝑃ℬ({𝑉𝑏≥0.5}) (7) and the difference in base rates by DBR =E𝑃𝒜[𝑉𝑎] − E𝑃ℬ[𝑉𝑏].(8) The difference in base rates can intuitively be understood as the balance of how much and how far the probability mass is moved to the left vs the right of the decision boundary when moving from 𝑃ℬ to 𝑃𝒜 . 4 The disparate impact, on the other hand, only depends on the balance of how much probability mass passes over the decision threshold. The base rate difference and the disparate impact can be directly related under specific conditions, for example, when 𝑃𝒜 is higher on one side of the threshold and 𝑃ℬ is higher on the other (proofs can be found in Appendix B): Proposition 1 (Relation between Base Rates and Disparate Impact). Assume 𝑓 is calibrated as in (4), satisfying ∀𝑣∈ [ 0 , 0 . 5 ) : 𝑃ℬ(𝑣) ≥ 𝑃𝒜(𝑣) and ∀𝑣∈ [ 0 . 5 , 1 ] : 𝑃𝒜(𝑣) ≥ 𝑃ℬ(𝑣) . Then for 𝐹 defined as in (1), DI(𝐹) ≥ DBR. 3.2 Special case: Logistic Regression We now investigate the relationship between disparate impact and the difference in base rates in the case of logistic regression. Logistic regression is a widely used type of statistical method for modelling the probability of a binary outcome based on one or more predictor variables. Its popularity is due to its simplicity, interpretability, and 4 This means that if 𝑃𝒜 ‘dominates’ 𝑃ℬ in the sense of ∃𝑇 : [ 0 , 1 ]→[ 0 , 1 ] with 𝑇(𝑣) ≥ 𝑣∀𝑣∈ [ 0 , 1 ] s.t. 𝑇(𝑉𝑏) ∼ 𝑃𝒜 , then the base rate difference equals the earth mover (Wasserstein-1) distance. Figure 1: Plot of the sigmoid function used in logistic regression. Note that the slope is steepest around 𝑧= 0and satisfies 𝜎(0)=0.5. effectiveness in tackling classification problems. The model class consists of linear models with a sigmoid output 𝜎, 𝑓:R𝑛→ [0,1],(𝑥1, ..., 𝑥𝑛) ↦→ 𝜎 𝑐0+∑︁ 𝑖 𝑐𝑖𝑥𝑖!(9) with 𝒳⊂R𝑛and the sigmoid function defined by 𝜎:𝑧↦→ 1 1+𝑒−𝑧. Training a logistic regression model consists of finding the values of the model’s coefficients 𝑐0, ...,𝑐𝑛∈R that best fit the data by minimising a logistic loss function (equivalent to maximising the likelihood) using optimisation techniques like gradient descent. If a protected attribute is part of the input space, there will be a corresponding coefficient 𝑐𝑖∈R which determines how the protected attribute affects the (pre-sigmoid) logit function, ˆ 𝑓(𝑥) : = 𝑐0+Í𝑖𝑐𝑖𝑥𝑖 . Therefore, for binary protected attributes where 𝑥𝑖 can be 0 or 1, being part of the disadvantaged group ( 𝑥𝐺 = 1) implies adding a term 𝑐𝐺 to the logit without further interaction with the rest of the variables. We focus on the case where 𝑃(𝑌|𝑋,𝐺 =∗) differs significantly between groups, as this makes the existence of a fixed global coefficient 𝑐𝐺 particularly problematic. 5 A difference in base rates means that, if the model is calibrated for both groups (5), the coefficient of the variable that corresponds to the group attribute will penalise all members of the disadvantaged group because the coefficient for that group will reduce the logit by a fixed value 𝑐𝐺 , before applying the sigmoid, which can then contribute to an even larger disparate impact. To understand why, consider the shape of the sigmoid function displayed in Figure 1: The curve is steeper around the classification threshold at 𝜎(𝑧) =0 . 5than for other values of 𝑧 , as highlighted by the red lines. Therefore, small changes in 𝑧 near the classification threshold would lead to significant changes in the value of the sigmoid. Consider the following example with a logistic regression classifier: Two individuals have feature vectors 𝑥𝑏 and 𝑥𝑎 that are identical except for their group membership, such that person 𝑏 belongs to group ℬ and person 𝑎 belongs to group 𝒜 . Therefore, the only element that differs between the two individuals’ predictions is the coefficient corresponding to the variable that denotes which group they belong to. Changes in this coefficient will shift the logit 5 In the case where 𝑃(𝑌|𝑋,𝐺 =∗) is the same across groups but 𝑃(𝑋|𝐺=∗) is different, the choice of model class may not have as immediate an effect. However, this choice still affects the resulting predictive distribution, which has an effect on the disparate impact as shown above. 244 Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA value 𝑧 . Imagine the output of the logistic regression function (predicted probabilities) for each of them is 𝜎(𝑧𝑏) =0 . 95 and 𝜎(𝑧𝑎) =0 . 9. To obtain the logit values 𝑧𝑎 and 𝑧𝑏 , we need to apply the inverse sigmoid 𝑧=𝜎−1(𝑝)=𝑙𝑛(𝑝 1−𝑝) . In the example, 𝜎−1( 0 . 95 ) ≈ 2 . 94 and 𝜎−1( 0 . 90 ) ≈ 2 . 20, such that their difference is ≈ 0 . 75. This difference comes from the assumed difference in base rates between groups in this example. This suggests that group 𝒜 has a group offset that raises the logit by 0.75 when compared to group ℬ for the same input features. Now, let’s consider an individual in group ℬ that has a predicted probability right at the classification threshold at 𝜎(𝑧)= 0 . 5(therefore, logit 𝑧= 0); and another individual with exactly the same feature vector but belonging to group 𝒜 . The logit would increase by 0.75 because of the group effect, such that 𝑧𝑎=𝑧𝑏+ 0 . 75 = 0 + 0 . 75, 𝜎( 0 . 75 ) ≈ 0 . 68. This means that the same features lead to higher predictions for members of 𝒜 due to the group offset: for every individual in group 𝒜 with scores between 0 . 5and 0 . 68, there is a counterpart in ℬ who would fall below the threshold, even though their other features are identical. Thus, a global group coefficient that is meant to account for overall base rate differences between groups can lead to large local effects, especially in the vicinity of the decision threshold, due to the non-linear shape of the sigmoid function. In the example, even a moderate difference of 0.05 closer to the margins corresponds to a large difference of 0.18 near the threshold, which can lead to a large disparate impact. Figure 2: Predictions for aware and unaware logistic regression models on the ACS Income NY dataset. The aware model uses education and sex as input variables. The unaware model only considers education, measured in years. Colored vertical lines correspond to the 50% cut-off. Note how the aware logistic regression model requires much higher education values for women (yellow/red) than for men (blue) to be selected. This phenomenon is not observed in the unaware model. This effect can be further illustrated with an example based on real data, namely the US Census data from New York, obtained from the ACS Income dataset from the folktables package [ 18 ] (see Section 5.1). In this case, the target variable is whether a person has an annual income of at least 50k. For illustration purposes, we restrict the logistic regression model to include only two variables: the most predictive attribute (education, in number of years) and the protected binary attribute (sex). Two logistic regression models using default hyperparameters are trained: one aware model, including education and sex, and one unaware model that uses education as the only input variable. The results are depicted in Figure 2: The aware model “pushes" the predictions for men and women apart, meaning that men need an education score of 20 (equivalent to an associate’s degree), whereas women need an education score of 22 (equivalent to a Master’s degree) to obtain a positive classification. Income for men is over-predicted around the threshold (the blue line being above the grey dashed base rate line), whereas for women it is under-predicted (the yellow line being below the grey dashed base rate line). Conversely, in the case of the unaware model, which does not include sex as an input variable, the blue line on the left and the orange line on the right are necessarily the same. In this case, men are predicted correctly at the threshold while women are slightly over-predicted (the yellow line being above the grey dashed base rate line). Overall, this results in a similar accuracy for the unaware model (69.9%), compared to the aware model (70.0%), while significantly reducing the disparate impact (more details can be found in Appendix A.3).6 Before empirically investigating larger datasets, we take a step back and theoretically analyse the effect as an instance of model multiplicity—the phenomenon that multiple models can have similar accuracy while differing in other properties, such as fairness. 4 Less Discriminatory Algorithms through Unawareness In this section, we show that large differences between models with similar accuracy can be explained theoretically and that datapoints with predictions close to the decision threshold are particularly important for this. We provide the first tight upper bounds on model multiplicity and, based on that, derive bounds on the difference in disparate impact. We also provide a lower bound specifically for logistic regression classifiers. 4.1 General upper bounds on model multiplicity We begin by establishing general results on model multiplicity and then explore their implications for disparate impact and unaware models that do not use the protected attribute. Let 𝑃 denote the empirical distribution over 𝑋,𝑌 , describing the available data; based on this we define the conditional 𝑝(𝑥) : 𝑥↦→ 𝑃(𝑌= 1 |𝑋= 𝑥) and marginal measure 𝜇(𝑈) : 2 𝒳→ [ 0 , 1 ],𝑈 ↦→ ∫𝑈𝑃(𝑥)𝑑𝑥 . Furthermore, let 𝐵 : 𝒳→ { 0 , 1 } be a Bayes-optimal classifier (with 𝐵(𝑥)= 0for 𝑝(𝑥)< 0 . 5and 𝐵(𝑥)= 1for 𝑝(𝑥)> 0 . 5). Now in line 6 While the single feature case is an unrealistic illustration, the fact that the accuracy is not much lower than when using all features indicates that the feature drives most of the prediction. This already suggests that similar results occur in full datasets; we turn to this in Section 5. 245 EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Benedikt Höltgen and Nuria Oliver with the closest previous work [7], we define 𝐿as the 0-1-loss 𝐿(𝑦1,𝑦2):=(0if 𝑦1=𝑦2 1otherwise.(10) We define the accuracy of a model 𝐹as 𝐴𝑐𝑐(𝐹):=1−E𝑃[𝐿(𝐹(𝑋),𝑌)] (11) and the disagreement between two models 𝐹,𝐺 as 𝑑(𝐹,𝐺):=𝜇({𝑥∈𝑋:𝐹(𝑥)≠𝐺(𝑥)}).(12) For ℱ𝒳:=2𝒳the set of binary classifiers on 𝒳, let 𝑅𝜖(𝐹):={𝐺∈ℱ𝒳:𝐴𝑐𝑐(𝐺) ≥ 𝐴𝑐𝑐(𝐹) − 𝜖}(13) be the 𝜖-Rashomon set of 𝐹. For 𝜆∈ [0,0.5], let 𝑈(𝜆):={𝑥∈𝒳:|𝑝(𝑥) − 0.5|=𝜆}and 𝑚(𝜆):=𝜇(𝑈(𝜆)) (14) measure how much each distance from 0 . 5occurs, which is non-zero only on the set of occurring distances Λ : ={|𝑝(𝑥) − 0 . 5 | : 𝑥∈𝒳} . Then we define new quantities 𝑒(𝜆):=∑︁ 𝜆′<𝜆 2𝜆′·𝑚(𝜆′)and 𝜆𝜖:=arg max 𝜆∈Λ {𝜆:𝑒(𝜆) ≤ 𝜖},(15) which are illustrated in Figure 7 in Appendix B. Intuitively, 𝑒(𝜆) is the additional error if for all points with label ratios closer to 50% than 𝜆 , the suboptimal classification is taken, and 𝜆𝜖 is the largest 𝜆 such that this error does not exceed 𝜖 . With these definitions at hand, we can prove a tight upper bound on model multiplicity, which is proved through a lemma on randomised classifiers. The dependence of our tight bound on 𝜆𝜖 and 𝑒(𝜆𝜖) demonstrates how important input space regions with a near-50% label ratio are for model multiplicity. Proposition 2 (Upper Bound on Multiplicity). max 𝐹∈𝑅𝜖(𝐵)𝑑(𝐹, 𝐵) ≤ 𝜖−𝑒(𝜆𝜖) 2𝜆𝜖 +∑︁ 𝜆<𝜆𝜖 𝑚(𝜆).(16) This bound is the tightest possible bound with only access to the cumulative distribution of 𝑝(𝑥). Lemma 1. The bound (16) can be achieved in any setting for every 𝜖by a randomised classifier defined as 𝐹𝜖(𝑥)=        1−𝐵(𝑥)for |𝑝(𝑥) − 0.5|<𝜆𝜖 1−𝐵(𝑥)with probability 𝜖−𝑒(𝜆𝜖) 2𝜆𝜖·𝑚(𝜆𝜖)for |𝑝(𝑥) − 0.5|=𝜆𝜖 𝐵(𝑥)for |𝑝(𝑥) − 0.5|>𝜆𝜖. (17) This means the maximum is achieved if the predictions with 𝑝(𝑥) closest to 0 . 5are flipped until the accuracy decreases by 𝜖 . All proofs, as well as illustrations of these results and the quantity 𝑒(𝜆) can be found in Appendix B. We now relate this bound to a previous result: Black et al . [7] provide a formal upper bound on how much a model can disagree with another model when the difference in accuracy is controlled. We can also recover their result in the proof of their Theorem A.2, that 𝑑(𝐹, 𝐵) ≤ 𝐴𝑐𝑐(𝐵) − 𝐴𝑐𝑐(𝐹) 2𝑐(18) under the assumption7 |𝑝(𝑥) − 0.5| ≥ 𝑐>0∀𝑥∈𝒳.(19) Corollary 1. Under assumption (19), max 𝐹∈𝑅𝜖(𝐵)𝑑(𝐹, 𝐵) ≤ 𝜖 2𝑐.(20) While the above bounds involved an optimal classifier, we can also give bounds for two non-optimal models. Proposition 3 (Multiplicity Bound for Non-optimal Models). We can bound the disagreement between two classifiers 𝐹∈𝑅𝜖(𝐵) and 𝐺∈𝑅𝛿(𝐵)via 𝑑(𝐹,𝐺) ≤ max 𝐻∈𝑅𝜖+𝛿(𝐵)𝑑(𝐻, 𝐵).(21) That is, the maximum possible disagreement between 𝐹 and 𝐺 is no greater than the largest disagreement that any model 𝐻 within accuracy drop 𝜖+𝛿 could have with 𝐵 itself. We now connect this bound to disparate impact. Proposition 4 (Upper Bound on Disparate Impact). For two classifiers 𝐹 and 𝐺 , the difference in their disparate impact can be bounded based on their disagreement and the size of the smaller group: |DI(𝐹) − DI(𝐺)| ≤ 𝑑(𝐹,𝐺) min∗∈{𝒜,ℬ}𝑃(𝑋∈ ∗) .(22) That is, the potential difference in disparate impact between two models increases with their disagreement, and it is inversely proportional to the size of the smallest protected group. We can also specify a bound for the context of Fairness through Unawareness. The above bound is not tight when an unaware model is involved, unless for all Ð𝜆<𝜆𝜖𝑈(𝜆) , the two groups are separable in 𝑈(𝜆) by the protected attribute alone. If this is not the case, then the unaware model will give the same decision for some 𝑥1, 𝑥2∈Ð𝜆<𝜆𝜖𝑈(𝜆) from different groups; so flipping the decisions on both 𝑥1 and 𝑥2 , which is necessary for equality in (16), will not lead to a proportional rise in DI, i.e., there is no equality in (22). However, we can show that half the bound is achievable for some 𝜖: Proposition 5 (Achievable DI for Unaware Models). Fix 𝑃 , an unaware model 𝐹 , and some 𝜖> 0. Then there is a model 𝐺∈𝑅𝜖(𝐹)with |DI(𝐹) − DI(𝐺)| ≥ 1 2·max∗∈{𝒜,ℬ}𝑃(𝑋∈ ∗) max 𝐻∈𝑅𝜖+𝛿(𝐵)𝑑(𝐻, 𝐵). (23) 4.2 A lower bound for logistic regression We now turn to the specific case of LR models. The following result provides a lower bound on the reducible DI for a fixed change in accuracy. This bound depends on the coefficient of the protected attribute and the calibration plots for both groups, including bin sizes. Similar to the results presented above, it crucially depends on changes in classification decisions near the decision threshold, highlighting the importance of data points near the decision threshold. 7 Black et al. assume |𝑝(𝑥) − 0 . 5 |>𝑐 but it is not necessary to assume that the inequality is strict. 246 Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Proposition 6 (Lower Bound on DI for LR). Let 𝐹 : 𝒳→ { 0 , 1 } be an aware classifier based on an LR model 𝑓 with coefficient 𝑐𝐺< 0for the PA, i.e., the disadvantaged group gets a pre-sigmoid penalty of 𝑐𝐺. Further, let 𝑄(𝑐𝐺):={𝑥∈ℬ:𝑓(𝑥) ∈ [𝜎(𝑐𝐺),0.5]}(24) denote the set of points that are in Group 𝑏 and get predictions between 𝜎(𝑐𝐺) and 0 . 5by 𝑓 . Then the corresponding unaware classifier 𝐹′ based on setting 𝑐𝑃𝐴 ←0in 𝐹has accuracy bounded by |𝐴𝑐𝑐(𝐹) − 𝐴𝑐𝑐(𝐹′)| ≤ 2𝜎(−𝑐𝐺) · 𝑃(𝑋∈𝑄(𝑐𝐺)) (25) and disparate impact given by DI(𝐹′)=DI(𝐹) − 𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ)(26) if group 𝑏 is still disadvantaged in 𝐹′ (i.e. DI(𝐹) ≥ 𝑃(𝑋∈ 𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ)), and otherwise |DI(𝐹) − DI(𝐹′)| ≤ 𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ).(27) The focus on one specific unaware model provides a loose but already interesting lower bound on the possible reduction of DI: As a quantitative example, consider the ACS Income dataset discussed in the following section. In this setting, a logistic regression model assigns a coefficient of − 0 . 86 for the protected attribute sex. We observe that the proportion of women is 𝑃(𝑋∈ℬ)= 0 . 48 and 𝑃(𝑋∈𝑄(𝑐𝐺)) = 6 . 9% is the set of women whose predicted probabilities fall within the range [ 0 . 5 , 0 . 70 ) , corresponding to the sigmoid of 0 . 86, i.e., 𝜎( 0 . 86 )= 0 . 70. If we were to flip decisions in this region, the accuracy would change by at most 0 . 7percentage points either down or up, while the disparate impact would decrease by at least 14.5percentage points.8 Note that this result only represents one possible model within the ( 2 𝜎(−𝑐𝐺) · 𝑃(𝑋∈𝑄(𝑐𝐺))) -Rashomon set, and therefore serves as a lower bound on the achievable 𝐷𝐼 reduction. It is unlikely to be optimal since it optimises the other coefficients ‘under the assumption’ that 𝑐𝐺 will be used; it also only modifies the predictions for women, leaving the men’s unchanged. In summary, we have provided general upper bounds on disparate impact reduction within some 𝜖 -Rashomon set that are achievable for unconstrained model classes and we have a naïve lower bound for logistic regression. In the next section, we empirically investigate how unaware and aware models compare on real datasets. 5 Empirical findings After the theoretical results above, we now empirically analyse the effect of Fairness through Unawareness (FtU) on Disparate Impact (DI) and Accuracy. We consider three datasets, with different model classes and protected attributes. All code, also for Section 3.2 above, is available at https://github.com/ben-hoeltgen/ftu-mm/. 5.1 Setup 1. Models: We analyse the behaviour of two model classes, logistic regression (LR) and gradient-boosted decision trees (GBM), using the popular scikit-learn package (more details in Appendix A.2). 8 With a base accuracy of 77,4% and a DI of 0.052, this translates to a relative accuracy change of 0.009 vs a relative reduction in DI of 0.64—see Figure 3. 2. Data: We perform experiments on three datasets: The ACS Income and ACS Employment datasets from the folktables package [ 18 ] from the US Census and the active labour market policy (ALMP) evaluation dataset from Switzerland [ 29 ]. Table 3 in Appendix A.1 contains a summary of the data. In ACS Income and ACS Employment, the target variables are whether a person’s income is above 50k and whether they are employed, respectively. We consider two binary protected attributes: sex (male/female) and race (white/black). Note that the US Census asks for sex rather than gender, which is reflected in the derived dataset. Race is self-indicated in the census, and we restrict to the white/black categories. 9 In the ALMP dataset, our target variable is whether a person will be employed for at least 6 months within the next two years. We consider two binary protected attributes: gender (male/female) and citizenship (Swiss/foreign). In line with previous literature [ 46 ], we exclude caseworker information and restrict data to Germanspeaking cantons. 5.2 Results Figure 3: Relative reduction in Accuracy vs in Disparate Impact (DI) across three datasets, with two model classes (LR: logistic regression and GBM: gradient boosted trees) and two protected attributes (PAs) each. Each dot corresponds to the difference in Accuracy and DI between the aware and the corresponding unaware model trained on the same random train/test split. Note how the unaware model achieves a reduction in accuracy which is always below 1% and sometimes it even increases, while yielding a reduction in DI that is much more substantial (between 15 −80%). 9 We want to emphasise that race categories are an over-simplification of the complex notion of racialization. Their use should always be contextualised and avoided where possible [19]. 247 EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Benedikt Höltgen and Nuria Oliver The main results, shown in Figure 3, indicate that Accuracy remains similar between aware and unaware models (i.e., models that exclude the protected attribute), while DI is often significantly reduced in the unaware models, yielding fairer models with similar levels of accuracy. 10 The results on the ALMP dataset show general multiplicity, whereas the results on the ACS datasets exhibit less multiplicity. This is not surprising, given that the number of data points is much larger in the ACS datasets than in the ALMP dataset (Table 3). The fact that the observed results hold on such large datasets suggests that the effect does not simply disappear with more data. When considering race as the protected attribute in the ACS datasets, the reduction in disparate impact is generally smaller than with sex, but reaches around 20% across all settings at virtually no loss in accuracy. For settings where sex is the protected attribute, there is an interesting difference in the results on the ACS Income vs the ACS Employment datasets. On ACS Income, the largest reduction in DI ( ≈ 80%) is obtained with the unaware LR model (consistent with, but slightly exceeding, the lower bound of Section 3.2, see footnote 8). In contrast, on ACS Employment, the largest reduction of DI (also ≈ 80%) corresponds to the unaware GBM. This result can be explained by two effects that are observable in Figure 4. First, aware LR can lead to particularly high disparate impact, as discussed in Section 3.2. This effect is particularly pronounced when sex is the protected attribute. For instance, in the case of the ACS Income dataset, the LR model leads to a disparate impact that far exceeds the base rate difference in the data (indicated by the red line). Second, in the ACS Employment dataset, the LR model with sex as the protected attribute actually results in higher positive classification rates for women than for men, suggesting that women may have stronger predictive features for employment than men. As a result, the change in disparity between men and women is more noticeable with LR than with GBM (see Figure 4), even though the absolute DI values change less (see Figure 3). This difference between ACS Employment and ACS Income can be attributed to the lower base rate differences in the former (see Table 3). 6 Discussion: Real-world use case We have shown that including protected attributes like sex or gender in models can lead to substantially less equitable classifications, often without meaningful gains in accuracy—and sometimes without improving accuracy at all. However, what ultimately matters is not just how models classify individuals, but the real-world impact that these systems have when used to guide decisions. In other words, what matters are the policies that these models inform [ 28 ]. As others have pointed out, “[i]t is policies and their effects that are just or unjust; ‘fair’ predictors can both support unjust policies and undermine just policy" [ 46 , p. 1984]. Machine learning models are increasingly used in resource allocation, where limited goods or opportunities—like counselling or training programs—are distributed based on risk predictions, often through threshold-based 10 This result is consistent with previous analysis on the German unemployment data, where it was observed that “inor excluding protected attributes has little effect on overall performance” [27]. However, this observation is not further discussed there. Figure 4: Disparate impact for the aware and unaware models in different settings. The red lines indicate the difference in base rates in the data. Note how the unaware models achieve lower disparate impact than the aware models. policies.11 An important question is whether predictions are even helpful in such contexts [ 35 , 36 , 44 ]. Furthermore, there is the concern that predicting risk irrespective of treatment may not suffice since this risk does not necessarily correlate with treatment effectiveness [ 43 , 46 ]. With these caveats, we now turn to examine the impact of Fairness through Unawareness in a real-world application that has attracted considerable attention in Europe in recent years: the statistical profiling of job seekers, which is increasingly used in labour market programs across the world to support decisions on how to allocate resources such as job training opportunities [ 17 ]. These systems aim to predict an individual’s risk of long-term unemployment such that costly services can be targeted more efficiently, ideally to those for whom the intervention will have the greatest positive impact. Different models use different predictors. For example, gender has been explicitly included in profiling models used in Australia [30] and Austria [25]. A recent example is the Austrian AMS algorithm, which uses an undisclosed stratification procedure [ 22 ] that is approximated by a published logistic regression model [ 25 ], and includes gender as a feature. The AMS algorithm has sparked significant legal and political controversy, leading to multiple revisions and delays in its deployment. As a result, it has not yet been implemented in practice. Critics argue that the model creates a false impression of objectivity and has been specifically accused of reinforcing gender discrimination [ 1 ]. The Austrian system stands out by targeting individuals of medium risk: the model categorizes job seekers into three groups—low (A), medium (B) and high (C) risk of long-term unemployment—but only those in the medium-risk group (B) are offered access to the costly job training programs. There are various issues with this approach, most notably its reliance on observational data to make causal claims [ 43 , 46 ], such as the assumption that individuals at medium risk benefit the most from the intervention. In line with the broader theme of this paper, we highlight another common but flawed assumption: that using more attributes necessarily leads to more objective (and hence fairer) decisions. One major criticism of the AMS algorithm has been its discriminatory impact, 11 While our analyses have focused on the 50% threshold, we show below that the underlying insights apply more broadly. 248 Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity EAAMO ’25, November 05–07, 2025, Pittsburgh, PA, USA Proposition 5 (Achievable DI for Unaware Models). Fix 𝑃 , an unaware model 𝐹 , and some 𝜖> 0. Then there is a model 𝐺∈𝑅𝜖(𝐹)with |DI(𝐹) − DI(𝐺)| ≥ 1 2·max∗∈{𝒜,ℬ}𝑃(𝑋∈ ∗) max 𝐻∈𝑅𝜖+𝛿(𝐵)𝑑(𝐻, 𝐵). (80) Proof. Fix an 𝐻 that achieves the maximum and let 𝑈 : ={𝑥∈𝒳 : 𝐻(𝑥)≠ 𝐵(𝑥)}. Then define two classifiers 𝐺>and 𝐺<via 𝐺>(𝑥):=        1for 𝑥∈𝑈∩𝒜and 𝐹(𝑥)=0 0for 𝑥∈𝑈∩ℬand 𝐹(𝑥)=1 𝐹(𝑥)otherwise, (81) and 𝐺<(𝑥):=        1for 𝑥∈𝑈∩ℬand 𝐹(𝑥)=0 0for 𝑥∈𝑈∩𝒜and 𝐹(𝑥)=1 𝐹(𝑥)otherwise. (82) Then {𝑥∈𝒳 : 𝐻(𝑥)≠𝐵(𝑥)} =𝑈={𝑥∈𝒳 : 𝐹(𝑥)≠ 𝐺<(𝑥)} ∪ {𝑥∈𝒳:𝐹(𝑥)≠𝐺>(𝑥)} and hence 𝑑(𝐹,𝐺#) ≥ 1 2𝑑(𝐻, 𝐵)(83) for one #∈ {<,>}. Furthermore, we get |DI(𝐹) − DI(𝐺#)| ≥ 𝑑(𝐹,𝐺#) max∗∈{𝒜,ℬ}𝑃(𝑋∈ ∗) (84) by the same derivation as for (72) but with equality in (75) due to the construction of 𝐺# and with ‘ ≥ ’ in (76) by switching min to max.□ Proposition 6 (Lower Bound on DI for LR). Let 𝐹 : 𝒳→ { 0 , 1 } be an PA-aware classifier based on a LR model 𝑓 with coefficient 𝑐𝐺< 0for the PA, i.e. the disadvantaged group gets a pre-sigmoid penalty of 𝑐𝐺. Further, let 𝑄(𝑐𝐺):={𝑥∈ℬ:𝑓(𝑥) ∈ [𝜎(𝑐𝐺),0.5]}(85) denote the set of points that get predictions between 𝜎(𝑐𝐺) and 0 . 5 by 𝑓 and are in Group 𝑏 . Then the corresponding unaware classifier 𝐹′based on setting 𝑐𝑃𝐴 ←0in 𝐹has accuracy bounded by |𝐴𝑐𝑐(𝐹) − 𝐴𝑐𝑐(𝐹′)| ≤ 2𝜎(−𝑐𝐺) · 𝑃(𝑋∈𝑄(𝑐𝐺)) (86) and disparate impact given by DI(𝐹′)=DI(𝐹) − 𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ)(87) if group 𝑏 is still disadvantaged in 𝐹′ (i.e. DI(𝐹) ≥ 𝑃(𝑋∈ 𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ)), and otherwise |DI(𝐹) − DI(𝐹′)| ≤ 𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ).(88) Proof. 𝐹′ and 𝐹 only differ on the datapoints that are classified as 0by 𝐹 but as 1by 𝐹′ . These are the datapoints that have prediction < 0 . 5 by 𝑓but ≥0.5by 𝑓′, i.e. {𝑥∈𝒳:𝜎−1(𝑓(𝑥)) <0∧𝜎−1(𝑓′(𝑥)) ≥ 0} ={𝑥∈ℬ:−𝑐𝐺≤𝜎−1(𝑓(𝑥)) <0}.(89) Now this set of points is 𝑄(𝑐𝐺) , and the maximum change in accuracy occurs if one model is right on all points, whereas the other model is wrong on all points, and all predictions are as extreme as possible. Then, assuming that one of the models is calibrated on that set, the label ratio is bounded from both sides by 𝜎(−𝑐𝐺) ≤ 𝑃(𝑌|𝑋∈𝑄(𝑐𝐺)) ≤ 𝜎(𝑐𝐺) . This means the change in accuracy is bounded by 2·𝜎(𝑐𝐺) · 𝑃(𝑋∈𝑄(𝑐𝐺)), giving us (86. Recall that disparate impact is defined as DI(𝐹)=|𝑃(𝐹(𝑋)=1|𝑋∈𝒜) − 𝑃(𝐹(𝑋)=1|𝑋∈ℬ)|.(90) Now only predictions on group 𝑏are changed, such that 𝑃(𝐹(𝑋)=1|𝑋∈𝒜)=𝑃(𝐹′(𝑋)=1|𝑋∈𝒜).(91) On the other hand, we have 𝑃(𝐹′(𝑋)=1|𝑋∈ℬ)=𝑃({𝐹(𝑋)=1∨−𝑐𝐺≤𝜎−1(𝑓(𝑥)) <0|𝑋∈ℬ}), (92) and thus 𝑃(𝐹′(𝑋)=1|𝑋∈ℬ)=𝑃(𝐹(𝑋)=1|𝑋∈ℬ)+𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ). (93) Putting things together, we get 𝑃(𝐹′(𝑋)=1|𝑋∈𝒜) − 𝑃(𝐹′(𝑋)=1|𝑋∈ℬ)(94) =|𝑃(𝐹(𝑋)=1|𝑋∈𝒜) − 𝑃(𝐹(𝑋)=1|𝑋∈ℬ)| −𝑃(𝑋∈𝑄(𝑐𝐺))/𝑃(𝑋∈ℬ)(95) if the terms inside the vertical bars remain positive. Otherwise, we still get (88). □ 255