Explainability Requirement Practices and Challenges from a Brazilian Industrial Research and Innovation Company

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 Explainability Requirement Practices and Challenges from a Brazilian Industrial Research and Innovation Company - APPENDIX Lívia Mancine Instituto Federal Goiano Universidade Federal de Goiás Ceres - GO, Brazil [email protected] Renata Dutra Braga Universidade Federal de Goiás Goiânia - GO, Brazil [email protected] Davi Viana Universidade Federal do Maranhão São Luís - MA, Brazil [email protected] Marcos Kalinowski Pontifícia Universidade Católica do Rio de Janeiro Rio de Janeiro - RJ, Brazil [email protected]rio.br Renato F. Bulcão-Neto Universidade Federal de Goiás Goiânia - GO, Brazil [email protected] Keywords Requirements Engineering, Explainability, Machine Learning, Industry, Experimentation, Interview, Grounded Theory ACM Reference Format: Lívia Mancine, Renata Dutra Braga, Davi Viana, Marcos Kalinowski, and Renato F. Bulcão-Neto. 2025. Explainability Requirement Practices and Challenges from a Brazilian Industrial Research and Innovation Company - APPENDIX. In .ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/ nnnnnnn.nnnnnnn 1 Background The general objective of this work is to investigate the perception of explainability in the context of Requirements Engineering (RE) for Machine Learning (ML)-based systems, in order to understand the state of practice and identify challenges, strategies, and opportunities for improvement. Explainability has been recognized as a cross-cutting non-functional requirement (NFR) closely associated with trust and transparency, gaining prominence as a critical quality attribute for AI-based systems. Its purpose is to make Machine Learning (ML) decisions more transparent and understandable. The literature indicates that, in critical systems, explainability is essential throughout the entire ML system lifecycle, especially when models are deployed in real-world contexts. Systems that incorporate explainability as a requirement enable users to better understand decision-making processes, promoting trust and transparency. Explainability is directly related to Requirements Engineering (RE), which systematizes the understanding of system needs and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA ©2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn contributes to other key quality attributes such as transparency, interpretability, and comprehensibility. As a NFR, explainability impacts the project scope and should be considered from the early requirements elicitation stages. Incorporating explainability early in the RE process helps ensure that the resulting system aligns with the principles of ethical and trustworthy AI. Moreover, explainability is associated with different stakeholder profiles, since each group requires a different level of explainability. End users, developers, managers, and regulatory authorities have distinct explanatory needs, ranging from general understanding of system behavior to technical details about model functioning and decision criteria. This diversity highlights the importance of defining explainability requirements tailored to the intended audience, balancing clarity, depth, and usefulness of the explanations provided. Beyond technical challenges, the rapid growth of AI has driven global regulatory initiatives, such as the EU AI Act 1 , 2 and UNESCO ethical guidelines 3 , Brazil’s Bill 2338/2023, and the Plano Brasileiro de Inteligência Artificial (PBIA). The ISO/IEC 42001 standard complements these efforts by providing a management framework for responsible and auditable AI. These initiatives emphasize transparency and understandable explanations, particularly in high-risk systems, reinforcing the role of Requirements Engineering in operationalizing explainability as a critical requirement for compliance, ethics, and stakeholder trust. 2 Method 2.1 Research Ethics Committee Information Responsible Committee: Research Ethics Committee – Federal University of Goiás (UFG) Email: [email protected] Public Title: Exploring Explainability in Requirements Engineering for Machine CAAE: 85638924.3.0000.5083 Researcher: Lívia Mancine Coelho de Campos 1https://artificialintelligenceact.eu/the-act/ 2https://www.oecd.org/en/topics/ai-principles.html 3https://unesdoc.unesco.org/ark:/48223/pf0000381137 1 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 Conference’17, July 2017, Washington, DC, USA Mancine et al. 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 E-mail: [email protected] Advisor: Renato de Freitas Bulcão Neto E-mail: [email protected] Each interviewee will receive a copy of the interview to follow the multiple-choice questions and facilitate understanding. For remote participants, the interview guide can also be followed via screen sharing. The purpose of the interview is to investigate how explainability is perceived and applied in RE for ML-based systems. We aim to understand the practices used, the challenges faced, the quality attributes achieved through explainability, and the trade-offs involved in integrating explainability. 2.2 Criteria for explainability Criterion for considering the requirement of explainability in those analyzed for the interview. Table 1: Criteria for Considering Explainability as a Requirement ID Criteria Context 1 Critical nature of the system Systems that make critical decisions for individuals, such as medical diagnosis, credit approval, social benefits authorization, or autonomous vehicles (without human intervention). 2 Business opportunity associated — not necessarily critical Explainability is a need or expectation of a specific stakeholder. 3 Data bias potential The data may contain biased patterns, replicating recurring unethical behaviors. 4 Need for compliance with legal requirements related to privacy The project must comply with legal and regulatory requirements that demand explanations for automated decisions, such as Brazil’s LGPD — Art. 20. 5 Requirement for expert validation Models that require prior validation to perform specific actions (e.g., medical or legal decision support). 6 Auditability requirement — not necessarily critical Models that need to be auditable to identify and justify their decisions, such as in internal investigations or bias analysis. 2.3 Method – Goal-Question-Metric (GQM) To better understand the object of this study, we applied the GQM method. This method allows the definition of quantifiable objectives by structuring questions that guide data collection and analysis. Thus, GQM helps link research goals to specific metrics, enabling a more systematic and evidence-based evaluation. Object of study [What will be analyzed?]: the practice of Requirements Engineering concerning explainability requirements in Machine Learning projects. Purpose [Why will it be analyzed?]: to characterize the state of practice of the explainability requirement in ML projects within professional environments. Quality focus [Which property of the object will be analyzed?]: characteristics related to processes, methodologies, and artifacts associated with the RE phase regarding explainability. Point of view [Who will use the collected data?]: professionals directly involved in the development of ML projects. Context [Where will the analysis take place?]: researchers and ML professionals, such as those working at the Centro de Excelência em Inteligência Artificial (CEIA) of the Universidade Federal de Goiás (UFG), whose projects involve applied research in AI, ML, Data Science, and Computer Vision, among others. 2.3.1 Evaluation Plan Development. This semi-structured interview is based on the hypothetico-deductive process, which consists of formulating a set of hypotheses and testing them through the collection and analysis of empirical data, allowing their validation or refutation. The interview will be conducted in person or via a web conferencing platform with recording for later transcription, lasting approximately one hour. Moreover, the interview covers three specific dimensions defined for this study: • Temporal dimension: refers to synchronous contact, in which interaction occurs in real time, either in person or remotely. • Spatial dimension: refers to the environment where the interview takes place, which can be face-to-face or remote, using synchronous technologies such as videoconferencing platforms. • Structural dimension: refers to a semi-structured interview guided by a predefined script but allowing a spontaneous conversational flow. The interviewees are researchers and professionals in the ML field, such as those working at CEIA/UFG, whose projects involve applied research in AI, ML, Data Science, and Computer Vision, among others. The nature of this research is exploratory, aiming to understand participants’ experiences and viewpoints to gain deep insights into the investigated topic. 2.3.2 Hypotheses. The hypotheses guide data collection and result analysis, helping confirm or refute the proposed assumptions. For this study, we formulated the following hypotheses to be tested and verified through a semi-structured interview addressing explainability as a non-functional requirement (NFR) in ML projects: • H01 – RE elicitation practices are not used to implement explainability in ML projects. • H02 – RE analysis practices are not used to implement explainability in ML projects. • H03 – RE specification practices are not used to implement explainability in ML projects. • H04 – RE validation and verification practices are not used to implement explainability in ML projects. • H05 – RE management practices are not used to implement explainability in ML projects. • H06 – Professionals consider explainability important but not a priority due to time, cost, and/or the lack of a systematic process to deliver explainable outcomes. • H07 – Professionals perceive conflicts between explainability and other quality attributes, such as performance and security. 2.4 Research Questions (RQs) • RQ1) What RE practices do professionals use to incorporate explainability in their ML projects? • RQ2) What challenges do practitioners face when defining and managing explainability requirements? • RQ3) Which quality attributes can be achieved through explainability? 2 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 Explainability Requirement Practices and Challenges from a Brazilian Industrial Research and Innovation Company - APPENDIXConference’17, July 2017, Washington, DC, USA 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 • RQ4) What trade-offs between explainability and other quality attributes need to be considered when integrating explainability into ML projects? 2.5 Metrics To answer RQ1, which addresses RE practices and explainability, the analysis will cover elicitation, analysis, specification, and validation concerning the following dimensions: techniques, requirement characteristics and structure. Regarding tool support, the use of specific explainability tools will be evaluated. RE management practices will be analyzed based on the following dimensions: change tracking and management practices. To address RQ2 and RQ3, which explore the challenges professionals face with explainability and the quality attributes achieved through it, the analysis will focus on the interview section titled “Challenges and Needs Related to Explainability.” Regarding RQ4, the semi-structured interview script includes a question in the “Explainability and Requirements Engineering” section (under Analysis) addressing this aspect: How do you analyze explainability to identify conflicts, dependencies, or implications concerning other requirements, such as transparency, traceability, reliability, and ethics (trade-offs)? 3