Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering - Supplementary Information Package
Abstract
How software developers interact with Artificial Intelligence (AI)-powered tools, including Large Language Models (LLMs), plays a vital role in how these AI-powered tools in turn impact them. While overreliance on AI may lead to long-term negative consequences (e.g., atrophy of critical thinking skills); underreliance might deprive software developers of potential gains in productivity and quality. Based on twenty-two interviews with software practitioners on using LLMs for software development, we propose a preliminary reliance-control framework where the level of control can be used as a way to identify AI overreliance and underreliance. We also use it to recommend future research to further explore the different control levels supported by the current and emergent LLM-driven tools. Our paper contributes to the emerging discourse on AI overreliance and provides an understanding of the appropriate degree of reliance as essential to developers making the most of these powerful technologies. Our findings can support SE educators in mitigating overreliance towards AI tools by their students.
Full text
Page 1 of 3: Introduction & Consent form CONSENT FORM Adopting LLM-based software development tools: a software practitioners perspective Introduction This study is conducted to gain an in-depth understanding ofcurrent industrial perspective on software practitioners adopting Large Language Model-based tools on software development tasks. The research is approved by the Human Ethics Committee of Monash University, Australia for five years on <date>. Reference Number: 44875. Confidentiality This survey is anonymous. Any research publication published as a result of the study will not identify you, your company or any non-participant in your company. Storage of data Storage of data Data in a Google Drive with restricted access (only investigators have access) for a period of 5 years and destroyed thereafter. Further information about this research is available atExplanatory Statement. Complaints 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 1/11 OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED
Should you have any concerns or complaints about the conduct of the project, you are welcome to contact the Executive Officer, Monash University Human Research Ethics Committee (MUHREC): Executive Officer Monash University Human Research Ethics Committee (MUHREC) Room 111, Chancellery Building D, 26 Sports Walk, Clayton Campus Research Office Monash University VIC 3800 Tel:+61 3 9905 2052Email: [email protected] Fax:+61 3 9905 3831 INVESTIGATORS Prof. Rashina Hoda (Chief Investigator), Prof. John Grundy, Assoc Prof. Christoph Treude, and Mr. Samuel Ferino. CONTACT INFORMATION If you have any questions regarding this interview and the project, feel free to contact Mr. Samuel Ferino, Prof. Rashina Hoda, or Prof. John Grundy at {samuellucas.demouraferino, rashina.hoda, john.grundy}@monash.edu. CONSENT I confirm that I have understood the nature of the research, and I have had the opportunity to ask questions and have had them answered to my satisfaction. I agree to take part in this study all information collected from me will be kept confidential to the research team and I will not be identified in publications arising from the study. 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 2/11 OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED OMITTED
OPTIONAL NOTE We assure details of the participants, and all other confidential information shared will be kept confidential. The names and details of the participants will not be specified in any of the publications or reports. Full Name (First name and Last name): I understand that I am free to withdraw my participation at any time, and to withdraw any data traceable to me up to 30 days from the date of the participation without giving a reason I understand some personal information (e.g., age range, gender) may be collected for anonymous reporting I understand that I will be audio-recorded during the interview I understand that interview recordings will be uploaded to a secure commercial server (i.e., Otter.ai or Nvivo) for transcription I give consent to share the collected from me with other researchers on this project who are included in the ethics approval I wish to receive the summary of findings of the study I wish to have my interview transcript returned to me and have the opportunity to edit the transcript for 30 days from the receipt of the transcript 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 3/11 OMITTED
Email Address: Signature: Page 2 of 3: Demographics Demographics This section is intended to gather basic information of the participants. NOTE We assure details of the participants, and all other confidential information shared will SIGN HERE × clear 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 4/11 OMITTED
be kept confidential. The names and details of the participants will not be specified in any of the publications or report. Please, provide your Linkedin account or any other professional social account that you use. How old are you? To which gender identify do you most identify? In which country do you currently reside? Male Female Non-binary / gender diverse My gender identity isn't listed. I identify as: Prefer not to say 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 5/11 OMITTED
What is the highest level of education you have completed? What is your current professional role or job title? High School graduate, diploma or the equivalent and some college credit, no degree Trade/technical/vocational training Associate or Professional degree Bachelor's degree Master's degree Doctorate degree Other Prefer not to say Software engineer Frontend developer Software developer Backend developer Web developer Mobile developer DevOps engineer Other Cloud engineer Prefer not to say 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 6/11 OMITTED
How do you describe your level of experience as a software developer? In what area/domain do you primarily work How many years of professional experience in software development do you have? Novice Highly experienced Advance beginner Other Intermediate Prefer not say Experienced Information Technology Telecommunications Finance Gaming Manufacturing Other Government Prefer not say Healthcare Less than 1 year 1 to 2 years 3 to 5 years 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 7/11 OMITTED
Page 3 of 3: LLM-based tools experience Large Language Model-based tools experience This section is intended to gather information regarding your experience using LLMbased tools. How long have you been using Large Language Model (LLM)- based tools in general? How long have you been using Large Language Model (LLM)- based tools for software development-based tasks? 6 to 10 years More than 10 years Prefer not to say 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 8/11 OMITTED
Which Large Language Model-based tools have you used (if any) at work?e.g. OpenAI ChatGPT; GitHub Copilot What development tasks do you usually seek help from LLM-based tools? What is your approximate number of prompts per day? Considering your experience using LLM-based tools, rate how much you agree with the sentences below: (please select answers to all items): Requirement-related tasks (e.g., requirement identification) Code-related tasks (e.g., coding, debugging) Test-related tasks (e.g., test case development) User-related tasks (e.g., user documentation) Other Prefer not say 02/10/2024, 22:06 Qualtrics Survey Software https://monash.pdx1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview?ContextSurveyID=SV_cNQcZ2GyUYRCy9… 9/11 OMITTED