Supplementary materials for the paper "Evolving with AI: A Longitudinal Analysis of Developer Logs"
Abstract
This repository contains the supplementary materials for the paper "Evolving with AI: A Longitudinal Analysis of Developer Logs". It contains the following: survey_questionnaire.pdf — the full list of questions that we asked in our survey. survey_results.csv — the responses to the main Likert questions about productivity, code quality, code editing, code reuse, and context switching, provided by the 62 participants who completed the survey, as well as how long they have been using AI tools for coding. interview_script.pdf — the script that we used for semi-structured interviews. statistical_tests.pdf — the results of mixed-effects modelling on the log data for each of the five aspects, together with the resulting trends and P-values.
Full text
Interview script: [greetings] [check if the connection is ok] My name is _____, and I am a researcher here at [anonymized] First of all, I would like to thank you for your willingness to participate in our interview. Through this study, we are trying to deeper understand your experience with AI in coding. This interview will last approximately 30 minutes. Have you ever participated in such interviews before? So then you know that I will be asking a set of prepared questions, but our conversation could go naturally where it goes. There are no right or wrong answers in such sessions. We are really interested in your experience and thoughts. Also, I would ask your permission to record this session so we can pay equal attention to everything you say. We keep all recordings confidential, make anonymized summaries out of them, and delete everything once the research is over so that no personal data will be used. Is that OK with you? [if yes] Now, let me start the recording. When it starts, I will ask for your permission to record this session. [start recording] I would like to have your permission to record this session. [get the permission] Great! All technical nuances we dealt with. Do you have any questions at the moment? [discuss questions] OK, then, let's proceed to the main part of our talk. Today, we're exploring your experiences and thoughts on AI in Development Environments. So, let's agree on a definition of AI here. How would you define AI for software engineering? What is it for you in essence? To warm up, how would you briefly summarize your opinion on AI products for software engineering, from when you first learned about them to now? What changes did you notice in your workflow after the introduction of AI into your work life? In the survey, you mentioned *CITATION TOPIC with follow-up question*.
Thank you very much for all your answers. Do you have anything to add to our conversation? Before we end, is there anything you would like to ask me about? [answer questions from respondent] If I noted correctly, you chose Amazon *Country* certificate/[anonymized]/Not to get any prize. Is it right? Perfect. During this week, we’ll email you the link to your gift. However, delays occur from time to time, so please, if there won't be a gift in two weeks after today, feel free to contact me, and I will do my best to help. [say goodbye] [stop the recording]