A conversation with ChatGPT about digital leadership and technology integration: Comparative analysis based on human-AI collaboration

Karakose, Turgut et al. Article A conversation with ChatGPT about digital leadership and technology integration: Comparative analysis based on human-AI collaboration Administrative Sciences Provided in Cooperation with: MDPI – Multidisciplinary Digital Publishing Institute, Basel Suggested Citation: Karakose, Turgut et al. (2023) : A conversation with ChatGPT about digital leadership and technology integration: Comparative analysis based on human-AI collaboration, Administrative Sciences, ISSN 2076-3387, MDPI, Basel, Vol. 13, Iss. 7, pp. 1-19, https://doi.org/10.3390/admsci13070157 This Version is available at: https://hdl.handle.net/10419/275622 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Citation: Karakose, Turgut, Murat Demirkol, Ramazan Yirci, Hakan Polat, Tuncay Yavuz Ozdemir, and Tijen Tülüba¸s. 2023. A Conversation with ChatGPT about Digital Leadership and Technology Integration: Comparative Analysis Based on Human–AI Collaboration. Administrative Sciences 13: 157. https://doi.org/10.3390/ admsci13070157 Received: 9 May 2023 Revised: 7 June 2023 Accepted: 19 June 2023 Published: 24 June 2023 Copyright: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). administrative sciences Article A Conversation with ChatGPT about Digital Leadership and Technology Integration: Comparative Analysis Based on Human–AI Collaboration Turgut Karakose 1, Murat Demirkol 2, Ramazan Yirci 3,* , Hakan Polat 2, Tuncay Yavuz Ozdemir 2 and Tijen Tülüba¸s 1 1Faculty of Education, Kutahya Dumlupınar University, 43100 Kütahya, Türkiye; [email protected] (T.K.); [email protected] (T.T.) 2Faculty of Education, Firat University, 23119 Elazı˘g, Türkiye; [email protected] (M.D.); [email protected] (H.P.); [email protected] (T.Y.O.) 3Faculty of Education, Sutcuimam University, 46050 Kahramanmaras, Türkiye *Correspondence: yir[email protected] Abstract: Artificial intelligence (AI) is one of the ground-breaking innovations of the 21st century that has accelerated the digitalization of societies. ChatGPT is a newer form of AI-based large language model that can generate complex texts that are almost indistinguishable from human-generated text. It has already garnered substantial interest from people due to its potential utility in a variety of contexts. The current study was conducted to evaluate the utility of ChatGPT in generating accurate, clear, concise, and unbiased information that could support a scientific research process. To achieve this purpose, we initiated queries on both versions of ChatGPT regarding digital school leadership and teachers’ technology integration, two significant topics currently discussed in educational literature, under four categories: (1) the definition of digital leadership, (2) the digital leadership skills of school principals, (3) the factors affecting teachers’ technology integration, and (4) the impact of digital leadership on teachers’ technology integration. Next, we performed a comparative analysis of the responses generated by ChatGPT-3.5 and ChatGPT-4. The results showed that both versions were capable of providing satisfactory information compatible with the relevant literature. However, ChatGPT-4 provided more comprehensive and categorical information as compared to ChatGPT3.5, which produced responses that were more superficial and short-cut. Although the results are promising in aiding the research process with AI-based technologies, we should also caution that, in their current form, these tools are still in their infancy, and there is a long way to go before they become fully capable of supporting scientific work. Meanwhile, it is significant that researchers continue to develop the relevant knowledge base to support the responsible, safe, and ethical integration of these technologies into the process of scientific knowledge creation, as Pandora’s box has already been opened, releasing newer opportunities and risks to be tackled. Keywords: digital leadership; technology integration; school leadership; ChatGPT; artificial intelligence; generative AI; AI in education; chatbot 1. Introduction People have witnessed numerous ground-breaking technologies that have altered fundamental operations in their lives, particularly since the beginning of the 21st century. However, recent developments in machine learning and artificial intelligence are establishing the ground for unprecedented breakthroughs that will revolutionize the way we live. Artificial intelligence is defined as the ability of machines to produce automated solutions to accomplish the tasks assigned to them, particularly with minimum or even without human intervention (Aghion et al. 2018;Blumenthal 2017;McCarthy et al. 2006; Adm. Sci. 2023,13, 157. https://doi.org/10.3390/admsci13070157 https://www.mdpi.com/journal/admsci Adm. Sci. 2023,13, 157 2 of 19 Shubhendu and Vijay 2013 ). It is, in fact, a learning system that uses mathematical algorithms to perform tasks that require human intelligence (de Saint Laurent 2018; Stokes and Palmer 2020 ) and incorporates the use of other systems such as fuzzy logic, intelligent factors, genetic algorithm, artificial neural networks, and expert systems ( Elmas 2021 ; Öngöz 2020). Machine learning, a sub-discipline of supervised and unsupervised learning fields, is a significant component of building artificial intelligence systems, if not the only one (Russel and Norvig 1995), and plays a significant role in optimizing the performance of artificial intelligence by training it on sample data or past experience (Alpaydin 2004). In other words, machine learning enables analyzing patterns in data in order to make predictions and suggestions based on their self-created algorithms (Mitchell 1997). Artificial intelligence technologies such as robotic systems, autonomous vehicles, facial recognition, natural language processing, and virtual agents are currently used to solve a wide variety of problems (Berente et al. 2021) and play an important role in the digitalization of societies (Cooper 2023). ChatGPT (Chat Generative Pre-trained Transformer) is a new form of generative artificial intelligence that is capable of generating texts with human-like language (OpenAI 2023a). Artificial intelligence tools such as ChatGPT are designed to generate complex texts that are indistinguishable from human-generated text, and they can be utilized in a wide variety of contexts (Dwivedi et al. 2023). With the use of machine learning algorithms, ChatGPT was trained on the structure of the language by processing terabytes of data (OpenAI 2022;Scharth 2022) so that it can present meaningful content upon users’ requests (Halaweh 2023). After its launch in November 2022 by OpenAI Limited Partnership, ChatGPT has rapidly reached more than 100 million users ( Milmo 2023 ), and people have tested its utility for a wide variety of use cases such as wiring poems and stories, taking life advice, producing software codes, performing mathematical calculations and statistical analysis as well as translating texts into different languages (Atlas 2023; D’Amico et al. 2023;Karakose 2023;Mhlanga 2023;Scharth 2022;van Dis et al. 2023). After the release of ChatGPT version 3.5, Open AI continued its work on ChatGPT to equip it with better capabilities while reducing the limitations of the previous versions. On 15 March 2023, they opened the latest version, i.e., Chat-GPT-4, for public use. As explained in its technical report (OpenAI 2023b), the performance of ChatGPT4 was much beyond that of ChatGPT-3.5 on many tasks. For example, ChatGPT-4 not only outperformed ChatGPT-3.5 but also most human test takers in various exams originally designed for humans. ChatGPT-4 was also proven to perform better than any other state-of-the-art-systems available at the time of its release, and it demonstrated superior performance in differentiating fact from incorrect information, following user intent, reasoning, and generating concise responses. This may have particularly resulted from the fact that ChatGPT-3.5 was trained on 175 billion parameters, while GPT 4 was trained on 100 trillion parameters ( Zaitsu and Jin 2023 ). The developments also helped reduce its likelihood of generating hallucinated or incorrect information, mentioned as a serious limitation of ChatGPT-3.5 (OpenAI 2023b). Recent research conducted after the release of ChatGPT-4 (e.g., Teebagy et al. 2023 ;West 2023;Tülüba¸s et al. 2023) also indicates that ChatGPT-4 demonstrates capabilities far beyond its previous version, even in showing emotions ( Zhao et al. 2023 ), performing inductive reasoning and inferring peoples’ feelings or perspectives (Michail et al. 2023), despite acknowledging that there is still much room for improvement if it is expected to act like a human. ChatGPT can continuously improve itself by using the feedback received from its users (Farrokhnia et al. 2023;Mann 2023), and has the ability to find and summarize information retrieved from the internet literature in response to users’ queries (Cascella et al. 2023). Hence, ChatGPT is capable of performing complex tasks such as assisting language learning (Jia et al. 2022), promoting critical thinking (Hapsari and Wu 2022), improving academic writing skills (Zhai 2022), and developing programming skills (Biswas 2023). It is even considered to be a useful tool for reducing teachers’ workload in designing and performing instruction (Qadir 2023). ChatGPT can also provide significant guidance at different stages Adm. Sci. 2023,13, 157 3 of 19 of a research process, such as brainstorming, data collection and analysis, critical evaluation of data, and reporting research results in written form. In addition to the above-mentioned benefits, ChatGPT also has some shortcomings and weaknesses. Some scholars state that responses generated by ChatGPT do not have philosophical depth (Bogost 2022;Gao et al. 2023), and it could produce answers irrelevant to the topic of the query (Gupta et al. 2023). Another significant concern is about the reliability and accuracy of the responses, which are generated based on the corpus data on which it is trained (Lecler et al. 2023;Sallam 2023;Stokel-Walker and Van Noorden 2023). Therefore, ChatGPT has the potential to produce biased responses since it cannot identify the bias existing in its training corpus or the internet resource it reaches ( Barocas and Selbst 2016 ; Zhai 2022). In addition, it is known that ChatGPT can produce inaccurate or irrelevant references when asked to provide the source of reference for the information generated (Choi et al. 2023), and could cause plagiarism and copyright problems as some responses could be directly taken from published research (Kasneci et al. 2023). Although much research is conducted on the use of ChatGPT, particularly ChatGPT3.5, with regard to its performance on several tasks (Sallam 2023;Taecharungroj 2023), a review of the literature reveals a limited amount of research on the use of ChatGPT to support scientific work. Considering this gap in the literature, the current study aims to conduct a research process in collaboration with human and artificial intelligence using the responses taken from ChatGPT-3.5 and ChatGPT-4 on a specific research topic in the field of educational administration. We particularly focused on the investigation of two recent and related topics in the field–digital leadership and teachers’ technology integration. To achieve this purpose, we first had a conversation with the two versions of ChatGPT about school principals’ digital leadership and teachers’ technology integration. Next, we conducted a comparative analysis of their responses. Using both versions of ChatGPT, we aimed to conduct a comparative assessment of whether they could provide accurate, clear, concise, and unbiased information. Digital Leadership and Teachers’ Technology Integration Digitalization has become an inevitable truth in the current age of technological breakthroughs. Digital technologies transform many operations in daily life, such as production, logistics, communication, and human resources management ( Oberer and Erkollar 2018 ). As modern organizations transform into digital work environments, digital leadership has become significant in enabling successful digital transformation (Arham et al. 2023; Ghavifekr and Pei 2023 ). Digital leadership is used as an umbrella term that encompasses several leadership styles, such as technology leadership, virtual leadership, e-leadership, and leadership 4.0, all of which are used interchangeably in the literature ( Karakose et al. 2022 ). Digital leadership also involves utilizing technology effectively so as to provide better work conditions for employees (Berisha-Gawlowski et al. 2021). In the context of education, Zhong (2017) defines digital leadership as using instructional technologies such as digital devices, services, and resources to inspire and lead the digital transformation of the school, create and maintain a culture of digital learning, support and develop technology-based professional development, and eventually enable a digitalfriendly school environment. From this point of view, digital leadership can be defined as a technology-based leadership model that involves the effective use of technology to promote schools’ functioning. The use of advanced technologies in the educational context has recently become widely acknowledged (Al-Ruz and Khasawneh 2011), and teachers of all grades are now expected to integrate technology successfully into their classrooms (Keengwe et al. 2008). Teachers felt this need even more intensely during the COVID-19 pandemic, which has accelerated the transformation of education systems around the world by encouraging or even mandating the integration of digital technologies into education ( Hamzah et al. 2021 ). Educational leaders and policy makers argue that digital technology integration has become the key to fostering student engagement and achievement (Howley et al. 2011). Scientists Adm. Sci. 2023,13, 157 4 of 19 have emphasized that technology integration improves students’ critical and creative thinking skills, and increases their academic success and motivation (Siddiqui et al. 2020; Yilmaz 2021). As a result, it is frequently recommended to develop the digital skills of both teachers and students by integrating advanced technologies into previously humancentered practices (Ruggiero and Mong 2015). The successful integration of technology into education entails teachers’ effective use of digital technologies in their classroom practices. Therefore, the attitudes and beliefs of teachers towards using these technologies significantly affect technology integration (Alghamdi and Prestridge 2015;Christensen 2002;Ertmer et al. 2012;Kim et al. 2013; Liu 2011 ). In addition, contextual factors such as the availability of these technologies and having easy access to such resources also promote technology integration to a great extent (Howley et al. 2011;Sauers and McLeod 2018;Shuldman 2004). As stated by several scholars, providing teachers with administrative encouragement and support is significant in enabling the successful integration of new technologies into education (Al-Ruz and Khasawneh 2011;Chang 2012;Keengwe et al. 2008;Kim et al. 2013;McLeod 2015). McLeod (2015) emphasized that the technology-based transformation of schools necessitates instructional leadership with a good vision of technology integration. Therefore, school principals are considered to have a significant role in developing teachers’ capabilities to conduct technology-integrated instruction and motivating them to use the latest technologies for the benefit of students (Chang 2012;Navaridas-Nalda et al. 2020). In the face of rapid digital transformation, both teachers and school principals are expected to become proficient in using technology. According to Hensellek (2020), as digital leaders, school principals not only need to provide a strategic vision for a digital future but also have the necessary digital skills and attitudes to achieve this vision in collaboration with all the stakeholders. Aksal (2016) emphasized that digital leadership could enhance technology integration into both the teaching-learning processes and the management of schools. School principals who perform successful digital leadership can also create a digital learning culture in schools, which supports digital transformation and technologybased professional development (Karakose et al. 2021; Karakose and Tülüba¸s 2023 ). Therefore, school principals should be active participants and role models in technology integration and should espouse technology integration as the core task of their leadership (McLeod 2015) so as to enable the digital transformation of schools. 2. Materials and Methods This study aims to conduct research with the collaboration of human and artificial intelligence in order to see the possible contribution of AI-based systems to generating accurate, clear, concise, and unbiased information. With this purpose, we conducted queries on digital school leadership and its potential impact on teachers’ technology integration using two versions of an AI-based large language model (LLM): ChatGPT-3.5 and ChatGPT4. We preferred to conduct a comparative assessment, which could help identify the similarities and differences of responses yielded by each version and observe whether ChatGPT-4 brings in any advances and innovations with regard to response generation. We conducted the research in three stages. First, we reviewed the relevant literature about digital leadership and teachers’ technology integration so as to develop our categories to interrogate ChatGPT. At this stage, we also prepared a list of questions to get responses for each category of query. During this initial stage, all the researchers first worked individually. Later, we performed a panel discussion to develop our final list of categories and related questions. Eventually, we agreed upon four categories with a single question to be used for each: (1) the definition of digital leadership, (2) the digital leadership skills of school principals, (3) the factors affecting teachers’ technology integration, and (4) the impact of digital leadership on teachers’ technology integration. Second, we made simultaneous queries on ChatGPT-3.5 and ChatGPT-4 on 13 April 2023, using the question developed for each category without including any additional prompts or questions to minimize human interference and guidance during the queries. Adm. Sci. 2023,13, 157 5 of 19 After recording responses given by each version of ChatGPT, we moved on to the third stage: the analysis of responses yielded for each category. While analyzing the quality of these responses, we used two different rating systems. To evaluate the inter-rater agreement of researchers on the quality of responses, we calculated Cohen’s kappa for each category. Cohen’s kappa is often used in social and medical sciences as a chance-corrected means of assessing inter-rater agreement on a nominal scale. It is considered to be an efficient measurement to determine whether the degree of agreement is obtained for real or by chance (Sun 2011;Warrens 2015). Hence, Cohen’s kappa is used as a robust statistic to assess inter-rater agreement, and the results are used to evaluate the quality of information assessed by two raters (Vieira et al. 2010). In the current study, we calculated the value of Cohen’s kappa for each category using the package program SPSS version 26 and interpreted the results according to the benchmark suggested by Landis and Koch (1977): 0.00–0.20 = slight inter-rater agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = strong agreement, and 0.81–1.00 = almost perfect agreement. To assess (1) the accuracy, (2) clarity, (3) conciseness, and (4) the potential for bias in the information provided by each version of ChatGPT, we developed a four-dimensional rating scale with three points of rating: (1) meant that the information is completely inaccurate, unclear, unconcise, or biased, (2) meant that it was partly accurate, clear, concise, or biased, and (3) meant that it was completely accurate, clear, concise, or unbiased. Using this rating scale, all researchers individually rated the responses yielded for each category in terms of accuracy, clarity, conciseness, and potential bias, and later the averages of these rating points were taken for each category of evaluation. As data for the current study were tested based on the individual understandings and evaluations of six researchers who participated in the research, it would also be proper to declare some characteristics of the researchers to support the credibility of the results. All six researchers who conducted the analysis were educational specialists working on two distinct sub-fields. Three of the researchers were experts on educational administration, management, and leadership, while the other three worked particularly in the area of educational technologies and were experts on designing newer technologies to support education. In fact, all researchers were familiar with the topics under investigation and were interested in exploring technological innovations. The diversity of backgrounds was positive in that expertise in educational management and leadership helped evaluate the accuracy and conciseness of information with regard to digital leadership, while expertise in educational technologies helped evaluate the responses with regard to teachers’ technology integration. It also enabled reflection of different views regarding the responses. However, it should also be noted that the personal characteristics, expectancies, and views of each researcher could also ignite some level of bias in their ratings, although this is difficult to uncover for sure. However, the results of Cohen’s kappa and the inclusion of ratings from six researchers (rather than a single researcher) were useful in eliminating personal bias in the analysis. 3. Results This section reports on the responses generated by ChatGPT-3.5 and ChatGPT-4 on the previously defined four categories of digital school leadership and its impact on teachers’ technology integration: (1) the definition of digital leadership, (2) the digital leadership skills of school principals, (3) the factors affecting teachers’ technology integration, and (4) the impact of digital leadership on teachers’ technology integration. In particular, the section provides the comparative evaluation of responses given by two versions of ChatGPT for each category, the results of researcher ratings on the accuracy, clarity, conciseness, and bias potential of information provided by these responses, and the inter-rater agreements on the quality of information with this regard (i.e., the Cohen kappa values). We also included the snapshots of responses generated by ChatGPT-3.5 and ChatGPT-4 to illustrate the results for each category. Adm. Sci. 2023,13, 157 6 of 19 We started our interrogation with ‘the definition of digital leadership in education’, and sample excerpts from the responses of ChatGPT-3.5 and ChatGPT-4 are presented in Figure 1. Adm. Sci. 2023, 13, x FOR PEER REVIEW 6 of 19 We started our interrogation with ‘the definition of digital leadership in education’, and sample excerpts from the responses of ChatGPT-3.5 and ChatGPT-4 are presented in Figure 1. RESEARCH THEME: The Definition of Digital Leadership Model: GPT-3.5 Model: GPT-4 Figure 1. Sample excerpts from responses of ChatGPT-3.5 and ChatGPT-4 for the definition of digital leadership in education (generated on 13 April 2023). ChatGPT-3.5 defined digital leadership as the ability of educational leaders to integrate technology into their learning and teaching processes, while ChatGPT-4 defined it as the ability to integrate technology into school management processes as well as learning and teaching processes, stating that school administrators, principals and teachers are educational leaders who can assume a digital leadership role in schools. In addition, Figure 1. Sample excerpts from responses of ChatGPT-3.5 and ChatGPT-4 for the definition of digital leadership in education (generated on 13 April 2023). ChatGPT-3.5 defined digital leadership as the ability of educational leaders to integrate technology into their learning and teaching processes, while ChatGPT-4 defined it as the ability to integrate technology into school management processes as well as learning and teaching processes, stating that school administrators, principals and teachers are educa- Adm. Sci. 2023,13, 157 7 of 19 tional leaders who can assume a digital leadership role in schools. In addition, ChatGPT-4 stated that digital leadership requires vision and strategy, digital citizenship, communication and collaboration, innovation and adaptability, and data-driven decision making, which would, in turn, help create a transformative and innovative learning environment. Both versions of ChatGPT underlined similar dimensions of digital leadership, such as supporting and promoting the professional development of educators, creating a digitalfriendly culture at school, and developing strategies to support this culture using innovative technologies, communicating and cooperating with all stakeholders, implementing effective decision-making processes, and enhancing technology used for the benefit of students. However, while ChatGPT-3.5 associated these skills with teachers and teaching-learning processes, ChatGPT-4 tended to associate them with the management roles of school administrators in addition to teaching and learning processes. In addition, ChatGPT-4 was able to generate more comprehensive, detailed, and categorized information as compared to ChatGPT-3.5. The average values of ratings based on the four-dimensional rating scale for the accuracy, clarity, conciseness, and potential bias in the information provided by ChatGPT3.5 were 2.67, 2.50, 2.16, and 2.83, respectively, while it was 2.83, 2.67, 2.83, and 2.83 for ChatGPT-4 responses (see Table 1). These results indicate that both versions of ChatGPT were able to generate satisfactory information on the definition of digital leadership in education, whereas ChatGPT-4 received better rates, particularly with regard to the accuracy and conciseness of information. For instance, as elaborated above, ChatGPT-4 underlined that educational leaders are not limited to school principals. Teachers and other staff with administrative roles could assume the role of digital leadership, while ChatGPT-3.5 did not make this clarification. Likewise, while presenting the roles of a digital leader, ChatGPT-4 presented these roles under categorical sub-headings such as vision and strategy, professional development, digital citizenship, communication and collaboration, innovation and adaptability, data-driven decision making, and offered detailed information about each of these categories in relation to digital leadership. ChatGPT-3.5, on the other hand, just summarized these roles in a single, short paragraph. Although both versions tend to offer a short summary of the information generated for each question, it was also evident that ChatGPT-4 represented a more positive stance towards adopting digital leadership (e.g., “By embracing digital leadership . . . ”), as also underlined in the literature, while ChatGPT-3.5 just summarized the information it provided (e.g., “Overall, digital leadership is about . . . ”). Table 1. The Rating Scores for the Definition of Digital Leadership. ChatGPT-3.5 ChatGPT-4 R1 R2 R3 R4 R5 R6 x R1 R2 R3 R4 R5 R6 x Accuracy 2 3 3 2 3 3 2.67 3 3 3 2 3 3 2.83 Clarity 2 3 2 3 2 3 2.50 2 3 3 2 3 3 2.67 Conciseness 2 3 2 2 2 2 2.16 3 2 3 3 3 2 2.83 Bias Possibility 2 3 3 3 3 3 2.83 2 3 3 3 3 3 2.83 The Cohen’s kappa value for the quality of responses under this category was calculated as 0.89 for ChatGPT-3.5 and 0.94 for ChatGPT-4. According to Landis and Koch’s (1977) benchmark, these results indicate an ‘almost perfect’ agreement of inter-raters with regard to the quality of these responses in terms of accuracy, clarity, conciseness, and potential bias. We continued our interrogation with the ‘digital leadership skills of school principals’, and sample excerpts from the responses of both versions of ChatGPT are shown in Figure 2. The digital leadership skills listed by both versions of ChatGPT were school principals’ proficiency in integrating technology into teaching and learning processes. The responses had some similar aspects as well as differences. Both versions underlined technology profi- Adm. Sci. 2023,13, 157 8 of 19 ciency/literacy, communication, collaboration, and change management as the required skills of digital leaders. However, unlike ChatGPT-3.5, ChatGPT-4 emphasized the role of school principals in driving digital transformation, focusing on newer concepts such as digital literacy and fostering digital citizenship. Adm. Sci. 2023, 13, x FOR PEER REVIEW 8 of 19 RESEARCH THEME: Digital Leadership Skills of School Principals Model: GPT-3.5 Model: GPT-4 Figure 2. Sample excerpts from responses of ChatGPT-3.5 and ChatGPT-4 for principals’ digital leadership skills (generated on 13 April 2023). The digital leadership skills listed by both versions of ChatGPT were school principals’ proficiency in integrating technology into teaching and learning processes. The responses had some similar aspects as well as differences. Both versions underlined technology proficiency/literacy, communication, collaboration, and change management as the required skills of digital leaders. However, unlike ChatGPT-3.5, ChatGPT-4 emphasized the role of school principals in driving digital transformation, focusing on newer concepts such as digital literacy and fostering digital citizenship. In addition, ChatGPT-3.5 stated that strategic thinking and ethical leadership are among the digital leadership skills of school principals. ChatGPT-4, on the other hand, Figure 2. Sample excerpts from responses of ChatGPT-3.5 and ChatGPT-4 for principals’ digital leadership skills (generated on 13 April 2023). In addition, ChatGPT-3.5 stated that strategic thinking and ethical leadership are among the digital leadership skills of school principals. ChatGPT-4, on the other hand, emphasized current concepts such as digital transformation, the digital age, and data-based Adm. Sci. 2023,13, 157 15 of 19 the potential use of LLMs like ChatGPT in the scientific research process. Although several concerns and potential use cases have already been identified in the literature, there is still a need for more comprehensive studies to ensure the responsible, safe and ethical integration of these technologies into the process of scientific knowledge creation because burying our heads into the sand with complete ignorance would be of no use since that box has already been opened. Therefore, we strongly recommend that future research should continue to address the opportunities and threats of ChatGPT with scientific methods and identify ways of maximizing its benefits while reducing possible risks, such as generating unethical or misleading results, as well as overcoming accountability or authorship concerns. Multidisciplinary research conducted with the participation of experts from a large scope of fields could particularly be useful to address the issue from multiple perspectives and yield well-rounded results. In addition, studies addressing the perspectives of scientists on the possible benefits and threats of using ChatGPT, conferring on their ideas regarding the effective or fraudulent use of these technologies for scientific work through qualitative research designs, could provide deeper insights in this regard. It should also be noted that AI-based technologies continue to develop, and their newer versions keep being released, so the relevant literature warrants updating in parallel with these developments. In addition, the current study particularly aimed to conduct a comparative analysis between the responses generated by the two versions of ChatGPT over a single set of queries. However, future research could also conduct a similar study using responses generated from a cycle of multiple queries so as to conduct temporal comparisons and identify any differences in the quality of the responses generated over different time intervals. Despite its contribution to the debates over the potential use of ChatGPT for scientific work through comparative assessment of ChatGPT-3.5 and ChatGPT 4 responses, the current study bears some limitations. One major limitation is that the assessments of ChatGPT content were based on the subjective evaluations of researchers to a large extent, although their evaluations relied on an extensive literature review of digital school leadership and teacher technology integration, and a variety of evaluation methods were used to obtain the results presented in the study. Another limitation is that queries in the current study were made in only English, so the results should not be generalized to other languages. Considering that ChatGPT could generate responses in several languages, a similar study design could be realized in other languages to identify divergent and convergent results. Finally, although the current study yielded optimistic results regarding the potential contribution of ChatGPT to identifying, accumulating, and categorizing a large amount of information in a matter of minutes, it is crucial to remember that we should continue critically evaluating the accuracy and utility of information generated by these LLMs as they still lack scientific reasoning and critical thinking, and have the potential to generate infodemics or artificial hallucinations (Alkaissi and McFarlane 2023;Ollivier et al. 2023) although not observed in the current analysis. This difference in results could have resulted from the topics under investigation or could be specific to research fields such as medicine, economics, physics, or law. Further investigations could increase our understanding with regard to this topic and would contribute greatly to ChatGPT literature. Author Contributions: Conceptualization, T.K.; methodology, T.K. and T.T.; formal analysis, T.K., T.T., R.Y., M.D., H.P. and T.Y.O.; data curation, T.K., T.Y.O. and R.Y.; writing—original draft preparation, T.K., T.T. and H.P.; writing, T.K., T.T., R.Y., M.D., H.P. and T.Y.O.; review and editing, T.K., T.T., M.D., H.P., T.Y.O. and R.Y.; supervision, T.K. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: No identifying information was collected or included. All the data used in the results section of this research was accessed through ChatGPT-3.5 and ChatGPT 4. Adm. Sci. 2023,13, 157 16 of 19 Acknowledgments: The authors acknowledge the contribution of both versions of ChatGPT and thank the working team of OpenAI for supporting this work through the development of this generative AI-based language model. Conflicts of Interest: The authors declare no conflict of interest. References Aghion, Philippe, Benjamin F. Jones, and Charles I. Jones. 2018. Artificial intelligence and economic growth. In The Economics of Artificial Intelligence: An Agenda. Edited by Ajay Agraval, Joshua Gans and Avi Goldfarb. Chicago: University of Chicago Press, pp. 237–282. Agustina, Rini, Waras Kamdi, Syamsul Hadi, and Didik Nurhadi. 2020. Influence of the principal’s digital leadership on the reflective practices of vocational teachers mediated by trust, self efficacy, and work engagement. International Journal of Learning, Teaching and Educational Research 19: 24–40. [CrossRef] Aksal, F. A. 2016. Okul kültüründe müdürler dijital lider mi? E˘gitim ve Bilim 40: 77–86. Alghamdi, Abdulmajeed, and Sarah Prestridge. 2015. Alignment between principal and teacher beliefs about technology use. Australian Educational Computing 30: 1. Available online: http://journal.acce.edu.au/index.php/AEC/article/view/52/pdf (accessed on 21 April 2023). Alkaissi, Hussam, and Samy I. McFarlane. 2023. Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15: 1–5. [CrossRef] [PubMed] Alpaydin, Ethem. 2004. Introduction to Machine Learning. Adaptive Computation and Machine Learning Series; Cambridge: The MIT Press. Al-Ruz, Jamal Abu, and Samer Khasawneh. 2011. Jordanian pre-service teachers’ and technology integration: A human resource development approach. Journal of Educational Technology & Society 14: 77–87. Arham, Ahmad Fadhly, Nor Sabrena Norizan, Ahmad Firdhaus Arham, Nornajihah Nadia Hasbullah, Irfah Najihah Basir Malan, and Shaliza Alwi. 2023. Digital leadership in education: A meta-analysis review. In Digitalisation: Opportunities and Challenges for Business: Volume 2. Edited by Bahaeddin Alareeni, Allam Hamdan, Reem Khamis and Rim el Khoury. Cham: Springer International Publishing, pp. 849–57. Atlas, Stephen. 2023. ChatGPT for Higher Education and Professional Development: A Guide to Conversational AI. Available online: https://digitalcommons.uri.edu/cba_facpubs/548 (accessed on 18 March 2023). Barocas, Solon, and Andrew D. Selbst. 2016. Big data’s disparate impact. California Law Review 104: 671–732. [CrossRef] Berente, Nicholas, Bin Gu, Jan Recker, and Radhika Santhanam. 2021. Managing artificial intelligence. MIS Quarterly 45: 1433–50. Berisha-Gawlowski, Angelina, Carina Caruso, and Christian Harteis. 2021. The concept of a digital twin and its potential for learning organizations. In Digital Transformation of Learning Organizations. Edited by Dirk Ifenthaler, Dirk Hofhues, Marc Egloffstein and Christian Helbig. Cham, Switzerland: Springer Open, pp. 95–114. Biswas, Som. 2023. Role of ChatGPT in computer programming: ChatGPT in computer programming. Mesopotamian Journal of Computer Science 2023: 8–16. [CrossRef] Blumenthal, David. 2017. Data withholding in the age of digital health. Milbank Quarterly 72: 15–18. [CrossRef] Bogost, Ian. 2022. ChatGPT Is Dumber than You Think. Available online: https://www.theatlantic.com/technology/archive/2022/12/ chatgpt-openai-artificial-intelligence-writing-ethics/672386/ (accessed on 28 April 2023). Bolte, Sophia, Joanna Dehmer, and Jörg Niemann. 2018. Digital Leadership 4.0. Acta Technica Napocensis-Series: Applied Mathematics, Mechanics, and Engineering 61: 637–646. Cascella, Marco, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023. Evaluating the feasibility of ChatGPT in healthcare: An analysis of multiple clinical and research scenarios. Journal of Medical Systems 47: 33. [CrossRef] [PubMed] Chang, I-Hua. 2012. The Effect of principals’ technological leadership on teachers. Technological literacy and teaching effectiveness in Taiwanese elementary schools. Educational Technology & Society 15: 328–40. Choi, Jonathan H., Kristin E. Hickman, Amy Monahan, and Daniel B. Schwarcz. 2023. ChatGPT goes to law school. Journal of Legal Education. forthcoming. Available online: https://ssrn.com/abstract=4335905 (accessed on 18 June 2023). [CrossRef] Christensen, Rhonda. 2002. Effects of technology integration education on the attitudes of teachers and students. Journal of Research on Technology in Education 34: 411–33. [CrossRef] Cooper, Grant. 2023. Examining science education in chatgpt: An exploratory study of generative artificial intelligence. Journal of Science Education and Technology 32: 444–52. [CrossRef] D’Amico, Randy S., Timothy G. White, Harchal A. Shah, and David J. Langer. 2023. I asked a ChatGPT to write an editorial about how we can incorporate chatbots into neurosurgical research and patient care. Neurosurgery 92: 663–64. [CrossRef] de Saint Laurent, Constance. 2018. In defence of machine learning: Debunking the myths of artificial intelligence. Europe’s Journal of Psychology 14: 734–47. [CrossRef] Dwivedi, Yogesh K., Nir Kshetri, Laurie Hughes, Emma Louise Slade, Anand Jeyaraj, Arpan Kumar Kar, A. M. Baabdullah, M. Ahuja, H. Albanna, M. A. Albashrawi, and et al. 2023. “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management 71: 102642. [CrossRef] Adm. Sci. 2023,13, 157 17 of 19 Eberl, Julia K., and Paul Drews. 2021. Digital leadership—Mountain or molehill? A literature review. In Innovation Through Information Systems. WI 2021. Lecture Notes in Information Systems and Organisation, vol 48. Edited by Frederick Ahlemann, Reinhard Schütte and Stefan Stieglitz. Cham: Springer, pp. 223–37. [CrossRef] Elmas, Çetin. 2021. Yapay zeka uygulamaları. Ankara: Seçkin Yayıncılık. Ertmer, Peggy A., Anne T. Ottenbreit-Leftwich, Olgun Sadik, Emine Sendurur, and Polat Sendurur. 2012. Teacher beliefs and technology integration practices: A critical relationship. Computers & Education 59: 423–35. Farrokhnia, Mohammedreza, Seyyed Kazem Banihashem, Omid Noroozi, and Arjen Wals. 2023. A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 1–15. [CrossRef] Gao, Jun, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of ChatGPT for event extraction. arXiv arXiv:2303.03836. Ghavifekr, Simin, and Xiaozhu Pei. 2023. Leading global digitalization in higher education. In Global Perspectives on the Internationalization of Higher Education. Hershey: IGI Global, pp. 1–15. Gupta, Puneet, Swati Raturi, and P. Venkateswarlu. 2023. ChatGPT for designing course outlines: A boon or bane to modern technology. SSRN Electronic Journal. Available online: https://ssrn.com/abstract=4386113 (accessed on 18 June 2023). [CrossRef] Halaweh, Mohanad. 2023. ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology 15: ep421. [CrossRef] Hamzah, N. Hafiza, M. Khalid M. Nasir, and Jamalullail Abdul Wahab. 2021. The effects of principals’ digital leadership on teachers’ digital teaching during the COVID-19 Pandemic in Malaysia. Journal of Education and E-Learning Research 8: 216–21. [CrossRef] Hapsari, Intan Permata, and Ting-Ting Wu. 2022. AI Chatbots learning model in English speaking skill: Alleviating speaking anxiety, boosting enjoyment, and fostering critical thinking. Paper presented at International Conference on Innovative Technologies and Learning, Porto, Portugal, August 29–31; pp. 444–53. [CrossRef] Hensellek, Simon. 2020. Digital leadership: A framework for successful leadership in the digital age. Journal of Media Management and Entrepreneurship (JMME) 2: 55–69. [CrossRef] Howley, Aimee, Lawrence Wood, and Brian Hough. 2011. Rural elementary school teachers’ technology integration. Journal of Research in Rural Education 26: 1–13. Jia, Fenglin, Daner Sun, Qing Ma, and Chee-Kit Looi. 2022. Developing an AI-Based learning system for L2 learners’ authentic and ubiquitous learning in English language. Sustainability 14: 15527. [CrossRef] Kane, Gerald C., Anh Nguyen Phillips, Jonathan Copulsky, and Garth Andrus. 2019. How digital leadership is (n’t) different. MIT Sloan Management Review 60: 34–39. Karakose, Turgut. 2023. The utility of ChatGPT in educational research—Potential opportunities and pitfalls. Educational Process: International Journal 12: 7–13. [CrossRef] Karakose, Turgut, and Tijen Tülüba¸s. 2023. Digital leadership and sustainable school improvement—A conceptual analysis and implications for future Research. Educational Process: International Journal 12: 7–18. [CrossRef] Karakose, Turgut, Hakan Polat, and Stamatios Papadakis. 2021. Examining teachers’ perspectives on school principals’ digital leadership roles and technology capabilities during the COVID-19 pandemic. Sustainability 13: 13448. [CrossRef] Karakose, Turgut, Ibrahim Kocabas, Ramazan Yirci, Stamatios Papadakis, Tuncay Yavuz Ozdemir, and Murat Demirkol. 2022. The development and evolution of digital leadership: A bibliometric mapping approach-based study. Sustainability 14: 16171. [CrossRef] Kasneci, Enkelejda, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, and et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences 103: 102274. [CrossRef] Keengwe, Jared, Grace Onchwari, and Patrick Wachira. 2008. Computer technology integration and student learning: Barriers and promise. Journal of Science Education and Technology 17: 560–65. [CrossRef] Kim, ChanMin, Min Kyu Kim, Chiajung Lee, J. Michael Spector, and Karen DeMeester. 2013. Teacher beliefs and technology integration. Teaching and Teacher Education 29: 76–85. [CrossRef] Landis, J. Richard, and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33: 159–74. [CrossRef] Lecler, Augustin, Loïc Duron, and Philippe Soyer. 2023. Revolutionizing radiology with GPT-based models: Current applications, future possibilities, and limitations of ChatGPT. Diagnostic and Interventional Imaging 104: 269–74. [CrossRef] Liu, Shih-Hsiung. 2011. Factors related to pedagogical beliefs of teachers and technology integration. Computers & Education 56: 1012–22. Mann, Douglas L. 2023. Artificial Intelligence discusses the role of artificial intelligence in translational medicine. JACC: Basic to Translational Science 8: 221–23. [CrossRef] McCarthy, John, Marvin L. Minsky, Nathaniel Rochester, and Claude E. Shannon. 2006. A proposal for the dartmouth summer research project on artificial intelligence, August 31, 1955. AI Magazine 27: 12–19. [CrossRef] McLeod, Scott. 2015. The challenges of digital leadership. Independent School 74: n2. Mhlanga, David. 2023. Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. SSRN. Available online: https://ssrn.com/abstract=4354422 (accessed on 18 June 2023). [CrossRef] Adm. Sci. 2023,13, 157 18 of 19 Michail, Aandrianos, Stefanos Konstantinou, and Simon Clematide. 2023. Uzh clyp at semeval-2023 task 9: Head-first fine-tuning and ChatGPT data generation for cross-lingual learning in tweet intimacy prediction. arXiv arXiv:2303.01194. Milmo, Dan. 2023. ChatGPT reaches 100 million users two months after launch. The Guardian. February 2. Available online: https://www.theguardian.com/technology/2023/feb/02/chatgpt-100-million-users-open-ai-fastest-growing-app (accessed on 18 June 2023). Mitchell, Tom Michael. 1997. Machine Learning, 1st ed. New York: McGraw-Hill. Navaridas-Nalda, Fermín, Mónica Clavel-San Emeterio, Rubén Fernández-Ortiz, and Mario Arias-Oliva. 2020. The strategic influence of school principal leadership in the digital transformation of schools. Computers in Human Behavior 112: 106481. [CrossRef] Oberer, Birgit, and Alptekin Erkollar. 2018. Leadership 4.0: Digital leaders in the age of industry 4.0. International Journal of Organizational Leadership. Available online: https://ssrn.com/abstract=3337644 (accessed on 6 March 2023). Ollivier, Matthieu, Ayoosh Pareek, Jari Dahmen, M. Enes Kayaalp, Philipp W. Winkler, Michael T. Hirschmann, and Jon Karlsson. 2023. A deeper dive into ChatGPT: History, use and future perspectives for orthopaedic research. Knee Surgery, Sports Traumatology, Arthroscopy 1: 1190–92. [CrossRef] Öngöz, Sakine. 2020. Yapay zeka teknolojinin kullanıldı˘gı yeni nesil ö˘gretim materyalleri. In E˘gitimde Yapay Zeka: Kuramdan Uygulamaya. Edited by Vasif Nabiyev and Ali Kür¸sat Erümit. Ankara: Pegem Yayıncılık, pp. 58–85. OpenAI. 2022. ChatGPT. Available online: https://openai.com/blog/ChatGPT (accessed on 24 April 2023). OpenAI. 2023a. ChatGPT: Optimizing Language Models for Dialogue. Available online: https://openai.com/blog/chatgpt (accessed on 24 April 2023). OpenAI. 2023b. GPT-4 Technical Report. Available online: https://cdn.openai.com/papers/gpt-4.pdf (accessed on 24 April 2023). Qadir, Junaid. 2023. Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. Paper presented in IEEE Global Engineering Education Conference (EDUCON), Kuwait, May 1–4; pp. 1–9. [CrossRef] Ruggiero, Dana, and Christopher J. Mong. 2015. The teacher technology integration experience: Practice and reflection in the classroom. Journal of Information Technology Education 14: 161–78. [CrossRef] [PubMed] Russel, Stuart, and Peter Norvig. 1995. Artificial Intelligence: A Modern Approach. Hoboken: Prentice-Hall Inc. Sallam, Malik. 2023. The utility of ChatGPT as an example of large language models in healthcare education, research and practice: Systematic review on the future perspectives and potential limitations. medRxiv. [CrossRef] Sauers, Nicholas J., and Scott McLeod. 2018. Teachers’ technology competency and technology integration in 1: 1 schools. Journal of Educational Computing Research 56: 892–910. [CrossRef] Scharth, Marcel. 2022. The ChatGPT Chatbot Is Blowing People Away with Its Writing Skills. Sydney: The University of Sydney. Available online: https://www.sydney.edu.au/news-opinion/news/2022/12/08/the-chatgpt-chatbot-isblowing-people-away-with-itswriting-skil.html (accessed on 8 March 2023). Shubhendu, Shukla S., and Jaiswal Vijay. 2013. Applicability of artificial intelligence in different fields of life. International Journal of Scientific Engineering and Research (IJSER) 1: 28–35. Shuldman, Mitchell. 2004. Superintendent conceptions of institutional conditions that impact teacher technology integration. Journal of Research on Technology in Education 36: 319–43. [CrossRef] Siddiqui, Sohni, Martin Thomas, and Naureen Nazar Soomro. 2020. Technology integration in education: Source of intrinsic motivation, self-efficacy and performance. Journal of E-Learning and Knowledge Society 16: 11–22. Stokel-Walker, Chris, and Richard Van Noorden. 2023. What ChatGPT and generative AI mean for science. Nature 614: 214–16. [CrossRef] Stokes, Felicia, and Amitabha Palmer. 2020. Artificial intelligence and robotics in nursing: Ethics of caring as a guide to dividing tasks between AI and humans. Nursing Philosophy 21: e12306. [CrossRef] Sun, Shuyan. 2011. Meta-analysis of Cohen’s kappa. Health Services and Outcomes Research Methodology 11: 145–63. [CrossRef] Taecharungroj, Viriya. 2023. What Can ChatGPT Do?: Analyzing early reactions to the innovative AI Chatbot on Twitter. Big Data and Cognitive Computing 7: 35. [CrossRef] Teebagy, Sean, Lauren Colwell, Emma Wood, Antonio Yaghy, and Misha Faustina. 2023. Improved performance of ChatGPT-4 on the OKAP exam: A comparative study with ChatGPT-3.5. medRxiv. [CrossRef] Tigre, Fernanda Bethlem, Carla Curado, and Paulo Lopes Henriques. 2023. Digital leadership: A bibliometric analysis. Journal of Leadership & Organizational Studies 30: 40–70. [CrossRef] Tülüba¸s, Tijen, Murat Demirkol, Tuncay Yavuz Ozdemir, Hakan Polat, Turgut Karakose, and Ramazan Yirci. 2023. An interview with ChatGPT on emergency remote teaching: A comparative analysis based on human–AI collaboration. Educational Process: International Journal 12: 91–108. [CrossRef] van Dis, Eva A., Johen Bollen, Willem Zuidema, Robert van Rooij, and Claudi L. Bockting. 2023. ChatGPT: Five priorities for research. Nature 614: 224–26. [CrossRef] Vieira, Susana M., Uzay Kaymak, and João M. C. Sousa. 2010. Cohen’s kappa coefficient as a performance measure for feature selection. Paper presented at International Conference on Fuzzy Systems, Barcelona, Spain, July 18–23; pp. 1–8. Warrens, Matthijs J. 2015. Five ways to look at Cohen’s kappa. Journal of Psychology & Psychotherapy 5: 1. [CrossRef] West, Colin G. 2023. Advances in apparent conceptual physics reasoning in ChatGPT-4. arXiv arXiv:2303.17012. Yilmaz, Adem. 2021. The effect of technology integration in education on prospective teachers’ critical and creative thinking, multidimensional 21st century skills and academic achievements. Participatory Educational Research 8: 163–99. [CrossRef] Adm. Sci. 2023,13, 157 19 of 19 Zaitsu, Wataru, and Mingzhe Jin. 2023. Distinguishing ChatGPT (-3.5,-4)-generated and human-written papers through Japanese stylometric analysis. arXiv arXiv:2304.05534. Zhai, Xioming. 2022. ChatGPT user experience: Implications for education. SSRN. Available online: https://ssrn.com/abstract=4312418 (accessed on 18 June 2023). [CrossRef] Zhao, Yukun, Liying Xu, Zhen Huang, Kaiping Peng, Martin Seligman, Evelyn Li, and Feng Yu. 2023. AI chatbot responds to emotional cuing. Research Square.Preprint. [CrossRef] Zhong, Lin. 2017. Indicators of digital leadership in the context of K-12 education. Journal of Educational Technology Development and Exchange (JETDE) 10: 27–40. [CrossRef] Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.