Digital Skills in Distance University Students: A Psychometric Analysis Competencias Digitales en Estudiantes Universitarios a Distancia: Un Análisis Psicométrico Luísa Aires1 LE@D, Universidade Aberta (Portugal) https://orcid.org/0000-0001-5649-276X Lúcia Amante LE@D, Universidade Aberta (Portugal) https://orcid.org/0000-0003-3210-7980 Esther Fernández-Márquez Pablo de Olavide University (España) https://orcid.org/0000-0003-0111-2533h Catarina Nunes Universidade Aberta (Portugal) https://orcid.org/0000-0001-9401-2616 Abstract In recent years, research on the digital skills of higher education students has increased significantly. However, many of the existing studies have focused primarily on identifying these skills rather than exploring the factors that influence their development. This study investigates the digital skills of first-year undergraduate students enrolled at an online distance education university, as well as their key determining factors. Employing a quantitative methodology, the students completed a Likert-scale questionnaire as part of an online familiarization module. Based on the findings, the students' digital skills were grouped into the following categories: collaboration and creativity, networking, content sharing, information management, information evaluation, and critical thinking. Additionally, the key influencing factors were identified, including the perceived ease of using ICT for problem-solving, orientation towards learning goals, personal initiative, and self-regulation in the use of ICT and the Internet. The results suggest that both individual and social strategies related to Internet use play a crucial role on the acquisition of digital skills. Future research should explore theoretical frameworks and, especially, refine assessment tools, shifting the focus from simply describing digital competencies to examining the factors that influence and predict their development. Such theoretical and methodological adjustments are expected to significantly reshape training strategies in this field. 1 Autor de correspondencia:
[email protected] Revista Fuentes 2025, 27(2), 149-163 https://doi.org/10.12795/revistafuentes.2025.26469 Recibido: 2024-08-11 Revisado: 2024-09-03 Aceptado: 2025-03-11 First Online: 2025-05-01 Publicación Final: 2025-05-15
REVISTA FUENTES,27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index Resumen La investigación sobre las competencias digitales se ha incrementado los últimos años. Sin embargo, estas investigaciones se han centrado más en identificar estas competencias que en considerar los factores que influyen en su desarrollo. En este estudio se analizan las competencias digitales de un grupo de estudiantes de una universidad de educación a distancia. Optándose por una metodología cuantitativa, se aplicó un cuestionario tipo Likert online a un grupo de estudiantes de nuevo ingreso. Los datos obtenidos han sido analizados con técnicas estadísticas descriptivas e inferenciales. Los resultados demostraron que las competencias digitales mejor valoradas por los estudiantes son: colaboración, creatividad, trabajo en red, intercambio y gestión de información, evaluación de información y pensamiento crítico. Además, se han identificado los siguientes factores clave que determinan el desarrollo de estas competencias: facilidad de uso de las TIC en la resolución de problemas; orientación hacia los objetivos de aprendizaje; iniciativa personal, autorregulación en el uso de TIC/Internet. Se concluye que estrategias individuales, sociales y actitudinales especificas hacia Internet tienen un impacto importante en la adquisición efectiva de estas competencias. En investigaciones futuras, se recomienda profundizar en los marcos teóricos y, especialmente, en los instrumentos de evaluación de estas competencias, ampliando el foco de la descripción de las competencias digitales hacia los factores que condicionan y predicen su desarrollo. Se estima que cambios teóricos y metodológicos de este tipo impulsarán transformaciones significativas en las estrategias formativas en este ámbito. Palabras clave / Keywords Competencia Digital, Internet, Educación a Distancia, Estudiantes, Educación Superior, Instrumento de Evaluación. Digital Skills, Internet, Distance Education, Students, Higher Education, Assessment tool, Learning. 1. Introduction In recent years, there has been a growing body of research on the digital skills of higher education students. However, many of the existing studies have focused primarily on describing the students' existing skills, rather than exploring the underlying factors that influence their development. The literature review highlights a prevailing tendency to address digital competencies as isolated entities, disconnected from the broader contexts and factors that determine them (Hargittai, 2021; Wu et al., 2021; Раkhomova et al., 2023). Apart from theoretical models, effective assessment tools are essential for measuring digital competence. However, despite the existence of various frameworks designed to evaluate digital skills (Schwarz et al., 2024), such as DigComp 2.0 or DigComp 21 (Vuorikari et al, 2016; Carretero et al., 2017), current assessment systems struggle to establish systematic and efficient evaluation methods. This highlights the need to improve measurement approaches (Sillat et al., 2021) and the identification of the key predictor variables of digital proficiency (CabezasGonzález et al, 2022). In light of ongoing digital transformations and the challenges educational communities faced during the Covid-19 pandemic, digital competencies (or lack thereof) and the methods to evaluate them have become a central topic of discussion across all educational levels, including universities (Zhao et al., 2021; Saienko et al., 2022; Monferrato, 2024). This study examines digital skills within the broader framework of 21st-century competencies, focusing not only on students’ digital skills, but also on the factors that influence their unequal development. The literature review conducted emphasized the relevance of the studies by Van Laar et al. (2017, 2020) and Helsper (2019), who offer valuable insights into the key determinants of digital skills and their development. Their findings are particularly relevant for online distance learning universities, which have long recognized the crucial role of digital competencies in promoting student inclusion within virtual campuses, enhancing academic success, and improving overall educational outcomes (Mohammadyari & Singh, 2015; Pham Tra & Dau Thi Kim, 2024). As Fernández-Gómez et al. (2021) pointed out, distance education models favor the adoption of more collaborative methodologies, which foster digital literacy. Therefore, as online distance universities prioritize
REVISTA FUENTES, 27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index the inclusion of digital skills in their pedagogical models (Pereira et al., 2007; Gros Salvat et al., 2009), they actively contribute to advancing research in this field (Quintas-Mendes et al., 2019; Sezgin & Fırat, 2024). This study is part of a research project that aims to analyze the digital skills of newly enrolled university students. The first phase focuses on evaluating the digital skills of first-year students and the factors that determine their development. A questionnaire was administered to the students during a mandatory online familiarization module (OFM) completed prior to the start of their degree programs. The results highlight the need for changes both in the methodological approaches and assessment tools currently employed to evaluate digital skills. 1.1 Digital Skills in the 21st Century Many studies associate the digital divide and digital skills to political, social, and educational agendas (EU, 2021), yet their role in key educational priorities in the 21st century is not evident (Voogt & Roblin, 2012; Soule & Warrick, 2015; van Deursen et al., 2017; Sezgin & Fırat, 2024). This gap emphasizes the need to define or clarify the concepts of digital skills and 21st-century skills. There are numerous definitions of digital skills depending on the framework used to categorize them. In general, digital skills refer to the ability to use digital technologies for work, leisure, and communication (From, 2017). However, a more detailed and operational definition, in line with Ferrari (2012), describes digital skills as the set of abilities required to perform tasks, solve problems, and create and share knowledge in a critical, ethical, and autonomous manner, across the domains of work, social participation, leisure, and personal development (Koch & Fehlmann, 2025). Current investigations on the topic has found that digital inclusion and digital skills also foster multiple experiences of internet use (Hargittai et al., 2018; Helsper, 2019; van Laar et al., 2020). Twenty-first-century skills encompass foundational personal, social, and professional abilities that empower individuals to engage in various dimensions of the knowledge society. These competencies are transversal and multidimensional, combining knowledge, attitudes, and values, as well as fostering higher-order thinking to solve complex problems, and deal with uncertain situations (OECD, 2018). Given that these constructs are interconnected, van Deursen et al. (2017) and van Laar et al. (2020) have proposed merging 21st-century skills and digital skills into a single framework, termed 21st-century digital skills (see Table 1). This proposal aims to enhance the role of digital competencies in education, work, and social participation (Helsper et al., 2016), placing individuals, their contexts, communities, and broader relationships at the core of digital education. Table 1 Twenty-first century digital skills (Source. van Laar et al., 2018, 2019b, 2020). Theoretical perspectives on digital literacy and digital skills have developed significantly and been consolidated over the past five years, but much still needs to be done regarding the development of assessment tools. There remains a critical need for tools that go beyond merely descriptive evaluations, capable of offering predictive and comprehensive assessments that account for contextual and individual differences. They should encompass a broader range of competencies, including critical thinking and other higher-order cognitive abilities, while also actively engaging students, teachers, and community members. Reliable assessment of digital skills and soft skills are essential, rooted in robust reliability methodologies that do not compromise students' readiness to face the diverse demands of daily life (Peláez-Sánchez et al., 2024). 1.2 Determinants of 21st century Digital Skills Research on digital skills has often relied on unidimensional approaches that fail to consider factors that influence digital skill acquisition. Yet, it is critically important not only to understand how individuals acquire 21st Century Digital Skills Information Management Communication Collaboration Creativity Critical Thinking Problem Solving
REVISTA FUENTES,27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index these digital skills but also to identify the factors that shape their development (van Deursen et al., 2020; Casillas-Martín et al, 2020; Cabezas-González et al, 2022). However, recent studies (van Laar et al.,2019a, 2020; Cabezas-González et al, 2022) have introduced a multidimensional approach to 21st-century Internet skills, highlighting the factors that influence their development. This proposal departs from the instrumental perspective that has traditionally dominated training in this domain, positioning the development of digital competences as a central priority within the context of lifelong learning. Our study is in line with this perspective by focusing on the social, individual, and motivational determinants that are considered relevant to examine individual differences. Factors such as ease of use, perceptions of ICT, self-directed learning, performance, and socio-contextual variables, including support for internet use, are proposed as potential predictors that can account for variations in digital skill acquisition (Table 2). Table 2 Determinants of 21st century Digital Skills Determinants ICT Attitude Perceived ease of use ICT self-regulation Self-directed learning Performance goal orientation Avoidance learning goal orientation Personal Initiative (Source: van Laar et al. 2019b; 2020) 1.3 Higher Education Students and Digital Skills When students enter Higher Education and begin their undergraduate studies, they are required to demonstrate a broad range of skills to ensure their participation in new learning environments. Among these, digital skills are critical for students to effectively engage in diverse campus contexts and are especially crucial in online campuses (Guttierrez et al., 2009, Almerich et al., 2019; Perera & Gardner, 2018; EU, 2017; 2021; Sezgin & Fırat, 2024; Smith et al., 2024). In online settings, students will encounter learning experiences that require more than basic technological usage. The online learning skills they must develop encompass pedagogical, individual and social competencies tailored to learning contexts. Key skills for successful online learning include independent learning, time management, online communication, interpersonal interactions, and wellbeing (Pereira et al., 2003; Koch & Fehlmann, 2025). Therefore, universities must prioritize the development of these skills in their educational goals and strategies (Palmeiro, Pereda & Aires, 2019; De Pablos, 2018; Saienko, Kurysh, & Siliutina, 2022). To better understand the students’ needs in the domain of digital competencies, this study pursued the following goals: G1. To analyze the digital skills of first-year students enrolled at an online distance education university. G2. To identify the key factors influencing the development of students’ digital skills. G3. To optimize an assessment tool to evaluate digital skills. 2. Methods This quantitative study analyzed the perceptions of incoming undergraduate students at a distance education university regarding their digital skills, aiming to answer the following questions: Q1: What are the students’ perspectives on their digital skills as they begin their undergraduate studies at an online distance university? Q2: What social and individual determinants influence the students’ digital skills? 2.1 Research Context The students who participated in this study were enrolled at an online distance education university, and were required to attend a mandatory online familiarization module (OFM). Specifically designed for newly
REVISTA FUENTES, 27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index admitted students, the OFM is in accordance with the university's pedagogical model (Pereira et al., 2007; Mendes et al., 2019; Aretio, 2014, 2021; Willems et al., 2019). The OFM serves to familiarize new students with the university’s digital learning environment and teach them "how to be an online student". It offers each student the opportunity to evaluate their readiness to adapt to this learning system. 2.2 Sample The study sample consisted of 327 students about to start their degrees in September 2020. Newly admitted undergraduate students were selected because it was crucial to understand their baseline digital skills, particularly their Internet skills, upon entering university. The insights obtained help to identify the students' potential and their needs in the area of digital literacy. Prior informed consent was obtained from all students involved in the study. 2.3 Data Collection Instrument Data were collected using a Likert-scale questionnaire adapted from previous studies (van Deursen et al., 2017; van Laar et al., 2018; van Laar et al., 2019a). The questionnaire was designed to assess a broad range of digital skills, such as information management, communication, collaboration, critical thinking, creativity, and problem-solving. Additionally, it incorporated constructs related to the students' personal, motivational, and social perceptions, including perceived ease of use, self-regulation of ICT use, autonomous learning, orientation towards learning goals, avoidance of learning-goal orientation, and personal initiative (van Laar et al., 2019b) (Table 4). This questionnaire included a total of 91 items: five items on descriptive data (gender, age, education level, academic status, and professional activity), and 86 items related to indicators that cover 21st-century digital skills and their determining factors, using a 5-point Likert scale (“Never,” “Rarely,” “Sometimes,” “Often,” and “Almost always”) (van Laar et al., 2019a, 2019b). For the purposes of this study, we focused on the following three sections of the questionnaire: (i) sociodemographic data, (ii) Internet use within the OFM, and (iii) strategies for online discussion and critical thinking. This research project complied with the ethical standards for educational research. All research procedures and questionnaires were reviewed and approved by the Ethics Committee of LE@D (Laboratório de Educação e Ensino a Distância, Universidade Aberta) in September 2019. 2.4 Data Analysis Procedures All the data analyses were conducted using IBM SPSS (version 25) and R software. Statistical significance was set at P < 0.05. The data analysis was performed in two phases. In the first phase, a descriptive analysis was performed on the results of the questionnaire’s two main sections: (i) Uses of the Internet within the OFM; and (ii) Online Discussion Strategies. The first section comprised 74 items aimed assessing the students’ self-perceived usage of the Internet within the scope of the course. In this section, students self-assessed their digital competencies in information management, communication, collaboration, creativity, and problem solving, as well as the determinants of digital competencies, such as perceived ease of use, self-regulation of ICT use, autonomous learning, learning goal orientation, performance goal orientation, avoidance of learning goal orientation, and personal initiative (van Laar et al., 2019a, 2019b). The second section consisted of 12 items focusing on online discussions in the OFM’s forums. In this section, students self-assessed the resources they mobilized in online discussions, such as critical thinking (Appendices 1 and 2). A descriptive data analysis was conducted for each section. To verify internal consistency, Cronbach's alpha was calculated (Cronbach, 1984). Normality was assessed using skewness and kurtosis. The suitability of applying factor analysis was tested using the Kaiser-Meyer-Olkin (KMO) coefficient and Bartlett’s test of sphericity (Hair et al., 2014). The second phase of the data analysis consisted of an Exploratory Factor Analysis using Principal Components and the Varimax rotation with Kaiser normalization. The main goal was to obtain the minimum
REVISTA FUENTES,27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index number of components that explained the greatest amount of total variance observed, and to identify the factors that students considered essential in each domain. In addition, a Confirmatory Factor Analysis (with unweighted least-squares estimation) was performed to verify whether the data confirmed the structural model obtained by the Exploratory Factor Analysis. In view of the results obtained, we believe that this is a reliable and trustworthy assessment tool, in line with goal 3 (G3) of this research. 3. Results 3.1 Sample Description A total of 800 access instances to the questionnaire link were recorded, resulting in 327 completed questionnaires, representing a 70% response rate. Table 3 presents the demographic data of the students who participated in the study. Table 3 Student demographics (authors’ elaboration) N (%) Gender Female 198 (60,6%) Male 129 (39,4%) Age (years) 18-30 61 (18,7%) 31-45 171 (52,4%) 46 - 60 95 (29,1%) Education Basic 6 (1,8%) Secondary 250 (76,5%) Undergraduate 55(16,8%) Master 14 (4,3%) PhD 2 (0,6%) Academic Situation Full-Time Student 24 (7,3%) Part-Time Student 303 (92,7%) Employment Situation Employed 288 (88,1%) Unemployed 36 (11%) Never had a job 3 (0,9%) The sample was mostly composed of female students, aged predominantly 31 to 45 years old. Secondary education is the most common level of completed education, and a significant portion of the sample were student workers (Table 3). 3.2 Data Analysis For the first section on “Uses of the Internet within the OFM,” Cronbach's alpha was 0.962, while in the second section “Online Discussion Strategies”, the Cronbach's alpha was 0.946, reflecting a high degree of internal consistency in both sections. All items were assessed for normality. Skewness was on average -0.114 ± 0.242 and kurtosis averaged 2.954 ± 0.435, indicating, therefore, that no transformations were required. The exploratory factor analysis conducted on the section “Uses of the Internet within the OFM” identified 11 factors, based on the Kraiser criteria, explaining 68.68% of the variance. The Kaiser-Meyer-Olkin (KMO) coefficient was 0.930 and the result obtained from Bartlett’s test of sphericity was P < 0.001. All items had factor loadings of greater than 0.5, and the component matrix exhibited a well-defined structure. Fifteen items with loadings below 0.5 were removed (Aires, Pereda, Palmeiro, Amante. & Nunes, 2021).
REVISTA FUENTES, 27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index The confirmatory factor analysis confirmed that all the paths in the model were statistically significant. Fit indices for residuals indicated a good model-data fit: Root Mean Square Error of Approximation (RMSEA) = 0.028; Standardized Root Mean Square Residual (SRMR)=0.066. The Comparative Fit Index (CFI) was 0.991 and the Tucker-Lewis Index (TLI) was 0.990, indicating a good model-data fit. The Cronbach's alpha for the remaining 59 items was 0.952, indicating a high degree of internal consistency. In the exploratory factor analysis of the “Online Discussion Strategies” section, the Scree Plot component extraction method was applied because there were only 12 items (Hair, Black, Babin, & Anderson, 2014). The KMO coefficient was 0.942 and a P < 0.001 was obtained in Bartlett’s test of sphericity. All items had factor loadings greater than 0.5, and the component matrix exhibited a well-defined structure. Two factors were identified, which explained 71.95% of the total variance. The confirmatory factor analysis showed that all the paths in the model were statistically significant. The fit indices for residuals indicate a good model-data fit: RMSEA < 0.001; SRMR=0.036. The CFI and TLI were both greater than 0.999, indicating a good model-data fit. The Cronbach's alpha remains the same at 0.946, since no items were removed. Overall, thirteen factors were identified in the exploratory factor analysis. These factors were grouped into two broad categories: seven factors related to 21st-century digital skills, and six factors associated with the determinants of digital skills (Table 4, Figures 2 and 3). Considering these two broad categories, Cronbach's alpha values were 0.956 and 0.914, respectively, reflecting a high degree of internal consistency. When examining the factors individually, Cronbach's alpha ranged from 0.894 to 0.940. Figure 1: Digital Internet Skills: Categories with the highest and lowest levels (authors’ elaboration). Figure 1 presents a comparative overview of the digital skill categories, highlighting the areas in which students obtained the highest and lowest scores. The highest-scoring categories include Information Management, Communication: Expressiveness, Critical Thinking, Perceived Ease of Use, and Autonomous Learning. These results suggest that students possess strong competencies in managing information, expressing themselves in digital communication, thinking critically, and learning independently, as well as perceiving digital tools as easy to use. The strategies adopted in online distance interactions, particularly critical thinking, yielded consistently high scores. All items in this category have a median of 4, except for the item "Use the Internet to justify [my] choices", which had a median is 3. In contrast, the lowest-scoring categories include Communication: Networking, Communication: Knowledge Sharing, and Avoidance Learning Goal Orientation. 4. Discussion The sample was predominantly composed of female students, the majority of which falling within the 31-45 age group. Most participants had completed secondary education, and a significant portion were student workers, highlighting the crucial role of distance education universities in lifelong learning. Highest Scores Information Management Communication: Expressiveness Critical Thinking Perceived Ease of Use Autonomous Learning Avoidance learning goal orientation Communication: Networking Communication: Knowledge Sharing Lowest Scores
REVISTA FUENTES,27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index The descriptive statistical analysis highlights that students are proficient in handling digital information effectively, articulating ideas clearly in digital interactions, applying critical thinking skills, and independently navigating digital learning environments with confidence in their usability. However, the findings also suggest that students face challenges in establishing academic or professional networks, actively engaging in digital knowledge-sharing practices, and maintaining a proactive learning mindset in online environments. These findings are in line with previous research indicating user proficiency in accessing information and digital communication. However, while these insights are valuable, further exploration of the collected data through factor analysis is essential to obtain more substantiated answers to the research goals and questions. The exploratory factor analysis identified thirteen factors. In the “Uses of the Internet within the OFM”, eleven factors were identified based on Kraiser criteria, explaining 68.68% of the variance. All items had factor loadings greater than 0.5, and the component matrix had a well-defined structure. Fifteen items with loadings below0.5 were removed (Aires, Pereda, Palmeiro, Amante & Nunes, 2021). The confirmatory factor analysis showed that all the model’s paths were statistically significant. Fit indices for residuals indicated a good modeldata fit: Root Mean Square Error of Approximation (RMSEA) = 0.028; Standardized Root Mean Square Residual (SRMR)=0.066. The Comparative Fit Index (CFI) was 0.991 and the Tucker-Lewis Index (TLI) was 0.990, indicating a good model-data fit. The Cronbach's alpha for the remaining 59 items was 0.952, indicating a high degree of internal consistency. In the exploratory factor analysis of the “Online discussion strategies” section, the Scree Plot component extraction method was applied because there were only 12 items (Hair, Black, Babin, & Anderson, 2014). All items had factor loadings greater than 0.5, and the component matrix had a well-defined structure. Two factors were identified, which explained 71.95% of the total variance. The confirmatory factor analysis showed that all the model’s paths were statistically significant. Fit indices for residuals indicate a good model-data fit: RMSEA < 0.001; SRMR=0.036. The CFI and TLI were both > 0.999, indicating a good model-data fit. Thirteen factors were clustered into two broad categories: seven factors related to 21st-century digital skills, and six factors associated with the determinants of digital skills (Table 4). In the following sections, we will discuss the results of each of these dimensions, in line with the main goals and questions of this study. Table 4 Resulting factors clustered in the 21st century Digital Skills and Determinants of 21st century Digital Skills (authors’ elaboration) 21st Century Digital Skills Determinants of 21st Century Digital Skills Factors Categories Factors Categories Factor 1 Collaboration and Creativity Factor 2 Ease of Use of ICT and Problem Solving Factor 3 Communication: networking Factor 4 Orientation towards learning objectives Factor 5 Communication: content sharing Factor 6 Personal Initiative Factor 9 Information management Factor 7 Autoregulation in ICT/ Internet Factor 11 Information evaluation Factor 8 Avoidance learning goal orientation Factor 12 Critical Thinking: Individual perspective Factor 10 Sharing Factor 13 Critical Thinking: Social perspective Table 4 presents the students' digital competences in the left-hand column and the key determining factors for these competences in the right-hand column. 4.1 Twenty-first century digital skills The results of the Factor Analysis related to 21st-century digital skills in learning contexts are presented in Figure 2. These results show the factors, items in each factor, respective loadings, and percentage of variance explained by each factor.
REVISTA FUENTES, 27(2), 149-163 www.revistascientificas.us.es/index.php/fuentes/index Figure 2: 21st Century Digital Skills identified through Factor Analysis (authors’ elaboration). In relation to the students' perceptions on their digital skills at the start of their undergraduate studies at an online distance university—addressing research question 1 (Q1)—the analysis grouped the competencies into seven factors (Figure 2). In turn, these seven factors highlight three key dimensions: • Factor 1 - Collaboration and Creativity: This factor includes joint activities and creative processes within the same context. It is the most significant, explaining 29.79% of the variance. • Factors 3, 5, 9, and 11 - Communication and Information: These factors have high loadings and emphasize networking communication, content-sharing competences, as well as two other fundamental competencies in the learning process: managing and accessing information. • Factors 12 and 13 - Critical Thinking: These factors highlight a key 21st-century digital competency in both social and individual dimensions. However, the individual perspective explains 64.08% of the variance, indicating a strong emphasis on personal information analysis and problem-solving competencies. Based on these dimensions, factors, and categories identified through the factor analysis, the data provides a provisional mapping of essential digital skills for distance higher education in the 21st century. Creativity in digital learning environments is reinforced through collaboration (Dwyer et al., 2014). Additionally, the vast amount of information available online requires students to develop specific skills in managing and evaluating information (Saienko et al., 2022; Sillat & Laanpere, 2021).