Assessing Undergraduate Engineering Students' Professional Skills

Research Paper Recommended citation: Carrillo-Fernandez, A. P., Dyehouse, M. A., Holloway, E. A., & Douglas, K. (2025). Assessing Undergraduate Engineering Students’ Professional Skills. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631853. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. ASSESSING UNDERGRADUATE ENGINEERING STUDENTS’ PROFESSIONAL SKILLS A.P. Carrillo-Fernandez a, M. A. Dyehouse b, E. A. Holloway c, K. A. Douglas d,1 a Purdue University, West Lafayette, US, 0009-0002-0905-6986 b Purdue University, West Lafayette, US, 0000-0002-7219-4137 c Purdue University, West Lafayette, US, 0000-0002-0343-1709 d Purdue University, West Lafayette, US, 0000-0002-2693-5272 Conference Key Areas: (1) Engineering skills, professional skills, and transversal skills. (2) Improving higher engineering education through researching engineering education. Keywords: Professional Skills opportunities, Self-assessment, Undergraduate, Microelectronics Workforce development program ABSTRACT This empirical full research paper looks to gain insight into how often engineering undergraduate students have opportunities to practice professional skills in their undergraduate experiences, depending on the type of program in research institutions. ABET, employers, and many national reports have emphasized the importance of engineering students developing professional skills. Still, these skills are difficult to assess due to their developmental nature, as students need opportunities to practice professional skills in and out of the classroom. A Professional Skills Opportunities (PSO) survey using a 7-point Likert scale covering Shared Leadership, Communication, Problem Solving, and Business and Management was administered to engineering students (n = 536) from various research institutions and a microelectronics workforce development program. The students’ responses were analysed to gain insights into the types of professional skills they reported to practice and to compare the different experiences across these programs. Findings indicate that each group of students from research institutions and a workforce development program got ample opportunities to practice professional skills such as Shared Leadership, Communication, and Problem-Solving. Additionally, on average, women showed higher values than men in all categories. These insights are essential as they allow educators and administrators to adjust courses and co-curricular activities to give undergraduate students more opportunities to practice professional skills. Future work can focus on conducting qualitative interviews with students to understand their professional skill formation. 1K A. Douglas [email protected] 1 INTRODUCTION: LITERATURE AND BACKGROUND Higher education institutions and stakeholders have increasingly recognized the importance of professional skills for undergraduate engineering, as seen through integrating these skills into the curriculum. ABET-accredited engineering programs must include these competencies in their curriculum (“ABET,” 2024). Furthermore, ASEE (2024) includes a framework of a “T-shaped professional” whose skills include technical expertise and interpersonal and intrapersonal competence. Despite recognition of the importance of these skills to the field of engineering, only recently has an instrument with evidence of validity capable of assessing the opportunities that students have to practice critical professional skills in their programs of study been developed (Li et al., 2022). In this study, we address the following research questions: (1) What professional skills did engineering undergraduate students have more and fewer opportunities to practice? (2) What are the differences in opportunities to practice professional skills between Research Universities and a Microelectronics Workforce Development Program? (3) What kinds of professional skills do engineering undergraduate students have more and fewer opportunities to practice by gender? 1.1 Opportunities for professional skills in undergraduate engineering There are many different professional skills undergraduate engineering students can engage with during their programs, such as leadership, effective teamwork, and communication. Professional development skills for undergraduate engineering students are essential for a sound transition and success in their workplace (Cox et al., 2012; Hartmann et al., 2016) but also are potential markers for persistence (Lent et al., 2016; Pierrakos et al., 2009) within their academic programs. Workforce development programs can provide additional opportunities for students to practice professional skills through curricular, co-curricular, and hands-on experiences and activities. These experiences are offered intentionally to students to foster professional experiences to enhance career preparation and persistence in a specific field or area, including specialized coursework (Tait & McBride, 2001), internships (SMART Scholarship-for-Service Program, 2025), boot camps (Vanderbilt Boot Camps Boost Workforce Talent to Meet Influx of Tech Jobs, 2021), research experiences (U.S. Department of Homeland Security Education Programs, 2025), apprenticeships (Kuehn et al., 2022), and mentorship (National Academies of Sciences, 2019). The Workforce Development Program specific to this study incorporates internships, research experiences, seminars, networking opportunities, and mentorship to create a community of practice, described as a shared domain of interest, a community of joint engagement, and a shared repertoire of practices (Wegner & Nückles, 2015). Notably, ABET’s (2024-25) criteria for undergraduate engineering student outcomes encompass various facets of professional skills development, further emphasizing the importance of practical experience in shaping well-rounded engineers. These include effective communication, teamwork, and ethical and professional responsibility recognition (ABET, 2024). Although all ABET-accredited institutions must provide students with opportunities to learn and/or practice these professional skills, how they do this, and thus, their potential effectiveness, can differ. Therefore, assessing those opportunities is critical for determining program effectiveness and areas for improvement. Most measures of professional skills, including engineering and STEM fields, have focused on assessing a particular skill rather than evaluating a broad range of skills (Li et al., 2022). However, while various assessments for professional skills have been developed (Ahn et al., 2014; Wageman et al., 2005), no instruments have measured students’ opportunities to practice professional skills until recently. 1.2 Theoretical Framework This research is framed by Dall’Alba’s ontological “ways of being” framework (Dall’Alba, 2009), which focuses not only on skills and technical knowledge as critical parts of being a professional but also on becoming one. Dall’Alba presents this process of becoming a professional as a transformation of the self by engaging and absorbing the culture of the profession in question (Dall’Alba, 2009). In engineering, this would translate into looking at students’ opportunities to practice processes and engage with dynamics specific to the engineering profession, such as teamwork, leadership, networking, etc. Previous assessment research based on Dall’Alba’s framework on professional skills opportunities doctoral students received in their engineering programs showed strong evidence of validity (Li et al., 2022) when measuring those opportunities. Therefore, evaluating students’ opportunities to engage with professional skills would allow us to know more about how students are becoming professional engineers. 2 METHODOLOGY 2.1 Overview: Survey Constructs In this study, we utilized the Professional Skills Opportunities (PSO) assessment instrument, validated for use with undergraduate engineering students in a large, multi-institutional study (Li et al., 2022).The PSO has demonstrated strong content and construct validity and fairness, having undergone think-aloud, exploratory factor analysis, confirmatory factor analysis, and item difficulty and discrimination analysis (Li et al., 2022). The PSO instrument consists of 21 items and measures undergraduate students’ opportunities to practice professional skills across four factors: 1) Shared Leadership (7 items), 2) Communication (4 items), 3) ProblemSolving (5 items), and 4) Business and Management Principles (5 items)(Gentry et al., 2024). The Shared Leadership dimension assessed respondents' opportunities to develop a confident attitude for fostering a supportive and collaborative group mindset. The Communication dimension assessed respondents' adaptability in oral and written communication with various audiences and stakeholders. The problemsolving dimension evaluated respondents' opportunities to identify project criteria and constraints and generate and assess the feasibility of final ideas. The Business and Management Principles dimension evaluated respondents' opportunities to handle managerial tasks, legal perspectives, and stakeholder needs. Each item for each factor was scored on a Likert scale from 1 to 7, wherein 1 = “Not at all” and 7 = “Very Frequently.” We averaged items associated with each construct, resulting in an individual score between 1 and 7, aligned with the original Likert scale. Thus, a score of 7 suggests that the respondent is highly performing for the evaluated factor. 2.2 Data Collection The data from this study comes from two different datasets: 1) Research-focused universities from different programs, and 2) a Microelectronics Workforce Development Program (WFD) across different universities and programs. The data from the first group was collected from 13 different higher education institutions in the U.S. at the end of the Spring 2022 Semester as part of a prior study. Participants received an Amazon gift card as an incentive to take the Professional Skills Opportunities (PSO) survey. For the current study, we focused only on those research-focused institutions from that large dataset, namely, two out of 13. The second group’s data was collected during the Fall 2023 Semester from 14 different research universities associated with a Microelectronics WFD program, which also offered participants who took the PSO survey an Amazon gift card. This study includes 460 participants from the research universities dataset and 76 from a Microelectronics Workforce Development (WFD) program dataset. Gender distribution showed 43.7% women and 54.1% men at research universities, compared to 23.7% women and 73.7% men in the WFD program. Racial/ethnic diversity was higher in the WFD program, with 23.7% Asian and 11.8% identifying as two or more races, compared to 15.4% and 5.0%, respectively, at research universities. Most participants were in their first to fourth years of undergraduate engineering programs, with a higher percentage of fourth-year students in the WFD program (52.6%). The most common majors were Mechanical Engineering and Computer Engineering at research universities and Electrical Engineering in the WFD program. 2.3 Data Analysis Before analysis, the data were cleaned and pre-processed. Survey responses were filtered using two criteria: completion rate (removing responses below 50%) and appropriate responses to filter item questions. Additionally, responses were excluded if participants did not clearly state they were undergraduate students. The Research Universities dataset has 460 responses (across two universities and more than 15 engineering and science programs), and the microelectronics WFD dataset has 76 (across 14 universities and 11 engineering and science programs). We assessed the normality assumptions by scrutinizing histograms of the constructs and computing the Shapiro-Wilk coefficient for each construct using SPSS. This statistical measure was employed to discern whether the data approximated a non-normal distribution. The Null hypothesis was that there was no statistical difference from a normal distribution. Upon comparative analysis across institutions (i.e., the two independent datasets), utilizing a 95% confidence interval and Shapiro-Wilks test, it was observed that responses about Communication and Problem-Solving factors exhibited nonnormal distributions across all groups (p<.001). Additionally, we observed in this test that the Shared Leadership (p=.055) and Business and Management (p = .762) factors displayed a normal distribution for the Microelectronics WFD program. Because some PSO constructs did not show normality across datasets, we used a non-parametric statistical test to compare 1) which PSO constructs scored higher and lower within each dataset, 2) the PSO constructs across datasets, and 3) the PSO factors that differed by gender across datasets. 2.3.1 Comparing PSO within each dataset To answer the first research question, we computed construct scores for each dataset by taking the average of items on each one of the PSO constructs as provided in the validity evidence for the PSO instrument (Li et al., 2022).Then, we tested the differences between PSO constructs. Due to a non-normal distribution of the data, the Friedman and post-hoc tests were applied using SPSS software to compare PSO constructs within each dataset. For these tests, the null hypothesis was that PSO construct distributions are the same (e.g., opportunities to practice shared leadership and communication were not significantly different between engineering undergraduate students), and the alternative hypothesis was that PSO construct distributions are distinct. 2.3.2 Comparing PSO constructs by datasets. To answer the second research question, we computed construct scores by taking the average of items on each of the PSO constructs as provided in the validity evidence for the instrument (Li et al., 2022). Next, we analysed the constructs’ mean and standard deviation based on differences in institutional settings. Then, we tested the differences between the datasets of the research institutions and the datasets of the microelectronics WDF program institutions for each construct. The Shapiro-Wilk test was implemented using SPSS software. Due to the non-normal nature of the responses to many constructs, we used the Mann-Whitney U Test to compare the two independent samples. For these tests, the null hypothesis was that the distribution of PSO constructs (i.e., Shared Leadership, Communication, ProblemSolving, and Business and Management) is the same across datasets, and the alternative hypothesis was that the distribution of PSO constructs is not the same across groups. 2.3.3 Comparing PSO constructs by gender. To answer the third research question, we computed PSO construct scores as provided in the validity evidence for the PSO instrument (Li et al., 2022). Then, we analysed the constructs’ mean and standard deviation based on gender. The Shapiro-Wilk test was implemented using SPSS software. Because of the nonnormal nature of the responses on gender categories and the small size of the nonbinary category (n = 9), we performed the Mann-Whitney U test only for women and men in gender categories. For these tests, the null hypothesis was that the distribution of PSO constructs (i.e., Shared Leadership, Communication, ProblemSolving, and Business and Management) is the same across categories of gender, and the alternative hypothesis was that the distribution of PSO constructs is not the same across gender categories. 3 RESULTS 3.1 Descriptive Statistics by Dataset Across both dataset groups, students demonstrated the highest responses in Communication (M=5.73, SD = .1.02 for Research Universities; M=5.70, SD = .87 for Microelectronics WFD) compared to other constructs. Problem-solving consistently scored the second-highest across both groups (M = 5.52, SD = 1.09 for Research Universities; M = 5.57, SD = .94 for Microelectronics WFD). In contrast, Business and Management received the lowest scores across groups and constructs (M = 4.12, SD = 1.43 for Research Universities; M = 4.27, SD = 1.15 for Microelectronics WFD). 3.2 Descriptive Statistics by Gender For this test, we combined the participants from the workforce development program and the Research Universities populations to generate the women and men groups for this section. Across all PSO constructs, women (n = 219) respondents exhibited higher overall mean (M) scores than men (n = 305): Shared Leadership (M = 5.49, SD = 1.02 for women; M = 5.27, SD = 1.03 for men), Communication (M = 5.89, SD = 1.00 for women; M = 5.63, SD = .94 for men), Problem-Solving (M = 5.55, SD = 1.11 for women; M = 5.51, SD = 1.01 for men), and Business and Management (M =4.21, SD = 1.42 for women; M = 4.09, SD = 1.37 for men). Non-binary (n = 7) respondents had higher overall mean scores than women and men in every PSO construct: Shared Leadership (M= 5.86, SD = .87 for Non-binary), Communication (M= 5.86, SD = .92 for Non-binary), Problem-Solving (M= 5.86, SD = 1.14 for Nonbinary), and Business and Management (M= 4.37, SD = 1.60 for Non-binary). 3.3 Significance Testing 3.3.1 Friedman’s two-way ANOVA by ranks In this section, we performed two Friedman tests to compare PSO constructs within the same dataset. Participants from both research universities and the Microelectronics WFD Program ranked communication and problem-solving as the most significant competencies, showing strong agreement. Microelectronics WFD participants slightly more valued shared leadership, while business and management were consistently ranked lowest by both groups. Figure 1 presents the results of this test. The null hypothesis of Friedman’s test is rejected (i.e., the distribution among factors within the dataset is the same) in both data sets (p < .001). Figure 1. Mean Rank Student Scores for Each PSO Factor by Data Set We then performed pairwise comparisons of group ranks between PSO constructs. Each row in Table 2 tests the null hypothesis that the Sample 1 and Sample 2 distributions are the same: 1) For the Research University dataset, all PSO factors reject the Friedman null hypothesis. Therefore, all PSO factors for this dataset show a statistically significant difference among PSO factors; 2) for the Microelectronics WFD dataset, only the Business and Management Principles factors compared with each of the other factors (i.e., Shared Leadership, Communication, and ProblemSolving) show a statistically significant difference among PSO factors. Table 2. Friedman’s Pairwise Comparisons of PSO factors within datasets Test Statistic (SE) Factor 1 vs Factor 2 Research Universities MWFD Business and Management vs Shared Leadership 1.267 (.085) (*) 1.533 (.209 (*) Business and Management vs Problem Solving 1.562 (.085) (*) 1.678 (.209) (*) Business and Management vs Communication 1.862 (.085) (*) 1.895 (.209) (*) Shared Leadership vs Problem Solving -0.295 (.085) (*) -0.145 (.209) Shared Leadership vs Communication -0.595 (.085) (*) -0.362 (.209) Problem Solving vs Communication 0.300 (.085) (*) 0.217 (.209) Note : (*) Significance at p < 0.001 3.3.2 Mann-Whitney U test comparing PSO across datasets. We conducted four Mann-Whitney U tests to compare professional skills opportunities between both datasets. The results showed no statistically significant differences between the two groups in shared leadership, communication, problemsolving, and business and management factors. The mean ranks for both datasets were similar across all constructs, with p-values indicating no significant differences. 3.3.3 Mann-Whitney U test comparing PSO across gender. We performed four Mann-Whitney U tests to evaluate whether PSO factors differed by gender. These tests allowed us to establish the difference, if any, between two different groups, women (n = 219) and men (n = 305), across the different PSO factors. The results indicated a significant difference between Shared Leadership between women and men (z= -2.657, p < .008), wherein women undergraduate engineering students exhibited higher scores in Shared Leadership compared to men. Communication also significantly differed (z = -3.827, p < 0.001). There was no significant difference between men and women on the Business and Management factor (z= -1.019, p < 0.066) and Problem-Solving (z = -0.939, p < .347). Table 3 shows each group's mean ranks (Mranks) for each factor and summarizes these results. Table 3. Mann Whitney U Test Results (women, n = 219 and men n = 305) Mranks z r (effect Size) Women Men Shared Leadership 283.22 247.62 -2.657 .116 Communication 292.24 241.14 -3.827 .167 (*) Problem-Solving 269.82 257.25 -0.939 .041 Business and Management 270.45 256.79 -1.019 .045 Note: (*) Significance at p < 0.001 4 DISCUSSION AND CONCLUSIONS In this section, we contextualize our results by identifying (1) differences among PSO constructs by dataset and (2) differences across PSO constructs based on gender. 4.1 Differences across the PSO construct and datasets All PSO constructs showed statistically significant differences for the Research University dataset. In contrast, only the business and management principles construct differed from the rest of the PSO constructs in the Microelectronics WFD dataset. Despite this, both datasets indicate that universities provide their students with valuable professional skill opportunities. Students’ scores on the Shared Leadership, Communication, and Problem-solving constructs were above 5.35 out of 7. These findings are expected as more and more engineering programs have developed content and project-based learning (PBL) opportunities in their curriculums in which students are asked to work in long-term teams to solve a problem by identifying actual client needs, keeping in consideration constraints while presenting the findings to different kinds of audiences (e.g., manager of the team, finance department, clients). These findings also align with the ASEE’s (2024) expectations for a framework of a “T-shaped professional” in engineering that includes technical skills and more universal skills such as communication, teamwork, and problem-solving. However, the Business and Management construct showed values between 4.12 and 4.27 out of 7, suggesting that engineering and science students need more opportunities to develop these professional skills. Within the Tshaped framework, 'business acumen' (Bierema, 2019, p. 68) is another crucial skill enhancing professional competence. The individual items for this construct (Gentry et al., 2024) ask about specific aspects of business and management, such as “anticipate possible future stakeholder needs when working on a project,” that might be out of the scope of the typical engineering design project. Thus, these findings indicate that engineering undergraduate students have more opportunities to practice Shared Leadership, Communication, and Problem-solving and fewer opportunities to practice professional skills relating to Business and Management. 4.2 Differences across PSO constructs based on gender Due to the tiny sample size of the non-binary population (n = 7), we focused our analysis on women (n = 219) and men (n = 305) undergraduate engineering students. Traditionally, women have been underrepresented in engineering (Morganson et al., 2010). Our findings show that differences existed, but women showed significantly more opportunities in the Shared Leadership and Communication constructs. The literature on gender differences in co-curricular and extracurricular participation shows that women tend to participate more in these activities than men. Wolniak, Chen-Bendle, and Tackett (2023) found that gender differences existed in college students’ participation in co-curricular and employment activities and employment, with women having a higher proportion of those experiences (Wolniak et al., 2023). These findings align with the present study’s finding that women are showing more opportunities to practice professional skills in the areas of Shared Leadership and Communication because opportunities to practice these skills are likely to be encountered in internships and employment situations. Similarly, Chachra, Chen, Kilgore, and Sheppard (2009) found that undergraduate women reported significantly greater participation in engineeringrelated extracurricular activities than undergraduate men and tended to take on more leadership roles than men (Chachra et al., 2009). Furthermore, an extensive study of over 4,000 U.S. students in engineering programs found that women who participated in both non-engineering clubs or activities, as well as engineering clubs specifically for women or underrepresented minorities, reported higher communication skills than their male counterparts (Ro & Knight, 2016). This study suggests that the benefits of co-curricular activities may not solely depend on the amount of participation, but that women may gain more advantages from these activities than men. The present study’s findings can only show that women reported having significantly more opportunities to practice the professional skills of Shared Leadership and Communication than men, because collecting data on students’ specific experiences was outside the scope of this work.