Use of an 'Importance-Performance' Tool to Identify Teachers' Training Priorities

Practice Paper Recommended citation: Sylvestre, E., Mathou, C., Laperrouza, M., & Lanarès, J. (2025). Use of an 'Importance-Performance' Tool to Identify Teachers' Training Priorities. In Kangaslampi, R., Langie, G., Järvinen, H.-M., & Nagy, B. (Eds.), SEFI 53rd Annual Conference. European Society for Engineering Education (SEFI), Tampere, Finland. DOI: 10.5281/zenodo.17631523. This Conference Paper is brought to you for open access by the 53rd Annual Conference of the European Society for Engineering Education (SEFI) at Tampere University in Tampere, Finland. This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License. USE OF AN "IMPORTANCE-PERFORMANCE” TOOL TO IDENTIFY TEACHERS’ TRAINING PRIORITIES E. Sylvestre a,1, C. Mathou b, M. Laperrouza c, J. Lanarès d a UNIL, Lausanne, Switzerland, 0000-0003-4575-7755 b UNIL, Lausanne, Switzerland, 0000-0002-9419-1682 c EPFL, Lausanne, Switzerland, 0000-0001-6316-254X d UNIL, Lausanne, Switzerland, 0000-0002-9732-0611 Conference Key Areas: 14. Building the capacity and strengthening the educational competences of engineering educators; 15. Improving higher engineering education through researching engineering education Keywords: Faculty development, Teaching competences, Importance-Performance analysis, STEM, Self-assessment tool ABSTRACT This practice paper presents the development and application of a self-assessment tool designed to identify teaching competences that university instructors consider both important and in need of improvement. Rooted in an analysis of nine international competence frameworks, the tool – IPTCHE (Importance–Performance of Teaching Competences in Higher Education) – evaluates seven core pedagogical competences through a double-entry questionnaire based on Best–Worst Scaling methodology. Administered to agricultural engineering faculty members in Tunisia, the instrument captures perceived performance and importance for each competence, allowing for a cross-analysis to identify key development priorities. The results highlight well-established strengths, such as course design and reflective practice, alongside critical gaps in areas like active learning facilitation and peer collaboration. Genderand rank-based analyses reveal differentiated perceptions: women place greater importance on continuous improvement, while early-career faculty members (assistant professors) report stronger performance in communicating teaching intentions but greater needs in mastering course design. The study demonstrates how the IPTCHE tool offers a robust framework for aligning faculty development programs with the expressed needs of educators, thereby 1 Corresponding Author E. Sylvestre [email protected] enhancing their relevance and impact. While promising, the findings are limited by the relatively small sample size (n=27), underlining the need for broader implementation and further research. Overall, this work contributes to more contextaware and responsive approaches to faculty development, particularly in STEM disciplines, by placing teachers' perceptions at the center of training design. 1 INTRODUCTION The development of the pedagogical competences of higher education teachers is one of the major levers for strengthening the quality of education (Gibbs & Coffey, 2004), particularly in engineering programs (Finelli et al., 2008). Several recent studies confirm that pedagogical development programs have a significant positive impact, not only on the knowledge and competences of higher education teachers (Bilal et al., 2019; Steinert et al., 2016), but also on their professional identity (Steinert et al., 2019) and on student learning (Condon et al., 2016). More specifically in the field of STEM subjects, several results show that the development of pedagogical competences, particularly through approaches centered on active learning (flipped classrooms, projects, problem-based learning, etc.), is an essential element of the quality of education provided in engineering schools (McCarthy Hintz et al., 2025; Frey et al., 2020). This raises a central question: how can we rigorously identify the teaching competences deemed a priority by teachers and assess their perceived level of mastery, in order to guide more contextualized, targeted and effective continuing education programs? We propose to use an “Importance-Performance” tool to address this question. To this end we have developed a questionnaire based on Best–Worst Scaling approach as an initial step to adapt a competence framework based on a benchmarking of existing competence frameworks. 2 LITERATURE REVIEW : TOWARDS MORE CONTEXTUALIZED FACULTY DEVELOPMENT - IDENTIFYING TEACHING PRIORITIES AND PERCEIVED COMPETENCE With the aim of identifying the factors that determine the effectiveness of these measures, Steinert and his colleagues (Steinert & McLeod, 2006; Steinert et al., 2016) analyzed more than a hundred articles evaluating various programs. They highlighted several key elements such as the relevance and contextualization of the content presented, the mobilization of experiential learning activities and opportunities for practical application and reflective moments. However, some studies emphasize that many programs are still too generic and insufficiently rooted in disciplinary or contextual realities, which limits their relevance in the eyes of teachers (Zwolińska-Glądys et al., 2024). With this in mind, several authors (Poumay, 2016; Demougeot-Lebel & Perret, 2011) emphasize the importance of conducting a detailed needs analysis to design and adjust professional development programs. It is within this framework that we are contributing to the development of the pedagogical competences of teachers of agricultural engineering in Tunisia, based on the priorities expressed by the teachers themselves. This research is part of a process of local appropriation of the issues of teacher professionalization by mobilizing a methodological approach combining importance (perception of what is essential for effective teaching) and perceived performance (perception of one's own mastery) inspired by the work of Huybers (2014). 3 METHODOLOGY 3.1 CONCEPTUAL FRAMEWORK To design a tool for gathering teachers' perceptions of their professional development priorities, we first conducted an in-depth comparative analysis of nine pedagogical competence frameworks1 from various French-speaking contexts (France, Belgium, Switzerland, Quebec, international networks). This review identified the competences most frequently mentioned and considered central to the university teaching profession. Four main areas of competence emerge, with a high degree of congruence between the reference frameworks: • Teaching design and planning; • Pedagogical implementation and learning support; • Learning assessment; • Professional development and institutional commitment. Some complementary competences, such as the use of technology or contextappropriate communication, are mentioned more marginally. Conversely, crossdisciplinary dimensions such as reflective posture, inter-teacher collaboration or continuous improvement of practices appear in the majority of the reference frameworks, although formulated differently depending on the source. Based on the comparative analysis, we selected seven teaching competences that appeared consistently across frameworks and were considered particularly relevant for the Tunisian agricultural engineering context . The selection was guided by their frequency, alignment with strategic educational goals, and their clarity for nonspecialist faculty members. 3.2 DEVELOPMENT OF THE SELF-ASSESSMENT TOOL: THE IPTCHE (IMPORTANCE–PERFORMANCE OF TEACHING COMPETENCES IN HIGHER EDUCATION) QUESTIONNAIRE The tool thus constructed, the IPTCHE questionnaire, is based on a methodology inspired by Best–Worst Scaling (Huybers, 2014), a forced choice approach that allows perceived priorities to emerge in a more refined way than traditional Likerttype scales, which are often subject to bias (neutrality, acquiescence, etc.) (Baumgartner & Steenkamp, 2001). The tool is composed of two main parts, each based on a series of 7 blocks of 5 items (each item representing a central pedagogical competence): Part 1: Perceived performance Respondents are invited to evaluate their own teaching practice. For each block, they must: • Select the statement that best represents their current practice, • and the one that represents it the least well. 1 MESRI – French Ministry of Higher Education and Research, 2019; Aix-Marseille University framework, 2020; Université catholique de Louvain – UCLouvain, 2018; Université de Liège – ULiège, 2016; Université de Sherbrooke, 2017; University of Lausanne – UNIL, 2022; EPFL – École Polytechnique Fédérale de Lausanne, 2021; AUF – Agence Universitaire de la Francophonie framework, 2020; RIFPES – International Francophone Network for the Professionalization of University Teachers, 2021. This part allows the identification of perceived strengths and areas for personal improvement in relation to the targeted pedagogical competences. Part 2: Perceived importance • The teachers then evaluate, according to the same forced choice logic, the importance they attach to each competence for ensuring quality teaching. They indicate: • The item they consider most important for promoting student learning, • and the one they consider least important. The questionnaire is based on seven pedagogical competences identified from the comparative analysis of nine international benchmarks presented above. These competences have been rephrased in clear and contextualized statements. They cover the following areas: • Designing and planning one’s teaching: alignment between objectives, activities and assessments. • Communicating with and for strudents: clear explanation of teaching intentions. • Facilitating teaching and learning activities: use of varied strategies to encourage student engagement. • Supporting and guiding student learning: feedback to students throughout the learning process. • Assessing student learning: relevant measurement of student achievement. • Collaborating within the teaching team: teamwork with colleagues on the program. • Reflecting on and improving one’s teaching: revision and evolution of teaching practices. Table 1. Translated version (from French) of the competences and corresponding statements used in the IPTCHE questionnaire Competences Performance Importance 1. Designing and planning one's teaching P1 – I design my courses ensuring that learning objectives, teaching activities, and assessments are coherently aligned. I1 – Courses should be designed so that learning objectives, teaching activities, and assessments are coherently aligned. 2. Communicating with and for students P2 – I clearly communicate my teaching intentions to students. I2 – Teaching intentions should be clearly communicated to students. 3. Facilitating teaching and learning activities P3 – I offer a variety of teaching activities to foster student engagement. I3 – Teaching activities should be varied to foster student engagement. 4. Supporting and guiding student learning P4 – I provide feedback to students throughout their learning process. I4 – Feedback should be provided to students throughout their learning process. 5. Assessing student learning P5 – I design assessments that effectively measure students’ learning outcomes. I5 – Assessments should be designed to effectively measure students’ learning outcomes. 6. Collaborating within the teaching team P6 – I actively collaborate with other instructors in the study program. I6 – There should be active collaboration among instructors within the study program. 7. Reflecting on and improving one’s teaching P7 – I continuously improve my teaching practices. I7 – Teaching practices should be continuously improved. In addition to the compulsory choice blocks, two open questions invite teachers to: • Identify their strengths in terms of teaching, • and the aspects that need improvement. A final open question invites them to suggest other elements considered important for ensuring the quality of teaching, thus offering a space for free expression to enrich the analysis. 4 RESULTS AND INSIGHTS The survey was conducted among a sample of STEM teachers of Institution of Agricultural Research and Higher Education in Tunisia in order to explore teachers' perceptions of the importance of and their own performance in relation to seven key pedagogical competences identified in the previous work of benchmarking competence frameworks. 4.1 Population surveyed A total of 62 teachers completed the questionnaire. The gender distribution was relatively balanced, with 60% women (n=36) and 40% men (n=25). In terms of academic status, the majority of the population were assistant teachers (n=33, or 53%), followed by professors (n=14, or 23%) and lecturers (n=10, or 16%). The population is also rather experienced, with: • 32% (n=20) having between 11 and 15 years of service, • 29% (n=18) with more than 20 years of experience, • 15% (n=9) between 16 and 20 years, • 13% (n=8) between 7 and 10 years. However, only 27 responses were considered usable for the cross-analysis “Importance/Performance”, due to incomplete or non-compliant responses in the other cases, which makes this survey a first exploratory phase of our study. 4.2 Cross-analysis of Importance–Performance of Teaching Competences Based on the mean scores and standard deviations presented in Table 2, we positioned the seven teaching competences in a four-quadrant Importance– Performance matrix presented in Figure 1. Table 2. Mean scores and standard deviations for each competence (Performance and Importance) Performance Importance Mean Std. Dev. Mean Std. Dev. 1. Designing and planning one's teaching 2.30 2.02 2.19 1.96 2. Communicating with and for students 0.52 1.99 –0.19 2.22 3. Facilitating teaching and learning activities 0.37 2.13 1.00 2.06 4. Supporting and guiding student learning –1.00 1.52 –1.15 1.59 5. Assessing student learning –0.30 1.64 –0.37 1.74 6. Collaborating within the teaching team –3.30 1.94 –2.30 2.46 7. Reflecting on and improving one's teaching 1.41 1.65 0.81 1.18 Fig. 1. Cross-analysis of Importance–Performance of teaching competences. This cross-analysis reveals well-established strengths, areas requiring targeted improvement, and underappreciated dimensions that can inform the strategic orientation of future faculty development programs. However, due to the relatively high standard deviations, only two of the four quadrants show statistically robust patterns. This limitation is acknowledged in our analysis. Quadrant 1 – High Importance / High Performance (Competences to maintain) • Competence 1 – Designing and planning one’s teaching: This competence is both highly valued and well-mastered by the respondents. It reflects a strong foundation in course design and alignment, and should be reinforced and supported in training initiatives. • Competence 7 – Reflecting on and improving one’s teaching: Also perceived positively, this competence indicates a proactive engagement in pedagogical development, even if it is less central than course design. Quadrant 2 – High Importance / Low Performance (Priority for development) • Competence 3 – Facilitating teaching and learning activities: The gap between importance and self-reported performance suggests this is a clear area for growth. Teachers see the value in diversifying their methods but may lack confidence or tools to do so effectively. This makes it a key target for professional development. Quadrant 3 – Low Importance / Low Performance (Competences to monitor) • Competence 4 – Supporting and guiding student learning & Competence 5 – Assessing student learning: These competences are both under-prioritized and under-developed. While they may not be immediate priorities, they are core to effective teaching and merit further exploration perhaps through qualitative inquiry into local perceptions or institutional constraints. • Competence 6 – Collaborating within the teaching team: This is the least valued and least practiced competence. It may reflect structural or cultural barriers to collaboration but also points to an opportunity to promote collective teaching practices over time. Quadrant 4 – Low Importance / High Performance (Potential overinvestment) • Competence 2 – Communicating with and for students: While well mastered, this competence is perceived as less critical than others. It may represent a routine strength no longer seen as a priority, or one that could be rebalanced in favor of emerging pedagogical needs. The gender-based analysis (n=27) revealed both significant and emerging trends. A statistically significant difference (p < 0.05) was observed for Competence 7 – Reflecting on and improving one's teaching: female respondents rated this competence as more important (M=1.17) than their male counterparts (M=0.11). This suggests a stronger emphasis among women on self-reflection and continuous improvement as key elements of effective teaching. In addition, a trend (p < 0.10) was identified for Competence 3 – Facilitating teaching and learning activities, where women reported slightly higher levels of self-perceived performance (M=0.89) than men (M=-0.67). Although not statistically significant, this tendency may point to greater confidence or experience among women in deploying diverse teaching strategies, and merits further investigation. These gender-related nuances, though limited in scope, highlight potential differences in pedagogical values and practices that could be explored in greater depth in future research or training initiatives. The analysis by academic rank revealed several notable differences in the perception of teaching competences. A statistically significant difference (p < 0.05) was found for Competence 2 – Communicating with and for students: assistant professors reported a higher level of self-perceived performance (M= 1.21) compared to senior academic staff (professors and associate professors, M= –0.23). This result may suggest that early-career faculty feel more confident in communicating their pedagogical intentions, possibly due to more recent exposure to student-centered approaches or pedagogical training. A trend (p < 0.10) was also observed for Competence 1 – Designing and planning one's teaching: assistant professors rated this competence as more important (M= 2.79) than senior academic staff (M= 1.54). This could reflect a heightened concern among early-career faculty for mastering the fundamentals of course design, particularly in contexts where teaching responsibilities are taken on early in their academic trajectory. These differentiated perceptions and performance levels—across both gender and academic rank—may also contribute to the observed lack of emphasis on certain competences, such as inter-teacher collaboration or assessment practices. This limited emphasis could stem from structural constraints, the absence of institutional incentives, or the persistence of a teaching culture that undervalues collective pedagogical engagement. Addressing these issues may require not only tailored training, but also a shift in organizational norms and recognition structures. These findings point to differentiated needs across academic ranks, underlining the value of tailored professional development programs that take into account both the career stage and pedagogical priorities of faculty members. 5 CONCLUSIONS AND IMPLICATIONS This exploratory study demonstrates the relevance of using a cross-analysis of perceived importance and performance to identify teaching development priorities from the perspective of university staff. By grounding our approach in a benchmark of internationally recognized pedagogical competences and using a Best–Worst Scaling methodology, we were able to highlight both well-established strengths and areas requiring targeted support. The results suggest that while some competences - such as course design and reflective practice - are well mastered and valued, others - such as active learning facilitation and collaboration among peers -represent significant opportunities for development. Moreover, genderand rank-based differences point to the need for differentiated training paths that consider the diversity of experiences, perceptions, and professional contexts among faculty members. These findings directly address our initial research question by providing a structured, perception-based framework for identifying and prioritizing pedagogical competences, thereby supporting the design of faculty development programs that are both targeted and context-sensitive. However, this study presents certain limitations. In particular, the relatively small number of valid responses limits the generalizability of the findings. Future studies should involve larger and more diverse samples, possibly complemented by qualitative data, to deepen the interpretation of trends and better tailor faculty development strategies to institutional contexts. 6 ACKNOWLEDGMENT The development of the tool is part of a research project supported by the HES-SO MENA (RPG 2023-5). The authors would like to acknowledge the contributions of the colleagues from INAT and IRESA who supported various aspects of this research project. We thank Prof. Aziza Boubaker and Prof. Thouraya Souissi for their input during the design, dissemination and data collection phases.