Reduction of the students´ evaluation of education quality questionnaire

Abstract—Assessment of students and the evaluation of their satisfaction has been an important element in the improvement of teaching quality in all the Higher Education areas. Specifically the student participation in Computer Science (CS) and Information System (IS) has been highlight valued. Thus a large number of methodologies and standard tools regarding student evaluation has been developed. Specifically, the Students´ Evaluation of Education Quality (SEEQ) is a tool that is validated for international use. But its use leads to several problems, such as the low voluntary participation of students. To solve these problems, a short version of this questionnaire developed using statistic tools is proposed. After using the proposed new version, the voluntary participation of students increased. The reduction of the number of questions facilitates the analysis of data, improving the flow of information and feedback between professors and students. I. INTRODUCTION: QUALITY IN HIGHER EDUCATION CONCEPTS of quality from the entrepreneurial world are increasingly being incorporated into the university field [1]. According to the “Declaration of Prague” (2001), quality should be an important and determinant aspect of Europe´s international attractiveness and competitiveness [2]. In 2003 in Berlin (Berlin, 2003), the Ministers responsible for Higher Education stressed that “the quality of Higher Education has proven to be the heart of the setting up of a European Higher Education Area” [2]. T When these ideas have been translated into action, the “teaching quality” concept has become outstanding. According to Kember, “it might be noted that concern about Teaching Quality is growing at the national level. This appears to be a worldwide phenomenon” [3]. In 1992, Stones pointed out [4] that “quality teaching is more properly conceived of as a unified field embracing both theory and practice in which teachers, teacher educators and researchers are jointly responsible for the development of theoretical understanding and the improvement of teaching”. Currently, there remains a high degree of concern about the improvement of teaching quality at the higher education level. Specifically, the improvement of teaching quality is one of the primary matters that must be addressed continuously by universities. The question is how to achieve this continuous improvement. Continuous improvement is not a tool or a technique but rather a way of life (or at least a cultural approach to quality improvement) [5]. According to the UNE 66178:20004 model [6], there are three steps that should be taken into account during an improvement process: 1. analysis of the information for the improvement; 2. the improvement project; and 3. monitoring, evaluating and reviewing the improvement. One method of attaining continuous improvement is the PDCA Cycle (also known as the Shewhart or Deming cycle: Plan-Do-Check-Act) [7], which emphasizes the continuous, never-ending nature of process improvement. The PDCA Cycle highlights and demonstrates that improvement programs must start with careful planning, result in effective action, and move back to careful planning in a continuous cycle. Figure 1 shows this global idea. Fig. 1 The PDCA Cycle as a tool to continuous improvement [8], [9] Clearly, it is necessary to collect and evaluate data in order to obtain conclusions. Improvement actions should be based on the data and the conclusions of the evaluation. This point of view is completely applicable to the university environment, especially to lecture theatres. To improve the teaching quality, it is necessary to evaluate the teaching process and its results. In this evaluation, the students’ opinions about learning and the teaching process are crucial due Reduction of the Students´ Evaluation of Education Quality Questionnaire Montserrat Corbalan R&D&I EduQTech group. Polytechnic University of Catalonia (UPC), Terrassa 08222, Spain) Email: montserrat.corbalan@u pc.edu Inmaculada Plaza R&D&I EduQTech group University of Zaragoza, Teruel 44003, Spain Email: [email protected] Eva Hervas R&D&I EduQTech group University of Zaragoza, Teruel 44003, Spain Email: [email protected] Emiliano Aldabas R&D&I EduQTech group Polytechnic University of Catalonia (UPC), Terrassa 08222, Spain Email: emiliano.aldabas@u pc.edu Francisco Arcega R&D&I EduQTech group University of Zaragoza, Zaragoza 55008, Spain Email: [email protected] Preprints of the Federated Conference on Computer Science and Information Systems pp. 713–719 c 2013 713 to the students’ roles as the primary consumers in higher education [10]. This paper focuses on students’ evaluations of the quality of higher education. The extensive body of research regarding student evaluation of teaching leads us to look for standard tools and methodologies (section 2). Section 3 details a widespread method that can be used internationally, the Students’ Evaluation of Educational Quality SEEQ) questionnaire, and considers the problems that teachers identified after its application. To solve these problems, section IV presents a proposal for the reduction of this questionnaire. Sections V and VI show an example of its application and give several guidelines regarding the use of this short version of the SEEQ questionnaire. The final conclusions are explained in section VII. II.REVIEW OF THE STANDARDS Spooren and Mortelmans underlined the value of student evaluations of teaching. They found that students reward good teachers with higher ratings on several scales of teacher performance [11]. The literature contains an overwhelming number of data collection instruments and scales. Several authors chose to develop their own form (see, for instance, [12] or [13]). In some cases, the forms were developed by faculty committees [14]. In general, there is an extensive body of research regarding student evaluation of teaching and how students contribute to assessments of teaching effectiveness [15], [16]. Thus, it is possible to conclude that there is a need to unify and standardize the approaches and specific tools used to evaluate the teaching quality. Standards are public technical documents that establish common terminology in a field (in this paper, quality). They set specifications extracted from experience, knowledge, and available technology [17], [18] and [19]. There are several international and national standards developed for the teaching field that are applicable to this work [20]. * UNE 66931 is a Spanish standard that aims to provide guidelines for the application of the ISO 9001 model in educational organizations. It is equivalent to the document IWA 2 published by ISO [21]. Section 8.2 (concerning monitoring and measurement) points out that the educational organization must have reliable methods to measure and control the satisfaction of the client (8.2.1.). Moreover, the educational organization should define and use methods to monitor the results of the educational product (8.2.4). Furthermore, the customer and stakeholder satisfaction surveys are described as important data (section 8.4). * The ISO/IEC 19796 family, focused on the information technology field, is an international standard under the general title “Information technology – Learning, education and training – Quality management, assurance and metrics” [22] and [23]. This family is a framework used to describe, compare, analyse, and implement quality management and quality assurance approaches. It will serve to compare different existing approaches and to harmonise these approaches towards a common quality model. [17] [22]. The ISO/IEC 19796-3 [23] is an instrument for the implementation and adaptation of the first quality standard ISO/IEC 19796-1 and, in particular, for the specification of the individual process descriptions [24]. According to Campo [25], this family has been defined abstractly and without specific guidelines to provide a mechanism to its implementation. Thus, it is possible to conclude that international and national standards specially developed for the teaching field provide professionals with frameworks and guidelines to improve the quality of education. However, these standards do not provide specific tools or mechanisms to evaluate teaching quality. In addition, other models and standards regarding quality that are widely used in the entrepreneurial environment can be applied to lecture theatres. The use of these standards helps institutions respond to the EHEA (European Higher Education Area) quality requirements [26]. In the present case, two standards could be applicable: UNE 66178:2004 and UNE 66176:2005. * UNE 66178:2004 The standard entitled “Quality management systems. Guide for the management of process for improvement” is focused on continuous improvement. This standard is based on the idea that every organization needs to improve. Its capability to satisfy the requirements of its stakeholders (such as the customers, staff and social environment) determines the survival of the organization. Furthermore, these needs are changeable [27]. This point of view is completely applicable to the university environment, especially to lecture theatres [26]. In Appendix A, teachers can find a list of techniques and tools that can be used in the improvement process. The cycle of Deming is listed in this Appendix. * UNE 66176:2005 This Spanish standard is titled “Quality management systems. Guide for measuring, monitoring and analyzing customer satisfaction”. According to its title, this standard specifies guidelines for the definition and development of a measuring process for customer satisfaction [6]. Its guidelines are generic and can be applicable to any organisation, regardless of its size or activity. Of particular note is Table 1 of the standard, which contains different techniques for data collection and indicates their advantages and disadvantages. Professors interested in its application will find appendices A to E significant [26]. The analysis of standards UNE 66176 and UNE 66178 produces the same results as the previous models (UNE 66931 and ISO 19796): they provide professionals with frameworks and guidelines but do not provide specific tools or mechanisms to control the satisfaction of students. The global conclusion of this section is that there is a high number of collection instruments for obtaining data about student satisfaction of teaching quality. Thus, it is necessary to look for other standard tools. The review of international and national standards can provide guidelines and global methodologies. Their application will allow for the extraction of ideas for improving traditional working 714 PREPRINTS OF THE FEDCSIS. KRAK ´ OW, 2013 methods at the university level [26]. However, to apply the previously cited standards, professors who wish to use them must carry out a tailoring process. Thus, the standards do not provide any specific tool to evaluate the students’ opinions about teaching quality. III. DESCRIPTION OF THE SEEQ The lack of a standardised tool to evaluate students’ opinions about teaching quality leads us to look for a widespread method that can be applied internationally. In terms of the more formal, internationally validated questionnaires in use in higher education, five are particularly worth mentioning [28]: • The Students’ Evaluation of Educational Quality (SEEQ), • The Course Experience Questionnaire (CEQ), • The Module Experience Questionnaire (MEQ), • The Postgraduate Research Experience Questionnaire (PREQ), and • The Experiences of Teaching and Learning Questionnaire (ETLQ). Richardson recommended using either the SEEQ or the CEQ, as both have been validated for international use through research studies [29]. Keane pointed out that the SEEQ and MEQ have potential because they have been statistically validated [28]. In the present study, the SEEQ questionnaire was chosen. There are several reasons for this choice: a robust factor structure, excellent reliability, and reasonable validity [30]. Furthermore, as has been mentioned, the SEEQ has been validated for use internationally. It is possible to find universities in many different countries that use the SEEQ. Although not exhaustive, the following list presents several examples: the Universities of Manitoba [31], Saint Mary’s [32] Mount Allison [33] and Saskatchewan [34] in Canada; Fordham University [35] and the Schreyer Institute for Teaching Excellence [36] in the U.S.; Oxford University [37] and University of Leicester [38] in the U.K.; Semnan University [39] in Iran; Curtin University [40] in Australia; and the Universities of Navarra [41] and Vigo [42] and the Polytechnic University of Catalonia [43] in Spain. This international use will enable the development of comparative analyses in the future. The SEEQ was developed by Dr. Herbert Marsh of the University of Western Sydney [44]. Dr. Marsh is an internationally recognised expert in the area of psychometrics. Now in the public domain, the SEEQ has been extensively tested and used in more than 50,000 courses with over one million students at both the graduate and undergraduate levels [33]. Using a five-point scale, the SEEQ questionnaire examines different characteristics of effective teaching. Each of these categories contains three or four questions. It is possible to find different versions of the SEEQ [45], [46]. The version used in this work consist of 37 questions. It is detailed in references [43] [33]. * The questionnaire finishes with three open questions. This SEEQ questionnaire was used in seven subjects in three cities and three different centres: Polytechnic School of Teruel (Spain), School of Engineering of Terrassa and Faculty of Engineering and Architecture of Zaragoza (University of Zaragoza (Spain). The use of the SEEQ enabled professors to identify strengths and weaknesses and to improve the teaching-learning process. However, the main problem was the low participation of students: a high number of students did not answer the questionnaire or answered only the first questions on the form [47]. Moreover, Verdugo and Cal remarked upon the fact that rapid feedback is needed. These authors explained that teachers could be overloaded with work and not provide rapid feedback [46]. In order to solve these problems, this paper proposes a reduction of the SEEQ questionnaire developed in collaboration with a student [48] in order to encourage the students’ research. IV. PROPOSAL OF REDUCTION To maintain the reliability of the short SEEQ questionnaire, statistical parameters should be used. Computing tools can help researchers in this process. In this work, analyses have been developed with SPSS® [49]. Specifically, this statistical package enables the calculation of the Cronbach's Alpha and Pearson's r parameters [50]. The values of Cronbach's Alpha and Pearson's r Correlation Coefficient for the long SEEQ questionnaire (37 questions-items) were calculated. To develop the analysis, a sample of 111 polls was used and items with different scales were recoded [48]. Table I shows the value of the global Cronbach´s Alpha. TABLE I. STATISTICAL VALUES OF RELIABILITY FOR THE LONG SEEQ QUESTIONAIRE (37 ITEMS) Number of Items Cronbach's Alpha 37 0.920 The Cronbach's Alpha can be used in the reduction process, as Cronbach's Alpha is an index of reliability. According to Santos [51], this coefficient ranges in value from 0 to 1 and may be used to describe multi-point formatted questionnaires or scales (i.e., in this rating scale, 1 = poor and 5 = excellent). The higher the score is, the more reliable the generated scale. Nunnaly [52] (cited by Santos) indicated 0.7 to be an acceptable reliability coefficient, but lower thresholds are sometimes used in the literature [51]. Table II show the same data after the reduction process, using Cronbach's Alpha as reduction criteria. TABLE II. STATISTICAL VALUES OF RELIABILITY FOR THE SHORT SEEQ QUESTIONAIRE (22 ITEMS) Reduction criteria: Cronbach's Alpha Number of Items Cronbach's Alpha 22 0.936 The global reliability for the initial long SEEQ questionnaire was 0.920, and the value for the short SEEQ is 0.936. MONTSERRAT CORBALAN ET AL.: REDUCTION OF THE SEEQ QUESTIONNAIRE 715 Thus, there is an improvement of the internal consistency of the questionnaire. Several statistics experts recommend developing the reduction process using two criteria: Cronbach's Alpha and Pearson's r Correlation Coefficient. Although both analyses tend to give similar results, the combination is useful because it provides more information with which to make decisions [50]. With the initial data, a new reduction process was developed using the correlation as new criteria (third column in our tables). Table III shows the statistical results after completing the process. TABLE III. STATISTICAL VALUES OF RELIABILITY FOR THE SHORT SEEQ QUESTIONAIRE (22 ITEMS) Number of Items Cronbach's Alpha 22 0.936 Both analyses lead to the same results: •the fifteen deleted items are the same; •there are no negative correlations; and •the final value of reliability (global Cronbach´s Alpha) is the same, 0.936, and there is an improvement in the questionnaire’s internal consistency. The short version of the SEEQ questionnaire proposed in this work is detailed below (the item number from the long version is in brackets). * Learning. 1 (1) - I find the course intellectually challenging and stimulating. 2 (2) - I have learned something that I consider valuable. 3 (3) - My interest in the subject has increased as a consequence of this course. * Enthusiasm 4 (5) – The instructor is enthusiastic about teaching the course. 5 (6) – The instructor is dynamic and energetic in conducting the course. 6 (7) – The instructor enhances presentations with the use of humour. 7 (8) – The instructor's style of presentation holds your interest during class. * Organisation 8 (9) – The instructor's explanations are clear. 9 (10) – The course materials are well prepared and carefully explained. 10 (12) – The instructor gives lectures that facilitate taking notes. * Group Interaction 11 (13) – Students are encouraged to participate in class discussions. 12 (14) – Students are invited to share their ideas and knowledge. 13 (15) – Students are encouraged to ask questions and are given meaningful answers. 14 (16) – Students are encouraged to express their own ideas and/or question the instructor. * Individual Rapport 15 (17) – The instructor is friendly towards individual students. 16 (18) – The instructor makes students feel welcome in seeking help/advice in or outside of class. 17 (19) – The instructor has a genuine interest in individual students. * Breadth 18 (21) – The instructor contrasts the implications of various theories. 19 (22) – The instructor presents the background or origin of ideas/concepts developed in class. 20 (23) – The instructor presents points of view other than his/her own when appropriate. 21 (24) – The instructor adequately discusses current developments in the field. * Examinations 22 (27) – The examinations/graded materials test the course content that is emphasized by the instructor. In conclusion, fifteen questions, approximately 40%, have been eliminated with this method. V. EXAMPLE OF APPLICATION The new short version of the SEEQ questionnaire was used in different university subjects. For instance, in Circuits and Electric Drives the previous academic year, students filled out the long version of the SEEQ questionnaire. As the teacher is the same, a comparison is possible. Table VII shows the level of student participation (number of completed questionnaires versus number of registered students). The percentage of voluntary involved students has increased. In table IV, the items corresponding to the four lowest-scoring questions from both versions are listed. In this case, two items are equal. In addition, higher-valued items are listed, and there are three equal items. TABLE IV. PARTICIPATION LEVEL Version % Participation Long 53.16% Short 69.23% Thus, the SEEQ questionnaire helps professors to detect weaknesses and strengths. These results can be used as the starting point of the following improvement plan. The proposed short version of the SEEQ questionnaire has been used in other subjects in which the teachers were not the same as those in previous years. Nevertheless, in this academic year, the long version was used voluntarily in three subjects. The average percentage of participation was 69.41%. In addition, the short version was used voluntarily in three other subjects, with an average percentage of participation of 86.84%. 716 PREPRINTS OF THE FEDCSIS. KRAK ´ OW, 2013 The conclusion is clear: the use of the proposed version increases the voluntary participation of students. VI. GUIDELINES FOR THE USE OF THE PROPOSED SHORT VERSION Several guidelines can help professors use the proposed short version of the SEEQ questionnaire. •As explained in the introduction, the SEEQ questionnaire can be used as a tool to develop improvement processes. Professors can distribute the questionnaire in the middle of the academic year. •It is advisable to give the questionnaire to the teachers before the analysis of the data [53], [47]. •After the analysis of the data, instructors should identify the lowestand highest-scoring questions on the questionnaire. These results can help instructors develop a plan to improve the weaker points while maintaining the strengths. The next step is to implement the plan. It is recommended that the questionnaire is used again at the end of the academic year and that the lessons learned be recorded in order to remember or to explain them to colleagues. TABLE V. THE DETECTED STRENGTHS AND WEAKNESS Version Weakness Long – Students are encouraged to participate in class discussions. – Students are invited to share their ideas and knowledge. - How does this course compare with other courses you have had at this university? – Your level of interest in the subject prior to this course Short - My interest in the subject has increased as a consequence of this course. - The instructor enhances presentations with the use of humour. - The instructor's style of presentation holds your interest during class. – Students are encouraged to participate in class discussions. – Students are invited to share their ideas and knowledge. Strengths Long – The instructor is friendly towards individual students. – The instructor makes students feel welcome in seeking help/advice in or outside of class. – The instructor has a genuine interest in individual students. – The instructor is adequately accessible to students during office hours or after class. – The examinations/graded materials test the course content that is emphasised by the instructor. Short - The instructor is enthusiastic about teaching the course. – The instructor is friendly towards individual students. – The instructor makes students feel welcome in seeking help/advice in or outside of class. – The instructor has a genuine interest in individual students. – The examinations/graded materials test the course content that is emphasised by the instructor. •Students require feedback as an element of motivation. As Chen remarked [54], “This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. (…) Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback.” Thus, teachers can give students feedback on the evaluation results. •According to Centra [55], student evaluations of teaching can only facilitate improvement when professors are able to access new and valuable information from them. Teachers must then understand how to translate the new evidence into action and must be motivated to do so [55] and [56]. •The short SEEQ questionnaire can be used in combination with other methods [57] [28]. VII. CONCLUSIONS. Concepts of quality taken from the entrepreneurial world are increasingly being incorporated into the academic field. Currently, there remains a high degree of concern about the improvement of teaching quality at the higher education level. Specifically, the improvement of teaching quality is one of the primary matters that should be addressed continuously by universities. To improve teaching quality, it is necessary to evaluate the teaching process and its results. In this evaluation, students’ opinions about learning and the teaching process are crucial because the students are the primary consumers in higher education. The extensive body of research regarding student evaluation of teaching leads us to look for standard tools and standard methodologies. The review of national and international standards enables us to obtain guidelines and global methodologies. Their application will allow for the extraction of ideas for improving universities’ traditional working methods. However, they do not provide any specific tool to evaluate the opinion of students regarding teaching quality. International organizations should work together to define standard tools and methodologies to evaluate students’ opinions. The SEEQ, developed by Dr Herbert Marsh, is a tool validated for use internationally. It has a robust factor structure, excellent reliability and reasonable validity. However, there are two problems with its use: the low participation of students (there are 37 questions to be answered) and the teachers’ sense of being overloaded if they try to provide rapid feedback. To solve these problems, a short version of the questionnaire is presented. It was possible to reduce the form using statistical methods. The proposed version consists of twenty-two questions. After using this new short version, the voluntary participation of students increased. The short SEEQ questionnaire can be used as a tool to develop a teaching improvement process as its use detects teaching weaknesses and strengths. It is recommended that the questionnaire be used in the middle and at the end of the academic year in order to establish an improvement cycle. The reduction of the number of questions facilitates data MONTSERRAT CORBALAN ET AL.: REDUCTION OF THE SEEQ QUESTIONNAIRE 717 collection and the analysis of the data, in both cases, with software tools. Also it improves the flow of information. In this way, the proposed version of the SEEQ questionnaire helps produce feedback intended to motivate students’ participation in the teaching evaluation process. In addition, the short questionnaire helps professors receive new and valuable information about their teaching from student evaluations more quickly. Professors can use the short SEEQ questionnaire with other tools. The proposed short SEEQ questionnaire can be used by other university professors regardless of the subject or the degree course. ACKNOWLEDGMENT The authors would like to acknowledge the “Chair in Innovation and Technological Quality” and the CTP for their help. Thanks to the IEEE Foundation “Gobierno de Aragón” and to the “Fondo Social Europeo” for their support to the EduQTech group. REFERENCES [1] R. G. Lewis and D. H. Smith, “Total Quality in Higher Education”. Delray Beach, FL: St. Lucie Press, 1994, Total Quality Series. [2] F. Jurado, et al, “A review of the Accreditation Bodies and Processes in Europe. A vision from the Engineering”, 35th ASEE/IEEE Frontiers in Education Conference. F2D, 2005, pp. 13 - 18. [3] D. Kember “Action Learning and Action Research: Improving the Quality of Teaching and Learning” Ed. Routledge. 2000. [4] E. Stones “Quality Teaching: a sample of cases”. Taylor & Francis, 1992. [5] I. Plaza and C. Medrano “Continuous Improvement in Electronic Engineering Education” IEEE Transactions on Education, Vol. 50, Nº. 3, 2007 pp. 259 – 265. [6] UNE 66176:2005. Quality management systems. Guide for measuring, monitoring and analysing customer satisfaction. 2005. [7] W. E. Deming, “Out of the Crisis”. MIT Center for Advanced Engineering Study. ISBN 0-911379-01-0. 1986. [8] Plaza, I. et al (2008) “Code of good teaching practices based on quality criteria” (Original in Spanish) Workshop of Educational Innovation, ITC and Educational Research (2ª Jornadas de Innovación Docente, TIC e Investigación educative). CD of the workshop. [9] JJ. Marcuello et al “Code of good teaching practices based on quality criteria”. EAEEIE Annual Conference, 2008 19th, 2008. pp: 70-75. Digital Object Identifier 10.1109/EAEEIE.2008.4610161. Available at IEEEXplorer. Extended version of the previous article. [10] F.M. Hill "Managing service quality in higher education: the role of the student as primary consumer", Quality Assurance in Education, Vol. 3 Iss: 3, 1995, pp.10 – 21. [11] P. Spooren and D. Mortelmans “Teacher professionalism and student evaluation of teaching: will better teachers receive higher ratings and will better students give higher ratings?” Educational Studies, Vol. 32, No. 2, June 2006, 2006, pp. 201–214. [12] E. Coşkun and M. Alkan “Evaluation of learning and teaching process in Turkish courses” International Electronic Journal of Elementary Education Vol. 2, Issue 3, July, 2010. [13] P. Ramsden “A performance indicator of teaching quality in higher education: The Course Experience Questionnaire” Studies in Higher Education, Volume 16, Number 2, 1991, pp. 129-150(22). [14] Joint Committee: The California State University, California Faculty Association and Academic Senate CSU. “Report on Student Evaluation of Teaching”. 2008. Available at: http://www.calstate.edu/AcadSen/Records/Reports/documents/Report _on_Student_Evaluations_of_Teaching.pdf. Last visit: April 2013. [15] H. T. Tagomori and L.A. Bishop “Student Evaluation of Teaching: Flaws in the Instruments”. Thought & Action, v11 n1 1995, pp 63-78. Spr [16] Z. Zerihun, W.Van Os and J. Beishuizen “Re-conceptualising approaches to the evaluation of teaching quality”. Chapter of the book: Access & Expansion: Challenges for Higher Education Improvement in Developing Countries. Cantrell, M., Kool, R. & W. Kouwenhoven (Eds.) VU University Press, Amsterdam, The Netherlands, 221 pp. 2010. Available at: http://hdl.handle.net/1871/15816. Last visit: April 2013. [17] http://www.iso.org/iso/home.html Website of the International Organization for Standardization. Last visit: April 2013. [18] www.iec.ch/ Website of the International Electrotechnical Commission. Last visit: April 2013. [19] http://standards.ieee.org/ Website of the IEEE Standards Association. Last visit: April 2013. [20] [Plaza, 2010] I. Plaza et al “Quality and innovation in Higher Education: Code of Good Practices” 40th ASEE/IEEE Frontiers in Education Conference. October 27 - 30, 2010, Washington, DC. 2010. [21] ISO/IWA 2:2007 “Quality management systems -- Guidelines for the application of ISO 9001:2000 in education”. ISO. [22] ISO/IEC 19796-1:2005, “Information technology -- Learning, education and training -- Quality management, assurance and metrics -- Part 1: General approach”. 2005. [23] ISO/IEC 19796-3:2009 “Information technology -- Learning, education and training -- Quality management, assurance and metrics -- Part 3: Reference methods and metrics”. 2009. [24] C.M. Stracke, (2009): "Quality Development and Standards in e-Learning: Benefits and Guidelines for Implementations"; in: Proceedings of the ASEM Lifelong Learning Conference: e-Learning and Workplace Learning. Bangkok (Thailand). [Also online available on: http://www.qed-info.de/downloads. Last visit: April 2013]. [25] E. Campo et al. “La evolución y adopción de estándares en la formación virtual”. CompDes 2010. 28-30 July. 2010. Available at http://www.redusoi.org Website. Last visit: April 2013. [26] I. Plaza et al. “The use of Quality Standards as Element of Innovation in Higher Education." 4ª Conferência Ibérica de Sistemas e Tecnologias de Informaçâo. Póvoa de Varzim - Portugal, 17 - 20 June. Proceedings book, 2009, pp. 567-572. ISBN:978-989-96247-0-2. [27] UNE 66178:2004. “Quality management systems. Guide for the management of process for improvement”. 2004. [28] E. Keane and IM. Labhrainn “Obtaining Student Feedback on Teaching & Course Quality” National University of Ireland, Galway. 2005. Available at www.nuigalway.ie/celt/documents/evaluation_ofteaching.pdf. Last visit: August 2011. [29] J. Richardson, “Instruments for obtaining student feedback: a review of the literature” Assessment and evaluation in higher education, Vol. 30, No. 4, 2005.pp. 387-415. [30] M. and G. Graham “The Evaluation of the Student Evaluation of Educational Quality Questionnaire (SEEQ) in UK Higher Education. Research Note” Assessment & Evaluation in Higher Education, v26 n1 Feb 2001, pp 89-93. [31] http://umanitoba.ca/computing/ist/teaching/seeqinfo.html Website of University of Manitoba (Manibota, Canada). Last visit: April 2013. [32] http://www.smu.ca/ Website of the Saint Mary’s University (Nova Scotia, Canada). Last visit: April 2013. [33] www.mta.ca/ Website of the Mount Allison University (New Brunswick, Canada). Last visit: April 2013. [34] http://www.usask.ca/. Website of University of Saskatchewan. ). Last visit: April 2013. [35] http://www.fordham.edu/ Website of the Fordham University (New York – U.S.) Last visit: April 2013. [36] http://www.schreyerinstitute.psu.edu/Tools/SEEQ Website of the Schreyer Institute for Teaching Excellence (The Pennsylvania State University – U.S.) Last visit: April 2013. [37] www.ox.ac.uk Website of the University of Oxford (U.K.). Last visit: April 2013. [38] http://www2.le.ac.uk/ Website of the University of Leicester. Last visit: August 2013. [39] http://english.semnan.ac.ir/. Website of the Semnan University (Iran) Last visit: April 2013. [40] http://www.curtin.edu.au/ Website of the Curtin University (Australia) Last visit: April 2013. [41] www.unav.es Website of the University of Navarra (Spain) Last visit: August 2013. [42] www.uvigo.es/ the University of Vigo (Spain) Last visit: August 2013. [43] http://www.upc.edu/ Website of the Polytechnic University of Catalonia (Spain). Last visit: August 2013. [44] H. W. Marsh “SEEQ: A Reliable, Valid, And Useful Instrument for collecting Students' Evaluations of University Teaching” British 718 PREPRINTS OF THE FEDCSIS. KRAK ´ OW, 2013 Journal of Educational Psychology, Volume 52, Issue 1, pages 77–95, February 1982. Article first published online: 13 May 2011 (http://onlinelibrary.wiley.com. Last visit: April 2013). [45] Tale S, Nazifi M. and, Bigdeli I. “Validation of the Iranian version of student's evaluation of educational quality questionnaire”. Journal of Behavioral Sciences, Vol. 3, No. 2, 2009 pp. 127-134. [46] M.V. Verdugo and M.I. Cal “(Teaching Assessment: SEEQ) Valoración de la enseñanza: SEEQ” Revista de Formación e Innovación Educativa Universitaria. Vol. 3, Nº 4, 2010.182-193. [47] M. Corbalan et al. “(Adaptation and Reduction of SEEQ questionnaire to know the opinion of students about teaching received) Reducción y adaptación del cuestionario SEEQ para conocer la opinión del alumnado sobre la docencia que recibe” CIDUI 2010 - New Areas of Quality in Higher Education - A Comparative and Trend Analysis. 30 June – 2 July. Barcelona (Spain). 2010. [48] E. Hervas “(SEEQ and GESTEST. Proposal of adaptation, reduction and computerization to engineering) SEEQ y GESTEST Propuesta de adaptación, reducción e informatización para ingeniería” Final Degree Project. EUPT. University of Zaragoza. [49] http://www-01.ibm.com/software/analytics/spss/ Website of IBM® SPSS® Statistics. Las visit: April 2013. [50] J.P. Lévy and J. Varela “(Multivariate Analysis for the Social Sciences) Análisis Multivariable para las Ciencias Sociales. Ed. Pearson-Prentice Hall [51] J.R.A. Santos “Cronbach's Alpha: A Tool for Assessing the Reliability of Scales” Vol. 37, Number 2 , Tools of the Trade - 2TOT3. Available at the URL: http://www.joe.org/joe/1999april/tt3.php? ref=Klasistanbul.com. Las visit: April 2013. [52] J. Nunnaly, “Psychometric theory”. New York: McGraw-Hill. 1978. [53] M. Valero-García, et al. “(Is it possible to do something else with the teaching polls?) ¿Se puede hacer algo más con las encuestas docentes? 2nd. Congreso Internacional: Docencia Universitaria e Innovación; (CUIEET Conference) Tarragona (Spain). July 2002. [54] Y. Chen and L. B. Hoshower “Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation” Assessment & Evaluation in Higher Education, Volume 28, Issue 1, pages 71-88. 2003. [55] J. A. Centra, “Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness” Jossey-Bass - San Francisco, CA, Jossey-Bass. 1993. [56] D. Cobb and V. Scott “Report of the 2010-2011 AQIP Student Evaluation of Teaching Committee”. Available at the URL: http://www.siue.edu/innovation/assessment/set/pdf/SET_Report_In_pr ocessv5_FINAL.pdf. Last visit: April 2013. [57] G. Gibbs and M. Coffey “The Impact Of Training Of University Teachers on their Teaching Skills, their Approach to Teaching and the Approach to Learning of their Students” Active Learning in Higher Education March 2004 vol. 5 no. 1, 2004.pp. 87-100. MONTSERRAT CORBALAN ET AL.: REDUCTION OF THE SEEQ QUESTIONNAIRE 719