ONLINE ASSESSMENT: MAKE IT SIMPLE

ONLINE ASSESSMENT: MAKE IT SIMPLE Margarida Amaral Universidade do Porto Porto / Portugal [email protected] Abstract There is an ongoing project on the University of Porto (U.PORTO) that aims the association of two didactical tools of information and communication technologies: elearning and computer based assessment (CBA). In order to make the convergence of e-learning with CBA there are several issues that have to be addressed: pedagogical problems in the assessment methods, technical problems in the implementation process and organizational problems that are particularly complex in an university campus like U.PORTO: large and spread. The e-Assessment topic is of general interest for those who work in e-learning, specially now in the era of Web 2.0, where many questions arise due to so many different approaches of methodology, strategies and evaluation. U.PORTO is no exception. The need of being more accurate in the assessment process when using the web, even if the teachers use a blended-learning approach, is real and work is being developed in order to try to respond to this need. Although the e-learning Unit (GATIUP) works towards different assessment methods, this particular paper focus on the implementation of online tests on campus using the University LMS – Moodle. Keywords - Assessment, online, multiple choice questions. 1 INTRODUCTION AND CONTEXT The main purpose of this paper is to give a brief description of an ongoing project at the University of Porto, which aims to articulate two areas of education and technology: e-learning and computer-based tests. The current situation of the University concerning the global e-learning Project is described below. The focus will mainly be on the outcomes and not particularly in the implementation strategy as it is not pertinent for the understanding of our current project on computer based assessment - CBA. The main focused however is the identification of some constraints that arise from an institutional approach of online assessment implementation. 1.1 U.PORTO e-Learning Project A. Implementation and Outputs Implementing an e-learning project in a large, spread and traditional university like the University of Porto, is a complex task that must involve a considerable number of actors, such as faculty managers, teachers, information systems administrators, librarians, data processing departments and, of course, students. Although we can now easily find reports of experiences carried out in scattered and decentralized large universities like the University of Sydney [1] or in the Manchester Metropolitan University [2], it was necessary several years ago for the University of Porto to find its own strategic path to address its specificities in the development of e-learning activities [3], [4]. In the school year 2003-2004, the U.PORTO began a pilot-project to develop pedagogical contents for a Learning Management System (LMS) that was increasingly integrated with the University Information System - SiGARRA. This project, hereafter “e-LearningUP”, has the following general objectives: 1. Improve the students’ learning conditions outside the classroom; 2. Improve the students’ autonomy and research capacity; 3. Work with the students to obtain from them a more energetic contribution for the learning process; 4. Give students the opportunity of having some support documents in advance such that their participation in the classroom becomes more active and consistent; Proceedings of INTED2010 Conference. 8-10 March 2010, Valencia, Spain. ISBN:978-84-613-5538-9 001550 5. Privilege strategies that use student activities’ elements, like reading and performing small tasks, which contribute to the training of hypothetical-deductive thought [5], [6]. 6. Quickly publish learning objectives, syllabus, bibliography and other documents of interest; 7. Increase the interactivity between the students and the faculty; 8. Promote a democratic access to information; 9. Increase the opportunity of access to the course contents to dislocated students; 10. Acquire experience in techniques of distance education through the Internet. E-learningUP has particular characteristics, not only because both teachers and students are oriented towards a common final objective in a wide range of areas, but also for the innovative purpose of integrating resources that are already available in an information system with a learning management system. Another pertinent fact is the policy of directing this project towards professors with technological competences as software users. One of the main objectives of this project was the future expansion of contents production to the entire professors’ community of the university. This was achieved since the pilot team of teachers involved, constitute a representative sample of the academic community with low technological skills. Periodically during the project, existing results were used to motivate other professors towards the production of contents following proved methodologies, and not to amaze them with overdeveloped multimedia resources, which might become unrealistic and even frightening for those who do not have advance technological knowledge. We would like to point out the fact of the teachers’ e-content development and instructional design on the LMS was, and still is, guided but not modeled by GATIUP. Three distinct platforms were already used – LUVIT, WebCT (both Campus Edition and VISTA) and Moodle. The LUVIT platform was abandoned in 2006, and was replaced by the Moodle platform, which was already widely used by several professors who had engaged in individual initiatives. Moodle became the unique platform of U.PORTO in 2009. The LMS is now a common resource in the university campus which students, technicians and teachers are familiar with. At the present time all Faculties are involved in the eLearningUP project, which comprises more than 621 on-line courses on Moodle. Some Faculties, like Sciences and Engineering, have their own support offices, – Moodle@FCUP offers 700 on-line courses to students; Moodle@FEUP offers about 1000 online courses. [7] By online course it should be intended a blended-learning support for presential courses, given that the University of Porto is a traditional campus-based institution. B. E-learningUP outcomes As a product of this project, several pedagogic materials and strategies for their use arose. These products are analyzed by the professors involved and appraised by the entire academic community, allowing for the sustainability of future decisions concerning actions for the continuous improvement of the quality of the universities’ courses. Five annual seminars were held, from which books of proceedings, CDs and DVDs containing the presented work were developed. Also, in 2005, U.PORTO created an internal Excellence Award for e-learning. This award intends to distinguish those that use the LMS with a proper pedagogical strategy in their teaching activities during the school year. The econtents developed for support of the course as well as communication tools or other innovative usage of e-learning is taken under consideration. [8] From the beginning of this process, the GATIUP’s philosophy included the teachers’ involvement on national and international conferences, national funding projects, as well as trying to include them in internal discussion groups about pedagogic and paradigmatic questions regarding new technologies in education. With time, a considerable number of teachers began to feel the increasing need to explore a diversity of traits, while maintaining the belief that they would complement and even improve their own experiences in e-learning with students. This is an interesting evolution of top-down process with a strong component of interest towards a middle-out process. 2 MOTIVATION FOR COMPUTER BASED ASSESSMENT Keeping in mind what was prior said about the use of electronic tools in teaching and learning, we will introduce at this point of the paper another relevant experience held in the University. For many decades multiple choice questions (MCQ) or other types of questions that allow automatic 001551 grading, were used in exams through the University. Some faculties even have a systematic digitalization by optic reading of all evaluation tests. The transformation of the optic readings into matrixes of answers allows the application of specific software which, through an iterative calculation procedure, adjusts the data to logistic equations of the IRT (Item Response Theory). The data analysis allows the measurement of difficulty and discrimination parameters for each item, as well as amount of information each item complies. Consequently, a continuously growing set of calibrated items has been built. Moreover, the developed work as been used to guide the current teaching activity of the courses. CBA was a smooth and natural evolution to these teachers: they had accumulated experience given by paper-pencil exams, and they have now the skills and knowledge to work with new technologies in the pedagogical process. If a LMS was already being used for pedagogic activities such as delivery of documents, assignments, discussions why not extend that to summative evaluation? Many teachers were using quizzes as formative evaluation tool but didn’t take the leap into the final exam. GATIUP had the role of supporting that leap: making it simple. 3 INDIVIDUAL TEACHING EXPERIENCES USING COMPUTER BASED ASSESSMENT A. How online exams were held On the school year of 2007-2008, one teacher engaged on an experience of examining 409 students using the computer and Moodle. This incredibly high number of students resulted of the merging of two different courses in one by the light of the Bologna process of redesigning the study plan. The course was on Moodle: students could access the LMS to consult documents for classes, texts, web pages, discussions and deliver assignments. Every student had access to Moodle and all the students were using it heavily. The teacher assembled on Moodle a question bank with 252 MCQ. The initial idea was to create individual exams for each student with 60 random questions. Since it was a rather new experiment in such a large scale it was decided not to engage in random assembly of exams but to make 2 different exams for two distinct background groups of students. Immediately one could identify several issues on this process that had to be managed and solved: - number of computers available - classrooms with computers connected to the internet - security issues - user identification issues GATIUP and the teaching team worked around this problem using the media available: - 3 rooms with computers connected to the internet on a total of 45 - Technical faculty support in order to provide security to the online exam On the day of the exam students appeared in groups of 45, previously assembled and posted on Moodle page of the course. One room was booked for the students leave all their belongings. One member of GATIUP accompanied the process all day. Groups of 45 students entered the room every 60 minutes with 15 minutes break to the distribution by the 3 computer rooms. On the technical side, the network was blocked only to access SIGARRA and Moodle (SIGARRA is where students authenticate their profiles and automatically enter the LMS). Also, on Moodle is possible to define the IP address of the machines that are connected to that particular exam. In addition to this every shift of students had a different password to enter the exam. There were no security fails, on authentication or during the performance of the students on the exam. The network services of the university were also engaged in this process in order to provide a stable internet connection trying to avoid any failures that might happen. This exam day started at 9:00 am and finished at 18:00pm with no lunch break. By the end 287 students were evaluated and graded. 001552 Each student could see their personal score by the time they submitted their exam. This allowed students to plan their recurring period of examinations since they knew their score in that particular course. The feedback from students was positive. The teachers were delighted with the fact of having the work done by the end of the day. It allowed the teachers to have time to spend in other pedagogical activities. Moodle also provides an item analysis tool. With a short training on how to interpret those data teachers were more aware of the quality of the questions produced for that exam. This process was repeated on the recurring period of examinations, using 60 different question of the question bank. In this period 122 students were evaluated and graded. Since this first online examination experiment other teachers already did their final examinations online always with the support of GATIUP. Some numbers: On the school year of 2008/2009, 1548 students did their final exams online; in 2009/2010 only in the first semester 869 students were evaluated and graded on Moodle. B. Main reasons that triggered the CBA The large number of students, time spent on the process (since development to classification), lack of adjusted tests, inadequate methods relating to new paradigms of teaching and learning are current problems that U.PORTO academics face. These lead to the engagement in the use of innovative assessment methods. Online assessment with automatic feedback presents itself as a possible solution to the problems listed above and also allows to the scalability, process liability, adjusted testing even when a large number of students is being evaluated, creation of new opportunities for students (accessibility) and institutions (new markets) and development of methodologies that allow a secure and liable assessment in online distance courses. This last item assumes vital importance for the University hence it’s engaging in the development of online distance courses in lifelong learning area. 4 INSTITUCIONAL SUPPORT FOR CBA IMPLEMENTATION The trust developed between the supporting unit for New Technologies in Education and the academic community is vital for the implementation of such project. Final assessment is a serious phase in the pedagogic process along a course and it has to be secure and reliable. Although the supporting team had already a sustained knowledge of educational technology, a hard work on researching and self-training in the area of computer-based assessment (CBA) and computer adaptive testing is being made. The process to extend from e-learning to CBA looks rather as a natural evolution and from the conceptual point of view doesn’t seem to bring special difficulties. However the extension to the entire campus raises scalability issues difficult to solve, of which two are especially concerning: 1. How to make this process transparent to faculty; 2. How to assure some specific competencies such as psychometrician expertise and human and physical resources are proper and reliable. It is not easy to find in the literature a report of a strategy of implementation of a computer adaptive system to a large community of academics. It is common to find detailed descriptions about CAT construction, delivery and analysis with all the mathematical models explicitly shown and explained within the context. The University of Porto intends to create a structure that contains in itself the possibility for the academics to use CBA and/or CAT in a transparent way. By this, the work will focus in creating conditions of availability of these tools for every teacher who wants to use the computer as a mean to produce, deliver and analyze a test. It is important that all the models and mathematical background are assured by a psychometrician, who must support the teachers who should not need to run through the theories and principles of a CAT system. 001553 Keeping in mind the goal of implementing CATs in more than one discipline in the University, the initial approach will be the use of CBA to evaluate students. The support office for new technologies in education traced several steps in order to pursue this goal: Training Faculty Support Staff - One of the functions of a faculty support office that is often overlooked, even though it is expensive and time-consuming, is the training of its own staff. Faculty support personnel must learn (and buy) new versions of software in a timely manner, and must frequently add adding new knowledge to their repertoire of skills and understanding. Training the academic community – building multiple choice questions, assembling a test based on the computer, IRT, principles of CAT and other subjects relevant to the understanding and execution of this project. Training of the academic staff in the development of multiple choice questions is a demand of the teachers that work in this project. GATIUP already ministered a 14 hours training course in the development of MCQ. The second edition of this course took place in January 2010 with the presence of an invited expert in Psychometrics that addressed 61 teachers on a workshop. The event was held under the punch line: “Is intuition enough to build an exam?”. It was a successful workshop which proved to be an indicator that this project, thru training is already increasing the awareness around questions construction and their quality analysis a priori and a posteriori. The teachers involved in online testing are now on the calibration phase of their questions. In addition several other teachers that still use paper-pencil in their exams are contacting GATIUP in order to analyse through Classic Test Theory their exams results in previous years. Analyzing software available on the market - the IMS Global Consortium, an industry and academic consortium, produced the IMS QTI (IMS Question & Test Interoperability Specification) to enable the exchange of question and test data and also results reports [9]. The QTI specification comprises several documents which are being taken under consideration in the decision of choosing assessment software with interoperability of question design. Ferl Becta [10], [11] lists 21 software tools for e-assessment. From that list only 4 products provide IMS QTI and SCORM support to some extent. In relation with learning platforms the scenario of interoperability doesn’t get any better. Moodle, which is used at UP, allows to export IMS QTI 2.0 format but not to import. [12],[13] Among other TDS analyzed [14] software alternatives we chose to use the Fast Test Pro [15] - Test Development System (TDS), Session Distribution Package (SDP) and Item and Test Analysis Package (ITAP). Also, we are carefully analyzing offline alternatives that can be synchronized with Moodle. This is a very important aspect because of the Moodle server load during online exams. The number of teachers and students is raising every exam season. Testing Rooms – to perform CBA it is necessary to guarantee rooms with a considerable number of computers prepared for assessment. These rooms with specific characteristics [16] to provide a distraction-free, secure testing environment with continuous candidate surveillance can be already found in some of our faculties. With consecutive experiences we expect to identify outcomes that allow us to make adjustments to our strategy in order to succeed with the development of resources for computer-based testing in campus-wide IT systems. 5 CONCLUSIONS CAT and other forms of computer based testing are obvious extensions of e-learning networks in academic settings. However, these tools require complex analysis and expensive equipment. The scientific background of CAT is very robust. Validity and reliability issues are also well characterized in many studies. However the logistics required establishing CAT at a large scale in a university campus can be daunting. Rather unexpectedly there is lack of case reports on well succeeded programs. However the learned experience of coping with the problems of establishing an e-learning project in a large, spread and traditional university like University of Porto may be useful. 001554 ACKNOWLEDGMENTS The author wish to thank Professor Daniel Moura, for his help and advice. The contribution from our teachers, technical personnel and students are warmly acknowledged. References [1] R.A. Ellis, N. Jarkey, M.J. Mahony, M. Peat and S. Sheely, “Managing Quality Improvement of eLearning in a Large, Campus-Based University,” Quality Assurance in Education, vol. 15, pp.923, 2007. [2] Anonymous, “Manchester Metropolitan University,” Technical Report Megatrends in eLearning Provision NO/05/C/F/RF-83250, Norsk Kunnskaps-Institutt, Bekkestua, Norway, available at http://www.nettskolen.com/in_english/megatrends/MMU_article.pdf, Mar. 2007 [3] A. Soeiro, M.L. Pinto and E. Pereira, Pedagogia e Novas Tecnologias: Estudos de Casos Universitários, CDROM, University of Porto Ed, 2000. (in Portuguese) [4] R. Falcão, “E-learning at the University of Porto from 1998-2005: Implementing E-learning at a Traditional University,” available at http:www.ai.tuwienac.at/elearning/innovation_lecture, Dec. 2006. [5] A.E. Lawson, Science Teaching and Development of Thinking. Belmont, Calif.: Wadsworth, pp. 261-299, 1995. [6] A.E. Lawson, The Neurological Basis of Learning, Development and Discovery: Implications for Teaching Science and Mathematics. Dordrecht, The Netherlands: Kluwer, pp. 225-260, 2003. [7] [email protected], Mar. 2008; https://sigarra.up.pt/up/web_base.gera_pagina?P_pagina=18385 (in Portuguese) [8] Senado da Universidade do Porto, Prémio Excelência e-LearningU.Porto, Mar. 2007; http://sigarra.up.pt/reitoria/LEGISLACAO_GERAL. ver_legislacao?p_nr=143 (in Portuguese) [9] IMS Global Learning Consortium. IMS Question & Test Interoperability Specification, Dec.2006 ; http://www.imsglobal.org/question/ [10] Ferl Becta, “On-line Assessment software”, Nov. 2006; http://ferl.bect.org.uk/display.cfm?page=193 [11] S. Suleman, “E-assessment”, Ferl First, BECTA, UK, 29 Apr 2003, Nov. 2006; http://ferl.bect.org.uk/display.cfm?resid=5367&printable=1 [12] Moodle Statistics, Dec. 2006; http://moodle.org/stats/ [13] Moodle. Release Notes, Dec. 2006; http://docs.moodle.org/en/Release_Notes [14] E.J. Gibson, P.W. Brewer, A. Dholakia, M.A. Vouk & D.L. Bitzer, “A comparative analysis of Webbased testing and evaluation systems.”, Proc. 4th WWW conference, Boston, 1995; http://renoir.csc. ncsu.edu/MRA/Reports/WebBasedTesting.html [15] FastTest Pro. (2000). [Computer program]. Assessment System Corporation, Mar. 2008; http://www.assess.com/FastTEST.html [16] Pearson Vue: Become a test center; Jan. 2007; http://www.pearsonvue.com/pvtc/facility_reqs.pdf 001555