Following-Up on Feedback through Repetition inAssessments (in the Science Disciplines)

AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.1 Following-Up on Feedback through Repetition in Assessments (in the Science Disciplines)* Séamus McLoone National University of Ireland Maynooth Ireland Abstract In practice, the student learning cycle involves learning, assessment and feedback. The student learns about a particular topic through a combination of lectures, research and/or action. They are then assessed on what they have learned and are given formative feedback on their performance. Unfortunately, the cycle seems to end at this point. A new topic is presented and the learning cycle repeats. This is typical of a module in which several different topics are covered over the course of a semester. However, some obvious questions emerge. What do the students do with the feedback? Do they really learn from it? How do we know? This paper proposes to follow-up on feedback by simply ensuring that a proportion of the next assessment covers material that students had difficulty with on the previous one. Students are informed of this fact so as to encourage them to actively engage with the feedback. Test results show a marked improvement in the students’ performance, particularly in the case of the repeated material, while student feedback responds favourably to this method of ensuring that the student learning cycle is properly completed. Details of the assessments, test results and student feedback, as well as some personal observations and discussions, are presented within. Keywords: assessment, feedback, the student learning cycle * Research Article. URL: http://ojs.aishe.org/index.php/aishe-j/article/view/5 All Ireland Journal of Teaching and Learning in Higher Education (AISHE-J) Creative Commons Attribution-NonCommercial-ShareAlike 3.0 AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.2 1. Introduction “Assessment is at the heart of the student experience” according to Brown and Knight (1994, p.1). As eloquently described by Brown (2001, p.4), assessment “defines what students regard as important, how they spend their time and how they come to see themselves as students.” The literature clearly highlights that assessment is indeed a key element in student learning – “if you want to change student learning then change the methods of assessment” (G. Brown et al. 1997, p.7). Shuell (1986, p.429) states that “what the student does is actually more important in determining what is learned than what the teacher does.” Rightly or wrongly, students focus more on what is assessed rather than on what is taught. As such, assessment should be regarded as an intrinsic part of the learning process and, along with the teaching method, should be carefully aligned with the learning outcomes in order to support effective student learning. This process of constructive alignment is fundamental to effective teaching (Biggs 1996, pp.347-364; Biggs 2002; Biggs 2004, pp.11-33). The issues associated with assessment, ranging from what assessment is to providing effective assessment, are well documented in the research literature (Knight 1994; Harris & Bell 1994; Heywood 2000; Race et al. 2005; Angelo & Cross 1993; G. Brown et al. 1997; Moon 2002). One important aspect that is evident throughout the literature is the distinction between the two types of assessment, namely formative and summative. The former is mainly used to improve student learning while the latter is used to identify what the student has learnt (Rust 2000). In general, formative assessment and the corresponding feedback is most effective when given throughout a semester, so that students can learn from their mistakes. This involves giving the student feedback several times over the course of a semester so that the student can improve upon future performance. Summative feedback is generally associated with the final exam at which point the only feedback a student receives is that of a final grade. In strict terms, formative assessment does not contribute to the final module grade. However, in practice, assessments are often designed to provide a mixture of both formative and summative feedback. As teachers, it is important that we strive for more formative feedback in our assessment by making more use of continuous assessment and moving further away from the final exam scenario. Formative feedback is a vital factor in supporting student learning. Race (1996) feels that lecturers can help learning to take place by providing feedback to students and helping them make sense of what they are learning. Gosling (2003, p.172) states that providing feedback is one of “the most important aspects of supporting student learning”. This sentiment is supported by Rust (2002, p.149) when he states that assessment aids student learning through feedback. Brown et al. (1995, p.81) state that the key to using assessment as the “engine for learning” is to make sure that each assignment provides plenty of opportunities for students to receive timely positive feedback. Ramsden (1992, p.99) lists appropriate assessment and feedback as one of the “six key principles of effective teaching”. Harris and Bell (1994) believe that “assessing should have a strong formative role” and, in addition, feel that in order for assessing to be truly formative, it should involve discussions between the lecturer and the students. AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.3 As already mentioned, assessments in practice strive to provide a blend of both formative and summative feedback. However, it is arguable that most assessments focus too much on the summative aspect and not enough on providing students with formative feedback. Moreover, even in the cases, where formative feedback is to the fore, another problem emerges. Brown (2001) argues that “unfortunately, the feedback provided is not always read, let alone used by the students”. He follows this point with a suggestion that students need to be taught how to use the feedback and that we, as lecturers, need to check that the students have used feedback from their previous assignments. Yorke (2002, p.39) feels that lecturers need to spend time helping students to use the feedback they receive. The current research literature contains numerous articles relating to the topic of feedback, ranging from collecting feedback to using and acting on feedback to help students learn (Hogarth et al. 2003; G. Brown 2001; Race 2005). The ‘Study Guides and Strategies’ web site1 provides a useful guide to students on how they themselves can make effective use of the feedback given to them. Similarly, Race (1999) offers advice to students on “using feedback to make action plans”. However, there appears to be, from what I can ascertain, a dearth of information relating to Brown’s suggestion on checking up on the students’ use of the feedback given. In other words, have they fully appreciated the feedback received? Will they use it correctly in relevant future circumstances? How do we know? This process of checking-up on the student’s use of feedback can be viewed as the final component of the student learning cycle. In simple terms, the full cycle involves learning, assessment, feedback and improvement. This fits closely with the process model of experiential learning described by Dennison and Kirk (1990). This model, based on the model by Kolb (1984), clearly identifies four key steps. These can be viewed as the student doing a new piece of work, reviewing their efforts through assessment, learning from their mistakes through feedback and applying their newly acquired information in a future context. It is this latter step that is typically missing in current teaching practice, with the exception of the final exam, in which a student has a chance to apply their knowledge. However, this occurs too late and is arguably of little or no benefit to the student’s learning. As teachers, it is our duty to provide an environment in which students are supported and encouraged to learn. As part of this, it is important that we ensure that the student learning cycle is complete. In other words, we need to follow the “do-review-learn-apply” process model of learning in its entirety. As such, it is important that we check that students are making use of the feedback we give them and that they apply this feedback in a correct fashion. I feel that a useful method for achieving this is to have repetition of material across several assessments. Here material can refer to both content and skill. It is the aim of this paper to show that repeating material across multiple assessments will encourage students to follow-up on the feedback given to them. The results obtained suggest that this is indeed the case. The rest of this paper is structured as follows. The next section outlines the methodology used for this research. A brief overview of the educational situation is also provided. Test results and analysis are outlined in section 3. This section also includes both student and peer feedback. The paper ends in section 4 with some conclusions and suggestions for future work. 1 “Since 1996 the Study Guides and Strategies web site has been researched, authored, maintained and supported by Joe Landsberger.” URL: http://www.studygs.net/ AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.4 2. Methodology I teach a mathematical module on optimisation, as part of the Bachelor of Engineering degree programme in the Department of Electronic Engineering, National University of Ireland (NUI) Maynooth. While this module takes place in the final year of the degree programme, it is nevertheless the first one on the topic of optimisation. As a result, a significant proportion of the content can be regarded as elementary in terms of the discipline. This module is structured to have three lectures per week for an eight week period. There are three class tests that take place on the third, fifth and seventh week respectively. It is these class tests that I used to carry out the research work for this paper. In 2008, the class consisted of eight international students and six Irish students, two of which were mature. The overall gender balance was 3 females to 11 males. Such figures represent a typical breakdown of most Electronic Engineering classes at NUI Maynooth to date. I use multiple class tests (one form of continuous assessment) for several reasons: they allow students to monitor their own learning; they provide students with the opportunity to receive feedback and, hence, improve their own learning; they encourage students to work consistently throughout the semester; they potentially improve student motivation, particularly when the class tests contribute towards the final module grade (McLoone 2003, pp.64-67). The class tests contain a mixture of questions, ranging from straightforward surface learning type questions to more complex ones which foster a deep learning approach, where the concentration of such questions depends mainly on the type of material being assessed. With the latter question style, it is hoped that students will develop a “good understanding, good long-term recall and better marks” (G. Gibbs 1995). In general the class tests are designed to cover different sections of the notes in a given module. Previously, there was no overlap from one test to the next in terms of the material being assessed. This was especially true in the case where one section of the notes is quite different to another. The problem, therefore, was that when I gave feedback to my students, I did not actually follow-up on it. Hence I did not know if the students had gained any benefit from the feedback or, indeed, if I needed to give further feedback to them. In an effort to remedy this problem, I decided to carry out the following procedural steps in relation to the three class tests for my Optimisation module: 1. Conduct class test 1 in Week 3. 2. Correct class test 1 and provide individual and group feedback to students, where group feedback relates to common mistakes made by several of the students. 3. Gather feedback from students regarding the first class test. 4. Inform students that class test 2 will contain a proportion of questions relating to feedback from class test 1. 5. Conduct class test 2 in Week 5. 6. Correct class test 2 and, once again, provide individual and group feedback to students. 7. Gather feedback from students regarding the second class test, with particular relevance to the questions relating to material covered in class test 1. 8. Conduct class test 3 in Week 8. AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.5 This methodology closely aligns with the cyclical nature of the action research model proposed by Elliot (1991). All class tests were conducted online using the Moodle virtual learning environment1. The tests consisted of eight or nine questions with varying length and difficulty. Each test lasts exactly one hour and contributes 5% to the final module grade. Sample questions from the class tests are shown below. Question X1 Consider the function f(x) = 2x2 + 8x + 1. Given the initial bracketing interval [-3, 1] and an error tolerance of 0.02, answer each of the following questions. Where applicable, give your answer to 3 decimal places. (i) applying one iteration of the Fibonacci search method would give the reduced interval of: [ , ]T (ii) without actually calculating the final local minimum estimate produced by the Fibonacci search method, we can still say that this value would have a maximum local error of (iii) had we,from the start, used the Golden Section search method instead (with a golden ratio of 0.618), the length of the first reduced interval would be (iv) again, without actually calculating the final local minimum estimate produced by the Golden Section search method, we can still say that this value would have a maximum local error of Question Y1 Consider the function f(x) = 3x3 + 1.5x2 + 3.5. We want to estimate the local minimum for this function using the Grid line search method. Select, from the following list, ALL the suitable starting bracketing intervals: Choose at least one answer. a. [-3, -1] b. [-0.33, 0.1] c. [-1, 1] d. [-2, -1] e. [-0.33, -0.1] f. [-0.1, 0.1] 1 URL: http://www.moodle.org/ AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.6 Question X2 Consider the function f(x) = 3x2 + 5x + 4. Given the initial bracketing interval [-3, 2] and an error tolerance of 0.1, answer each of the following questions. (i) applying one iteration of the Fibonacci search method would give the reduced interval of [ , ]T (ii) applying one iteration of the Golden Section search method, using a golden ratio of 0.6, would give the reduced interval of [ , ]T (iii) applying one iteration of the Grid search method, using 5 divisions, would give the reduced interval of [ , ]T (iv) that actual minimum of f(x) is Question Y2 The Quadratic search method is to be applied to the function f(x) = 8x2 + 4x + 3. Select from the following list ALL suitable sets of starting points for this approach: Choose at least one answer. a. -1, 0 and 1 b. -1, 0 and 2 c. 0, 1 and 2 d. -3, -2 and -1 e. 0, -2 and 2 f. -10, -5 and 20 g. None of the above Questions X1 and Y1 are taken from class test 1 while questions X2 and Y2 are taken from class test 2. These particular questions were chosen as they represent material that students struggled with on the first class test. The remainder of the questions on class test 2 related to new material. In the next section, the results from the class tests are summarised and analysed. AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.7 3. Results and Analysis 3.1 Test results The average result for each of the class tests are given in Table 1 below. The first class test had no prior feedback given and, thus, can arguably act as the control parameter. The other two tests were subject to some prior feedback, relative to the class performance on the previous test. As such, these tests can be viewed as the test parameters. At first glance, the process of giving feedback and partly re-assessing the same material seems to have worked very well, with a significant increase in the class average from class test 1 to class test 2. However, it is important to consider these results in the appropriate context. Class Test 1 2 3 Class Average (%) 41.4 70.4 69.0 Table 1: Class Test Results Firstly, it should be noted that the students were relatively unprepared for the first class test. They did not know what to expect in terms of questions but, more importantly, they did not know what to expect from an online class test. It was the first time for all the students to take an online class test and not only did they have to face the challenge of the test itself but they also had to develop a new strategy for coping with an online assessment. Interestingly, one student reported that they felt that the idea of looking at the computer screen for the question and then working out the answers on paper and then typing the answer on the computer was not the ideal test scenario for them. For class tests 2 and 3, they knew what to expect in terms of the test format and therefore could concentrate completely on the test itself. I feel that this was one of the key reasons for the vast improvement from the first class test to the others. In my previous year of teaching this module, I had a similar increase in marks from class test 1 to 2. The class at the time, consisting of 13 students, obtained an average of 53% in the first class test and 78% in the second one. In this case, there was no repetition in the material assessed and yet the students’ performance improved dramatically from one test to the next. Feedback from those students was similar in nature to the current feedback, i.e., they weren’t used to the format of the online class test. Some had requested a trial run of the online class test before the real thing. This is something that I intend implementing on a trial basis in the future. Secondly, it’s important to note that the class size was 14 students and therefore, with such a relatively small sample, it is difficult to draw any absolute conclusions from the results given. Future experimentation on this topic should provide an increase in data and better verification of conclusions. A more interesting and relevant set of results can be obtained by looking at a specific comparison between two questions in class test 1 (questions X1 and Y1) and a related two questions in class test 2 (questions X2 and Y2). These sets of questions, while not identical, covered similar material which was studied prior to class test 1, i.e., the two questions on class test 2 were re-assessing the material which, on average, the students struggled with on AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.8 the first class test. The students were given feedback regarding their mistakes on these questions after class test 1. The accumulative marks obtained for these two specific questions were recorded for each student for both class tests and are given in Table 2 below. For the purposes of clarity, the difference between the two sets of marks is plotted for each of the 14 students in Figure 1. These results clearly show that all but 2 of the students improved their marks in relation to the re-assessed material. Thus, these students clearly used the feedback given to them in a positive manner. It is interesting to note that student 3 shows the biggest increase in marks obtained. In this case, the student obtained almost zero marks in the aforementioned questions in the first class test and almost full marks on the second attempt. At the other end of the spectrum, 2 of the 14 students received a worse grade at the second time of asking. While the questions on class test 2 were not identical to that of class test 1, I still find it surprising that some students could do significantly poorer on the second attempt. Perhaps it is possible that the two students in question concentrated more on the new material to the detriment of the older material. After all, their overall individual grade for class test 2 showed vast improvement over their respective class test 1 grades. This is an issue for future study. Student No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 % Mark for two specific questions on Class Test 1 70 0 10 20 30 10 30 20 70 27 30 70 70 60 % Mark for two specific questions on Class Test 2 100 25 98 59 64 22 5 73 53 86 65 79 100 52 % Difference 30 25 88 39 34 12 23 53 -17 59 35 9 30 -8 Table 2: Comparison of Test Results for two specific questions Figure 1: Percentage difference in marks from class test 1 to class test 2 for two specific questions Student No. Percentage difference -40 -20 0 20 40 60 80 100 1 2 3 4 5 6 7 8 9 10 11 12 13 14 AISHE-J Volume 1, Number 1 (Autumn 2009) Page 5.9 3.2 Student feedback After class test 2, I asked the students to provide feedback on the effectiveness of my action research idea. In summary, the students found that repeating examined material from class test 1 was quite helpful (average rating of 3.86 out of 5) and found that the feedback from class test 1 was very helpful (average rating of 4.43 out of 5). No student responded negatively to either aspect. Due to time constraints, I was unable to repeat the questionnaire after class test 3, but nevertheless, one interesting event did take place. One of my students came to me to seek clarity on one of the questions from the previous class test on the morning that class test 3 was taking place. He did not ask any questions about the new material. The idea of having questions on previously covered material clearly had the desired effect on this student. 3.3 Peer Feedback Several of my peers showed positive interest in my research idea and concurred with my assessment that, in practice, we typically fail to follow-up on feedback. This was eloquently summed up by one member who remarked that “… we do the assessment, we provide the feedback and we tick the box – our work is done … ”. Unfortunately, too many teaching staff do their job because they have to and not because they want to and, as a result, their attitude sometimes becomes one of doing the bare minimum. One critical friend also expressed how simple my idea was and yet how effective it could potentially be. The fact that the idea was so surprisingly simple made it even more effective and some of the teaching staff, particularly those who conduct multiple assessments, felt that it was something that they could easily implement since it was not an overly time-consuming exercise. An interesting comment was mentioned in relation to the repetition of assessed material, as one of my peers wondered if I was excessively penalising a student for effectively making the same mistake twice. This would not be an issue if I was carrying out strictly formative assessments, which did not contribute to the final module grade. Since my assessments are both formative and summative, where the assessed marks do contribute to the final grade, the raised concern warranted some discussion. However, we both felt, on reflection, that awarding marks for the repeated assessment strongly encouraged students to actively engage with the feedback. Furthermore, as the repeated section constituted a relatively small part of the overall assessment, we both felt that any penalties encountered by a student were minimal. 4. Conclusions and Future Work The work in this paper clearly shows that the idea of following-up on feedback through repetition of assessed material has potential value. My fellow teaching staff felt that it was a worthwhile exercise and my students wholeheartedly agree. More importantly, the results from the class tests certainly showed a marked improvement from one test to the next in relation to the repeated material. However, it should be noted that familiarisation of online testing may also have impacted on this improvement. Future work will determine the degree to which this is indeed the case.