Contrastive perspectives on the reliability of grammar testing: The case of the verb phrase in L2 English

ELIA 3, 2002, pp. 33-48 CONTRASTIVE PERSPECTIVES ON THE RELIABILITY OF GRAMMAR TESTING: THE CASE OF THE VERB PHRASE IN L2 ENGLISH Geert Jacobs (University of Antwerp) Chis Braecke (University of Antwerp) In this article we present the preliminary results of empirical research on the reliability of grammar testing. The following testing techniques are examined: multiple choice, translation, gap-filling and, finally, performance. In particular, for the purposes of this paper we set up four different tests to identify knowledge of the English verb phrase among native speakers of Dutch. Drawing from a detailed examination of selected results it is concluded that a renewed interest in a contrastive approach to language testing and practice is to be recommended. Key words: grammar testing, contrastive analysis, L2 English, the English verb phrase. 1. Introduction This article reports on the preliminary results of an empirically-based investigation into selected grammatical issues in advanced-level non-native English. In particular, we set out to examine the reliability of the various techniques that we used to identify knowledge of the English verb phrase among Dutch-speaking first-year undergraduates at the Faculties of Applied Economics at two Belgian (i.c. Flemish) universities. For our purposes, reliability can be defined as the consistency of results from one test to another (cf. Bachman & Palmer 1997). Estudios de lingüística inglesa aplicada 34 G. Jacobs and C.Braecke ELIA 3, 2002, pp. 33-48 Within the framework of error analysis, the following testing techniques will be dealt with: - multiple choice - translation from Dutch (L1) into English (L2) - gap-filling (contextualized completion assignments) - performance: i.c. information-gap and decision-making dyads between testees1. The first three techniques are based on written tasks and they were used in different tests with different sets of between 400 and 500 testees. As far as the results are concerned, we will be presenting percentages, some of which are based on representative subcorpora. No statistical analysis has been carried out. The fourth and final test is based on oral communication: we used audio-recordings of dyads involving some 35 testees. Crucially, the present article is meant to provide no more than a first exploration into what the pros and cons of the different techniques could be. That’s why, from the wide range of data available, we will only focus on the following four specific features of the grammar of the English verb phrase, all of which tend to be problematic for speakers of Dutch: viz. - information questions (or wh-questions) - the past and past participle inflections of irregular verbs - the continuative use of the present perfect - absence of obligation 1 Egÿud & Glover (2001) argue that such dyads lead learners to produce highly varied language. In addition, Garcia Mayo & Pica (2000) have shown that they are not significantly different from dyads between learners and native speakers. Constrastive perspectives on the… 35 ELIA 3, 2002, pp. 33-48 For all of these features of the grammar of the English verb phrase, the problems for native speakers of Dutch have been sufficiently described (cf. Zandvoort’s 1975 Handbook of English Grammar as well as, more recently, Vandenbergen, 1991 and Lemmens & Parr, 1995). Therefore, the purpose of our tests, while diagnostic, was not to make an inventory of problems but rather to point to a possible hierarchy of problems as well as formulating tentative hypotheses for refining the description. 2. Information questions To start with, we shall look at information questions in which the question word is not the subject, as in translation-based multiple choice test (1). Since this is a part of basic grammar, few problems are to be expected, except perhaps for the use of the operator, which is unknown in Dutch. (1) Wat betekent dat Engelse woord? a. What does that English word mean? (82%) b. *What means that English word? (4%) c. *What does that English word means? (14%) The results are predictable and surprising at the same time. Predicable, because most of the testees opted for a. But surprising, because the major source of mistakes was not the use of the operator, but the inflection of the main verb. Interestingly, translation (2) – with no possible answers suggested to the testees – yielded rather different results: 36 G. Jacobs and C.Braecke ELIA 3, 2002, pp. 33-48 (2) Wat betekent dat Engelse woord waarover je gisteren sprak? [What does that English word mean you talked about yesterday?] First of all, it should be pointed that one out of six testees came up with a rather free translation, viz. a paraphrase, in which the problem of the use of the operator was avoided: (2a) What is the meaning of that English word you talked about yesterday? (17%) Of course, from a communicative point of view this answer is perfectly adequate but we should keep in mind that it does point to a first restriction in using translation to test students’ knowledge of specific items of grammar (cf. Ellis 1998 for an overview of research on avoidance strategies in language learning and language testing). Let’s return to (2) now. The interesting thing is that this time only few testees came up with: (2b) *What does that English word means you talked about yesterday? (4%) In contrast, many more failed to use the operator, which points to the L1 transfer that we were expecting: (2c) *What means that English word you talked about yesterday? (18%) Indeed, it looks as if, by offering the right alternative, multiple choice seems to underestimate the testees’ real problems (in this case the use of the operator). Its role seems to be limited to that of checking whether testees recognize the right form rather than testing active grammar knowledge. At the same time, the question can be raised whether presenting ungrammatical alternatives does not lead testees to make mistakes that they would not normally make (-s form). Constrastive perspectives on the… 37 ELIA 3, 2002, pp. 33-48 Alternatively, it remains to be seen if in (2) – as for the use of the operator - L1 transfer was not induced by the translation format. The results for gap-filling test (3) seem to suggest that this is not the case, since they are almost identical to those for the translation test: (3) What observable truth _____ (CONTAIN, IT)? (3a) *What observable truth contains it? (14%) (3b) *What observable truth does it contains? (4%) Apparently, even in the gap-filling assignment, testees seem to resort to a translating strategy. This is in line with Mattar (1999), whose research actually reveals less mother tongue interference with translation tasks than with gap-filling. Let’s return to the use of the operator. Interestingly, in contrast with the results for information questions, our data show that the testees don’t have a problem with the use of the operator in simple yes-no questions, like “Do you agree?” Hardly anyone comes up with “Agree you?” Why this difference? We believe that the answer may lie in the second type of information questions, viz. those in which the question word is the subject. For these no ‘do’ is required, as in gap-filling test (4): (4) Who _____ (BELIEVE) that wars can really solve humanitarian problems? Still, quite some testees used do-support here: (4a) *Who does believe that wars can really solve humanitarian problems? (4%) Strangely, even more used do-support in translation test (5), where L1 transfer should have led testees away from it: 38 G. Jacobs and C.Braecke ELIA 3, 2002, pp. 33-48 (5) Wat kost er meer? (5a) *Which does cost more? (7%) These are relatively small numbers but in our view it is surprising that any mistakes were made at all because, this time, they cannot be related to L1 transfer. Unless, of course, testees mix up information questions in which the question word is the subject (and for which no do-support is used) with those in which the question word is not the subject (and for which do-support is used). If this is the case, then it seems to reconfirm the need for a contrastive approach to selected issues of grammar and of course translating should be part of it, both for practice and testing. In the next section, we shall indicate that a contrastive approach need not (and indeed should not) be restricted to translating. 3. Past and past participle inflections of irregular verbs In various gap-filling exercises we tested knowledge of the past and past participle inflections of irregular verbs. In the following table it is shown how many testees made a mistake for each of them. In presenting the verbs we have followed Quirk et al’s (1985) morphological classification2. 2 Quirk et al (1985) use the following three criteria: - suffixation (i.e. the simple past and past participle include a suffix) - V-ed identity (i.e. the simple past and past participle are identical) - Vowel identity (i.e. the infinitive, the simple past and the past participle show no difference in base vowel) Note that on the basis of these criteria Quirk et al (1985) have seven classes. For the purposes of this article we have brought Quirk et al’s first and second classes together in our first class and we have renumbered their third class as our second, their fourth class as our third, etc. Constrastive perspectives on the… 39 ELIA 3, 2002, pp. 33-48 Class 1: suffixation, V-ed identity, vowel identity BUILD 21% LEND 44% SEND 54% Class 2: suffixation, V-ed identity BUY 8% FLEE 47% KEEP 0%, 3% MEAN 18% SEEK 18% Class 3: suffixation BREAK 4% DRIVE 5% FORGET 0% KNOW 5% Class 4: V-ed identity, vowel identity SET 5% Class 5: V-ed identity GET 4% LEAD 75% MEET 1% READ 6% UNDERSTAND 4% Class 6: / BEGIN 12% RING 4% 40 G. Jacobs and C.Braecke ELIA 3, 2002, pp. 33-48 Apparently, class 1 is the only class that poses major problems throughout. Certainly – and perhaps surprisingly -, those irregular verbs that have vowel variation (or so-called ‘ablaut’) seem far more simple, except for relatively unusual verbs like ‘seek’ and ‘flee’. The high scores for ‘mean’ and ‘lead’ can be related to confusion about spelling. Generally speaking, though, vowel variation seems relatively easy. But why then do the verbs in class 1 prove so problematic? We think the problem lies in devoicing the final phoneme: while vowel variation is a loud and clear indicator, one which speakers of Dutch also know in their L1, devoicing the final phoneme is more subtle and completely unfamiliar to them. It follows that Dutch-speaking learners of English should be made aware of the different classes of irregular verbs in general and of the voicedvoiceless opposition for a number of the class 1 irregular verbs in particular. There are two interesting points to be made about this contrastive issue: it was revealed by a gap-filling assignment (and not by translation) and it falls beyond the reach of translation-based practice. 4. The continuative use of the present perfect The continuative use of the present perfect refers to the use of the present perfect (simple or progressive) ‘to invoke a state of affairs that began in the past, has continued until the speaker’s now and may be expected to endure into the future’ (Mackenzie: 1997: 44). Dutch traditionally uses a simple present here (Lemmens & Parr: 1995: 36). This continuative use of the English present perfect was tested in the final two sentences of translation (6): (6) Toen hij jong was, heeft hij een jaar als kelner gewerkt. Toen is hij verhuisd naar Londen. Daar studeert hij nu al sinds twee jaar aan de universiteit. Hij heeft er een nieuwe flat sinds april. [When he was young, he worked as a waiter for a year. Then he moved to London. There he has studied/been studying at university for two years. He has had a new flat since April.] Constrastive perspectives on the… 41 ELIA 3, 2002, pp. 33-48 The results are predictable: (6a) *There he studies at university for two years. (42%) (6b) *There he is studying at university for two years. (24%) (6c) *He has a new flat since April. (67%) Again the question can be raised whether or not this type of error was induced by the translation format of the test. So let’s also look at gap-filling exercise (7): (7) Since Nato’s air campaign against Slobodan Milosevic got underway, Macedonia – a small, impoverished state that broke away from Yugoslavia in 1991 – ____ (FACE) a nightmare. Present perfect 30% Simple 15% Progressive 15% Present 49% *Simple 17% *Progressive 32% Past 18% Here, still a lot of students opt for the present. Just like for information questions, it looks as if the effect of translation should not be exaggerated. But there are two further remarks to be made. First, quite some students decided to select a past, which indicates that the extract was misunderstood as representing a past time frame. This seems to be a recurring problem with discourse-based gap-filling exercises. While contextualization is more important for some grammatical issues (e.g. tenses) and less so for others (like inflection of irregular plurals), it should be pointed out that it can also complicate matters. 48 G. Jacobs and C.Braecke ELIA 3, 2002, pp. 33-48 References Bachman, L.F. & Palmer, A.S. 1997. Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: OUP. Berry, R. 2000. “’Youser-friendly’ Metalanguage: What Effect Does it Have on Learners of English?” IRAL 38: 195-211. Együd, G. & Glover, P. 2001. “Oral Testing in Pairs”. ELT Journal 55: 7076. Ellis, R. 1998. Second Language Acquisition. Oxford: OUP. Garcia Mayo, M. & Pica, T. 2000. “L2 Learner Interaction in a ForeignLanguage Setting: Are Learning Needs Addresssed?” IRAL 38: 35-58. Lemmens, M. & Parr, T. 1995. Handboek voor de Vertaler NederlandsEngels. Leuven: Wolters. Mackenzie, J.L. 1997. Principles and Pitfalls of English Grammar. Bussum: Coutinho. Mattar, H. 1999. “Translation Elicitation Techniques and Mother-Tongue Interference: Any Significant Connection?” IRAL 37: 307-320. Nunan, D. 1998. “Teaching Grammar in Context”. ELT Journal 52: 101-109. Petrovitz, W. 1997. “The Role of Context in the Presentation of Grammar”. ELT Journal 51: 201-207. Stenson, N. 1991. “Induced Errors” in B.W. Robinett & J. Schachter (eds.) Second Language Learning: Contrastive Analysis, Error Analysis, and Related Aspects. Ann Arbor: The University of Michigan Press. Vandenbergen, A.-M. 1991. Exploring English Grammar. Leuven: Garant. Zandvoort, R.W. 1975. A Handbook of English Grammar. 7th edn. London: Longman.