Picture Books as Multimodal Source Texts : Contrastive Analysis as a Tool for Depicting Image-text Interaction

Ketola, A. (2017). “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”. Estudios de Teoría Literaria. Revista digital: artes, letras y humanidades, 6 (12), 49-60. Esta obra se encuentra bajo una Licencia Creative Commons Atribución-NoComercial-CompartirIgual 4.0 Internacional. 49 Libros ilustrados como textos de partida multimodales El análisis contrastivo como herramienta para mostrar la interacción imagen-texto Picture Books as Multimodal Source Texts Contrastive Analysis as a Tool for Depicting Image-text Interaction Anne Ketola 1 Recibido: 17/08/2017 Aceptado: 30/08/2017 Publicado: 08/09/2017 Resumen En este artículo nos proponemos: i) mostrar el rápido crecimiento de un nuevo ámbito de estudios dentro del amplio rango cubierto por los Estudios de Traducción, conocido como Estudios Multimodales; ii) describir la interacción imagen-texto de las traducciones finlandesas de The Tale of Peter Rabbit de Beatrix Potter, desde un punto de vista multimodal; iii) indagar en el modo en que las imágenes en las traducciones de este libro ilustrado pueden haber afectado las opciones léxicas realizadas por su traductor; iv) analizar la relación entre las unidades léxicas en los textos de origen y de partida, y luego comparar estas observaciones con la información ofrecida por la imagen que corresponde a cada unidad; v) finalmente, determinar si es posible afirmar que las imágenes motivan los modos en que las unidades léxicas se han visto traducidas. Palabras clave Traducción; estudios multimodales; libros ilustrados; imagen-texto. Abstract The purposes of this article are: i) to show the rapid increase of interest in a new field of studies within the large scope of Translation Studies, known as Multimodal Studies; ii) to describe the image-text interaction of the Finnish translations of Beatrix Potter’s The Tale of Peter Rabbit, from a multimodal point of view; iii) to look into how the images in the translations of this picture book may have affected its translator’s word choices; iv) to analyze the relationship between lexical items in the source and target texts, and then compare these observations against the information offered by the image with which the item is presented; v) finally, to determine whether an explanation can be derived from the images as to why a certain item was translated the way it was. Keywords Translation; multimodal studies; picture books; image-text. 1 PhD Student. University of Tampere, School of Language, Translation and Literary Studies. Finland. Contact: anne.ketol[email protected] Anne Ketola 50 Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 Introduction he study of multimodality approaches communication, representation and interaction as always consisting of different modes, such as spoken or written language, still or moving images, gestures, sound, or gaze. During the last decade or so, multimodality has been the subject of rapidly increasing research interest in a variety of fields of study, including Translation Studies [TS] (see e.g. Kaindl 2004, Taylor 2003, 2004, and Hirvonen y Tiittula 2010). Picture books [PB] are pronouncedly multimodal; their message is created in the interaction of words and images. However, the multimodality of PB has only very recently begun to be addressed in research (Moya 2010, 2013). Extensive PB research carried out over the last decades, recent developments in the study of multimodality and TS are fields that need to be linked through converging quests. A multimodal source text [ST] consisting of written word and image, such as a PB, can be thought of consisting of two separate parts, the verbal ST and the visual ST, which nonetheless operate seamlessly to create the text’s overall meaning. The translators of such texts interpret information offered by two separate modes. Furthermore, the information conveyed by these two modes is always different in at least some aspects. The words of a PB can effortlessly express what things and characters are called, what the characters think and say, when events take place, and so on (Graham 2004: 211). Unlike images, words may also convey causal and temporal information (Nikolajeva 2002: 97). Images, for their part, can be used to express what the characters and locations look like exactly. They may also quite accurately convey the characters’ emotions and the relationships between them by showing their facial expressions, body language and the distances between them (Nikolajeva 2002: 93). In fact, words and images may convey the same information only very superficially (Nodelman 1988: 193). When presented together, words and images may afford new kinds of meaning. As PB theorist Perry Nodelman (1988: 220) states, the information offered by the words of a PB change the way the reader construes the meaning of the images, and the images change the way the reader interprets the words. This gives us a reason to suspect that visually presented information might, in some cases, alter the way in which the words are translated in multimodal texts combining words and images. If the image, in one way or another, changes the meaning of a certain word with which it is presented, the translation of the word might no longer be what could be considered as its most obvious “word-for-word” translation. The purpose of this article is to discover whether the images of a translated PB may have affected its translator’s word choices. I analyze the relationship between lexical items in the source and target texts [TT] to identify any alterations and differences, and then compare these observations against the information offered by the image with which the item is presented. In other words, I aim to determine whether the images provide an explanation as to why a certain item was translated the way it was. The data of the study is comprised of the Finnish translations of The Tale of Peter Rabbit by Beatrix Potter. The Tale of Peter Rabbit is an exceptional PB in the sense that it has been re-illustrated dozens of times. The publisher of the book failed to register the book for copyrights in the United States when they first started marketing the book in the country in early 20th century. Therefore, anyone could compile a new set of illustrations and market the book without restrictions (Mackey 1998: xix). Out of the various illustrated versions, three have been translated into Finnish. The original book illustrated by Potter herself was translated in 1963 by Riitta Björklund. In 2003, Ritva Toivola translated a version illustrated by Cyndy Szekeres and, in 2005, Ritva Brander translated a slightly abridged version of the story illustrated by Lisa McCue. Reading the same verbal story coupled with three different T Anne Ketola Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 51 illustrations provides quite an exceptional opportunity to examine whether certain words are afforded different kinds of meanings when presented together with different images. Theoretical background Gunther Kress and Theo van Leeuwen (1998: 186), two of the most notable scholars in the field of multimodal research, state that all texts are multimodal. A person who talks creates their message using not only words but also intonation, rhythm, facial expressions, gestures and postures. If one would receive the same verbal message in a written form, the material on which the message is written, the font or the handwriting as well as the layout used would all affect the way in which the message is interpreted. Each mode used in the creation of a message imparts something about the intentions and the personality of its sender (Lemke 2002: 302). It is therefore impossible to create a message using only one individual mode. As Kress and van Leeuven (1998: 186) underline, no text analysis that fails to account for the text’s multimodal aspects can explain all layers of meaning present in the text. One of the central claims of multimodal research is that all modes are of equal importance (Jewitt 2009b: 13). Within multimodal research, language –although having been of tremendous research interest in the past– is merely one meaning making resource amongst many. This means that within a multimodal text combining words and images, as it is the case in PB, words and images play equally important roles in creating the overall message of the book. It may hence be claimed that when translating a PB, the information offered by the verbal and the visual modes are of equal importance. Another important principle of multimodal research is that each mode within a multimodal text participates in creating its overall meaning by realizing their own communicative task, expressing what each mode expresses best (Jewitt 2009c: 15). Analogically, this means that when studied separately, each mode of a multimodal ensemble is incomplete and cannot represent all meanings present in the multimodal whole. Research addressing the translation of multimodal texts should, for this very reason, always cover the analysis of each of the modes involved. As stated above, PB combine two different modes, the visual and the verbal, images and written language, but how does this affect the readers of PB? Jay Lemke (2002: 303) discusses what he calls “the essential incommensurability” between different modes: A verbal text is never able to produce the same and only the same meanings as an image, and vice versa. In Lemke’s words, this incommensurability inevitably leads to genuinely new meanings being created in the combination of modes (2002). The meanings created in the combination of modes are therefore always greater than the sum of the meanings created by the same modes individually (Flewitt, Hampel, Hauck & Lancaster 2009: 46). In accordance with the above observation, the combination of words and images in a PB always creates genuinely new meanings. When we read PB, it is possible for us to find meanings that are present only when the two modes are presented together; meanings that are not present in either mode alone. It might then be possible that the genuinely new meanings created within the multimodal ST are conveyed to the translator’s interpretation of the text and, moreover, occasionally to the translation choices the translator makes. Research method The aim of this study is to discover whether the images of a translated PB may have affected its translator’s word choices. With a view to determining this, I first contrast the words or expressions that can be thought of occupying identical positions in the ST and TT and analyze Anne Ketola 52 Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 how they correspond to each other. I then compare these observations to the information the visual ST expresses about a given item to find out whether the images can be used to explain why the items were translated as they were. In the first stage of my analysis I divide the TT phrases into lexical items (words, phrasal verbs, collocations, etc.) and compare them with their counterparts in the verbal ST (in this article, I refer to these two items as “equivalents” even in cases where no imaginable form of equivalence exists between them). I then aim to identify similarities and differences between the meanings the contrasted items carry. The research could hence be considered as a form of cross-linguistic contrastive analysis with a focus on analyzing translational equivalence. One may easily think of different relationships existing between a ST and a TT item. The perhaps most common –and, in most cases, desirable– situation would be a word-forword translation. By this somewhat assailable term I mean a relationship where the two items correspond to each other as closely as it is possible between two languages or, in Eugene Nida’s (1969, 495) terms, where the translation is “the closest natural equivalent of the message of the source language” (a simple example of this could be the English word “cat” translated into Spanish as “gato”). Another possible relationship between the two items is a situation where the TT item is more precise in meaning than the verbal ST item, in other words, where the translator moves from hypernyms to hyponyms, from superordinate to subordinate terms (for instance, the English word “animal” translated into Spanish as “gato”). In the opposite case, the translator moves from hyponyms to hypernyms, the TT item being more general in meaning than the verbal ST item (the English word “cat” being translated into Spanish as “animal”). It might also be possible that the lexical item is changed in translation in a way that correspondence between the target and verbal ST items can hardly be said to exist (for example, the English word “cat” translated into Spanish as “perro”). It may also be that a lexical item in the TT has no equivalent in the verbal ST, in other words, something has been added in the translation process. The opposite case would be a lexical item being left out in the translation process altogether, and the verbal ST item would then have no equivalent in the TT. We may now compile the following six categories to describe the relationship between each element of the TT and its equivalent in the verbal ST: A. The TT item corresponds to its verbal ST equivalent word-for-word. B. The TT item is more precise in meaning than its verbal ST equivalent. C. The TT item is more general in meaning than its verbal ST equivalent. D. The TT item does not correspond to its verbal ST equivalent. E. The TT item has no equivalent in the verbal ST. F. The verbal ST item has no equivalent in the TT. I present this stage of the analysis in a table where I divide the translated sentence into lexical items placed on the rows of the first column (with English back-translations in square brackets). The verbal ST equivalents of these elements are placed on the rows of the second column. The third column describes how the Finnish and English expressions correspond to each other (options A–F). To illustrate the method, I present an analysis of a sentence from my data, taken from the translation of Cyndy Szekeres’ illustration version. The verbal ST goes: “Mr. McGregor hung up the little jacket and the shoes for a scarecrow to frighten the blackbirds” (Potter 2002) and its Finnish translation goes “Herra Jyry ripusti Petterin pikku takin ja kengät puutarhaansa linnunpelättimeksi rastaille” [Mr. Jyry hung up Peter’s little jacket and shoes in his garden for a scarecrow for the thrushes] (Potter 2003). Anne Ketola Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 53 Table 1. An example of a contrastive analysis between target and verbal ST items. TT item Verbal ST equivalent How items correspond Herra Jyry [Mr. Jyry] Mr. McGregor A ripusti [hung up] hung up A Petterin [Peter’s] E pikku [little] Little A takin [jacket] the - - jacket A ja [and] And A kengät [the shoes] the shoes A puutarhaansa [in his garden] E linnunpelättimeksi [for a scarecrow] for a scarecrow A to frighten F rastaille [(for the) thrushes] the blackbirds C As it can be seen from the table, most TT items correspond to their verbal ST equivalents word-for-word (option A). However, two of the elements (Petterin, meaning “Peter’s” and puutarhaansa, “in his garden”) do not have equivalents in the verbal ST (option E), and the verb “to frighten” does not have an equivalent in the TT (option F). Also, the word “blackbird” (mustarastas, “the black thrush” in Finnish) has been translated with an expression with a more general meaning (option C), rastas, meaning “thrush” (the blackbird is a subspecies of thrush). After analyzing how the ST and TT items correspond to each other, I then compared these results against the information the visual ST provided the translators. The second stage of my analysis is what I call a multimodal ST analysis, where I contrast the visual and verbal ST, in other words, I go through the data sentence by sentence and compare each one to the image with which it is presented. The questions 2 I pose in the multimodal ST analysis are the following: What is presented redundantly between visual and textual elements? What is presented only in the visual mode? What is presented only in the verbal mode? What is presented differently in the two modes? Which elements change their meaning in the combination of the two modes? The last question of the list refers to cases where the visual and verbal ST convey slightly different but closely related aspects of meaning about a given item. The reader then gets additional information about the item, for example in cases where the verbal ST expresses what is done and the visual ST expresses exactly how it is done. To illustrate the method, I present the analysis of the sentence used in the example above. Since I was denied the permission to print the images of my data, I am limited to describing the visual information verbally. In this case, the visual ST offers an image of a scarecrow made of two long sticks in the shape of a cross. The scarecrow has a large onion for a head, and is wearing a small, blue jacket as well as black shoes where its hands supposedly are. The scarecrow is placed in the middle of a garden. There are six brown birds flying around the scarecrow and one of them is sitting on its arm. 2 The questions posed at this stage of the analysis are based on a set of questions that Lemke suggests should be asked about the combination of the visual and verbal modes in his article Doing Multimedia Analysis of Visual and Verbal Data: A Guide (2011). Anne Ketola 54 Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 Table 2. An example of a multimodal ST analysis What is presented redundantly between visual and textual elements? What is presented only in the visual mode? What is presented only in the verbal mode? What is presented differently in the two modes? Which elements change their meaning in the combination of the two modes? Small jacket and shoes have been hung for a scarecrow. There are six brown birds, one of which is sitting on top of the scarecrow. The scarecrow has been placed in a garden. The jacket and shoes have been hung by Mr. McGregor. The color of the birds: brown according to visual, black according to verbal ST. Scariness of the scarecrow: not scary for the birds according to visual, scary according to verbal ST. – The visual ST repeats some of the information offered by the verbal ST, but it also offers some information that is not available verbally. There are two details that are presented differently between the two modes. The verbal ST states that the scarecrow is meant to frighten the birds in the garden, yet, according to the visual ST, the birds do not find the scarecrow scary at all. The color of the birds in question is also presented differently in the two modes: they are black according to verbal information, but presented as brown in the image. I may now compare the observations made in the two stages of analysis. I do this by taking the table of the contrastive analysis and adding to that five new columns which correspond to the questions posed in the multimodal ST analysis (Was the element presented redundantly between visual and textual elements? Was it presented only in the visual mode or the verbal mode? Was it presented differently in the two modes? Was its meaning changed in the combination of the two modes?) In the five new columns, I tick with an X the option that is valid for each element, leaving out connectors and other words that are impossible to be expressed visually and are therefore of no interest in a multimodal analysis. The table may then be used to determine what kind of information the visual ST offered about the items analyzed in stage 1. To illustrate this, I present the combination of the tables used in the examples above: Anne Ketola Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 55 Table 3. An example of combining the two stages of analysis. These tables may now be used to detect, for instance, whether items that do not have an equivalent in the verbal ST (option E in the third column) are presented in the visual mode and therefore have an equivalent in the visual ST, as is the case with the word puutarhaansa (“in his garden”). The tables may also be used to determine what happened in the translation to the elements that are presented differently in the different modes. Do they correspond to the verbal ST elements word-for-word (option A), or did the conflicting visual information somehow affect the translator’s interpretation? In Table 3, an example of a contradictorily presented element is the word “to frighten”, which has been left out of the translation altogether (option F). The word “blackbird” is also presented differently in the two modes. This element, however, has been translated with an expression more general in meaning (option C). Research results The main objective of this study has been to elicit whether the images of a translated PB may have affected its translator’s word choices. As might be expected, the analysis indicated that most TT items were direct “word-for-word” translations of their verbal ST equivalents. Whether visually offered information affected these choices (or not) is beyond the scope of this kind of a research method and may only be speculated on in this context. Therefore, my main research interest lies on the other-than-normal translation choices, in other words, categories B–F presented in the previous chapter. The analysis revealed that these atypical TT item Verbal ST equivalent How items correspon d Presented redundantly between visual and textual el. Presented only in the visual mode Presented only in the verbal mode Presented differently in the two modes Element changed its meaning in the comb. Herra Jyry [Mr. Jyry] Mr. McGregor A X ripusti [hung up] hung up A X Petterin [Peter’s] E - - - - - pikku [little] little A X takin [jacket] the - - jacket A X ja [and] and A - - - - - kengät [the shoes] the shoes A X puutarhaansa [in his garden] E X linnunpelättimeksi [for a scarecrow] for a scarecrow A X to frighten F X rastaille [(for the) thrushes] the blackbirds C X Anne Ketola 56 Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 translation choices could, to great extent, be explained by the information offered by the visual ST or the interplay of visual and verbal information in the multimodal ST. I will present the findings by going through the six categories of possible relationships between target and verbal ST items presented in the previous chapter and comparing them to how the visual ST related to these cases. As mentioned above, category A, where the TT item corresponds to its verbal ST equivalent word-for-word, is of little interest in this stage of the analysis, since it represents what is “supposed to happen” in a translation process, and offers no insight into the central research questions. Categories C, D, and F, that is, TT items that were more general than their verbal ST equivalents, TT items that did not correspond to their verbal ST equivalents and verbal ST items that had no equivalents in the TT, were often linked to a contradiction between the information presented in the visual and verbal modes. It may then be suggested that in cases where the words and images of the multimodal ST present incoherent information, the translators might feel the need to modify the verbal information to avoid contradiction in the translation. As the data of the study demonstrates, one strategy that may be used to avoid contradiction in some cases is moving from hyponyms to hypernyms, translating the item with an equivalent so general in meaning that contradiction between image and verbal text no longer exists. The translation of the word “blackbird” in Table 3 is an example of such a strategy. The birds are presented as brown in the visual ST but described as black in the verbal ST. By translating “blackbird” with the name of the superordinate species, the translator avoids using the name of the color all together, and contradiction between word and image is avoided in the translation. Another available strategy to avoid contradiction is to follow the information presented in the visual ST and ignore information presented verbally. This obviously results in the fact that the given TT item no longer corresponds to its verbal ST equivalent. The third strategy used in the data was simply leaving out the item presented differently by the two modes of the multimodal ST (the verbal ST item is now obviously left without an equivalent in the TT). Examples of category E, where the TT items did not have an equivalent in the verbal ST, turned out to be closely associated with visually presented information, since most of such items proved to have an equivalent in the visual ST. What this means is that the translators, either consciously or unconsciously, verbalized information that was presented only visually in the multimodal ST. An example of such an addition can be seen in Table 3, where the Finnish word puutarhaansa (“in his garden”) that does not have an equivalent in the verbal ST, is presented in the visual mode, in other words, the added item has an equivalent in the visual ST. Images offer a significant amount of additional, (seemingly) superfluous information about the characters, settings and actions in the book, and some of this information had been explicated into verbal information in all of the three translations. These additions are obviously not necessary for the target language reader to understand the text. Moreover, one could even ask whether changing the text in such a way in the translation process is acceptable. Category B, where the TT items are more precise in meaning than their verbal ST equivalents, is probably the most interesting of all from the point of view of multimodal research. Nearly all these translation choices were linked to elements whose meaning was changed in the combination of the two modes, in other words, that were presented slightly differently in the two modes. An obvious example of such a change are the translations of the word “plant” in a scene where Peter Rabbit knocks over a couple of plants. In two of the illustration versions, the word is presented together with an image of distinctively small plants that grow in pots of clay (in the third illustration version, this scene has been left out). Anne Ketola Estudios de Teoría Literaria, 6 (12), “Libros ilustrados como textos de partida multimodales. El análisis contrastivo como herramienta para mostrar la interacción imagen-texto”: 49-60 57 Visually offered information may be suggested to have influenced both translators’ interpretations of the item, since the word “plant” has been translated into Finnish as taimi (“seedling”) in one translation and a taimiruukku (“seedling pot”) in the other. Here the combination of verbal information (“plant”) and visual information (“small” / “growing in a pot”) has clearly defined the meaning of the given item. As well as nouns, the data offered various examples of verbs that had been translated with more precise expressions. A particularly illustrative example of such modifications are the translations of the verb “to eat” in a scene where Peter Rabbit eats radishes. The action of eating is presented in different ways in the three illustrations. In Potter’s original illustration, the rabbit is eating two radishes at a time, with its head tilted back and eyes half-closed with pleasure. Szekeres has depicted the scene very differently: the rabbit sits holding a radish from which only one bite has been taken. It stares into distance and its mouth is tightly closed. In McCue’s illustrated version the action of eating is depicted as the most animated of all. The rabbit is leaning against a cabbage with its face beaming with delight. It is holding two radishes, one of which it is just about to take a large bite of. The different atmospheres of the illustrations have also been conveyed to the translations: in the translation of Szekeres’ illustrated version, where the action of eating per se was left to little attention, the verb “to eat” is translated with the expression maistella hieman (“to taste a little”). In the translations of McCue’s and Potter’s illustrated versions, the verb has been translated as ahmia (“to gobble”) and as herkutella (“to relish”) respectively. One could claim that the combination of verbal information (“to eat”) combined with visual information (“take pleasure in eating”) results in a translation more precise in meaning (“to relish”). I propose that in these cases, the information coming from the visual and the verbal ST combine to define and delimit the meaning of a given element. The visual and textual elements become semantically associated, and their concurrence delimits the meaning of the whole. In Lemke’s words (2002), the combination of visual and verbal information results in something genuinely new; a meaning more precise than what either of the modes express alone. Conclusions The readers –including the translators– of PB read both words and visual images. Their understanding of the book’s overall meaning is created in the interaction of these two modes. As verbal and visual modes are never able to convey the same and only the same information, the combination of the two results in something genuinely new in the reader’s interpretation (Lemke 2002: 303). The observation is particularly interesting when modeling the process of PB translation. If a given word is afforded genuinely new meanings when presented together with an image, this might also affect the way in which the word is translated. The analysis revealed that in cases where the images of the PB convey information that is, in one way or another, contradictory to the information conveyed verbally, the translators seemed to have felt the need to modify the verbal information to avoid contradiction in their translation. The strategies used to accomplish this in the data of this study were translating the given word with an equivalent more general in meaning, ignoring the verbal information and conveying what is expressed visually, or omitting the word altogether. Another way in which visually offered information seemed to affect the translators’ word choices was the explication of visually offered information into verbal. All three translators had, either consciously or unconsciously, extracted details from the illustrations and verbalized this information in their translation. The third way in which visually offered information seemed to have affected the translators’ word choices had to do