Hemispheric dominance of metaphor processing for Chinese-English bilinguals : DVF and ERPs evidence

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ Hemispheric dominance of metaphor processing for Chinese-English bilinguals : DVF and ERPs evidence © 2022 Elsevier Ltd. Accepted version (Final draft) Zhu, Xichu; Chen, Hongjun; Otieno, Susannah C.S.A.; Cong, Fengyu; Leppänen, Paavo H.T. Zhu, X., Chen, H., Otieno, S. C., Cong, F., & Leppänen, P. H. (2022). Hemispheric dominance of metaphor processing for Chinese-English bilinguals : DVF and ERPs evidence. Journal of Neurolinguistics, 63, Article 101081. https://doi.org/10.1016/j.jneuroling.2022.101081 2022 Hemispheric dominance of metaphor processing for Chinese-English bilinguals: DVF and ERPs evidence Xichu Zhua,b, Hongjun Chena,*, Susannah C.S.A. Otienob, Fengyu Congc, Paavo H.T. Leppänenb a School of Foreign Languages, Dalian University of Technology, Dalian, China b Department of Psychology, University of Jyväskylä, Jyväskylä, Finland c School of Biomedical Engineering, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China * Corresponding author. School of Foreign Languages, Dalian University of Technology, Dalian, China E-mail address: chenh[email protected] (H. Chen). Declarations of interest: none. Abstract: This study investigated whether metaphors are predominantly processed in the right or left hemisphere when using Chinese and English metaphors in Chinese bilingual speakers. The role of familiarity in processing of metaphorical and literal expressions in both the first and second language was studied with brain-event-related potentials using a divided-visual-field paradigm. The participants were asked to perform plausibility judgments for Chinese (L1) and English (L2) familiar and unfamiliar metaphorical and literal sentences. The results obtained using parameterfree cluster permutation statistics suggest a different pattern of brain responses for metaphor processing in L1 and L2, and that both metaphoricity and familiarity have an effect on the brain response pattern of both Chinese and English metaphor processing. However, the brain responses were distributed bilaterally across hemispheres, suggesting no clear evidence for lateralization of processing of metaphorical meanings. This is inconsistent with the Graded Salience Hypothesis and Fine-Coarse Semantic Coding Theory, which posited a right hemisphere advantage of non-salient and coarse semantic processing. Keywords Metaphor processing; Chinese-English bilingual; Hemispheric dominance; Dividedvisual-field; Event-related potential; Familiarity 1. Introduction Figurative language such as metaphor in particular has aroused the interest of researchers for centuries as metaphors are pervasive in our daily life. The processing of metaphor is metaphorical rather than literal, conceptual rather than rhetorical (Lakoff and Johnson, 1981), which makes the myth and controversy of its neural mechanism. In recent decades, a number of metaphor studies using brain imaging and electrophysiological techniques, such as functional magnetic resonance imaging (fMRI) and event-related potentials (ERPs), have explored how the human brain responds to metaphor processing (Kasparian, 2013; Diaz & Eppes, 2018). Cognitive physiological studies have found that patients with right hemisphere damage have impaired metaphor cognition, suggesting a right hemispheric dominance in metaphor processing (Brownell et al., 2000; Pobric et al., 2008). Many fMRI studies on normal participants have also indicated a right hemispheric dominance in figurative language comprehension (Bottini et al., 1994; Mashal et al., 2007; Shibata et al., 2007; Yang et al., 2016; Tang et al., 2017; Koleva et al., 2019). Several behavioral and ERP studies have been conducted using the divided-visual-field (DVF) paradigm to investigate the hemisphericity of metaphor processing. The DVF paradigm involves measuring task performance when visual stimuli are presented to one hemi visual field at a time, thus highlighting hemispheric advantage based on reaction time and accuracy rate (Anaki et al., 1998; Faust & Mashal, 2007; Kacinik & Chiarello, 2007; Schmidt et al., 2007). These behavioral studies suggested a right hemisphere advantage for metaphor processing. Other studies have applied ERPs in combination with the DVF paradigm to address hemispheric asymmetry in language comprehension (Coulson & Van Petten, 2007; Metusalem et al., 2016). These studies obtain different results, with some suggesting that the right hemisphere plays a crucial role in semantic processing (Metusalem et al., 2016) and others suggesting the bilateral processing of metaphorical meanings (Coulson & Van Petten, 2007). Other studies have found no evidence for the special role of the right hemisphere in metaphor processing while using fMRI and ERPs (Stringaris et al., 2006; Rapp et al., 2007; Coulson & Van Petten, 2007; Briner et al., 2018; Segal & Gollan, 2018). The current study investigated the hemispheric dominance of metaphor processing in Chinese (L1) and English (L2) using ERPs in a DVF paradigm with the aim to bring more evidence to the semantic processing mechanism in the human brain. Based on these conflicting findings, a number of hypotheses on the brain mechanisms involved in figurative language processing have been proposed, of which the Fine-Coarse Semantic Coding Theory and the Graded Salience Hypothesis are widely quoted. Beeman (1998) outlined a general model of coarse semantic coding in the right hemisphere and fine semantic coding in the left hemisphere, known as the Fine-Coarse Semantic Coding Theory (FCSC Theory). That hypothesis suggested that the right hemisphere appears to maintain broader meaning activation and recognize distant meaning relations. When processing semantic meanings, large semantic fields of related information, including information about words that are only distantly related, are weakly activated in the right hemisphere. This relatively coarse semantic coding accounts for the right hemisphere’s linguistic inferiority, as well as its sensitivity to semantic overlap involving distantly related words. By contrast, the left hemisphere is strongly activated by narrow semantically closely related information. According to the hypothesis, this relatively fine semantic coding indicates the left hemisphere’s linguistic superiority, and explains its relative insensitivity to distant semantic relations and vulnerability to misinterpretation in certain discourse and problem-solving contexts. Another psycholinguistic hypothesis addressing the lateralization of semantic processing is the Graded Salience Hypothesis (GSH; Giora, 2003), which argues that salient meanings rather than literal-metaphorical distinction determine the hemisphere dominance of metaphor processing. If the meaning of an expression can be extracted directly from the mental lexicon, it is regarded as salient. This hypothesis predicts a left hemisphere dominance in processing salient meanings (such as familiar metaphors) and right hemisphere dominance in processing less salient meanings (such as unfamiliar metaphors) (Giora, 1997, 2003). Both hypotheses have gained considerable support from behavioral and neurocognitive studies (Laurent et al., 2006; Lee & Dapretto, 2006; Faust & Mashal, 2007; Schmidt et al., 2007; Rapp et al., 2012; Mashal et al., 2015). For example, Schmidt et al. (2007) used three behavioral DVF experiments to investigate the influence of familiarity in metaphor processing and found a right hemisphere advantage for unfamiliar sentences with faster responding in the LVF/RH (left-visual-field/right hemisphere), regardless of metaphoricity (whether the sentences were metaphorical or literal). However, results inconsistent with the hypotheses of GSH and FCSC Theory have also been presented (Coulson & Van Petten, 2007). For example, Coulson and Van Petten (2007) showed in an ERP study, while investigating the effects of cloze probability and metaphoricity, that metaphoricity effects (metaphorical vs. literal) were similarly distributed across hemifields, suggesting that metaphorical meanings were integrated in both hemispheres. The hemispheric dominance of second or foreign language (the terms L2 and FL seem to be interchangeably used in the literature, and we report here what has been used in the publications) metaphorical processing among bilinguals is even less clear and only scarcely studied. The extent to which hypotheses of native semantic processing (such as familiarity, literality vs. metaphoricity) are applicable to bilingual metaphor processing and the extent to which bilingual metaphor processing can inform the predominant role of hemispheres in native language metaphor comprehension still remain unknown. To the best of our knowledge, only a few studies have investigated brain lateralization of bilingual metaphor processing (Mashal et al., 2015; Segal & Gollan, 2018). Mashal et al. (2015) explored the differences in hemispheric involvement in English and Hebrew metaphor processing with the DVF paradigm behaviorally. The results of their experiments demonstrated a left hemisphere advantage for conventional metaphor processing in the native language (L1), with a shorter reaction time of conventional metaphors (Hebrew as L1) presented to the RVF/LH (right-visual-field/left hemisphere), but a right hemisphere advantage for second language (L2) processing of the same type of material with a shorter reaction time of conventional metaphors (Hebrew as L2) presented to the LVF/RH (Mashal et al., 2015). Besides metaphors, several studies have investigated the hemispheric asymmetry of L2 idiomatic language processing (Cieślicka & Heredia, 2011; Cieślicka, 2013; Cieślicka, 2015). For example, when testing the effect of salience and context on the cerebral asymmetries of Polish (L1)–English (L2) bilingual idiom processing, those researchers found that several factors (for example, language, context, and idiomaticity) had a joint effect on the lateralization of L2 figurative language processing and that both hemispheres were involved in idiom processing over time. However, only several studies have examined brain activation involved in bilingual metaphor processing with Chinese as L1 and English as L2 (Chen et al., 2013; Wang, 2018). In these studies, both metaphoricity (Chen et al., 2013) and L2 proficiency (Wang, 2018) were found to be factors that influenced bilingual metaphor processing separately. It was found that the right hemisphere had a dominant role during the comprehension of L2 metaphors with larger N400 amplitudes of English (L2) metaphors in the right hemisphere (Wang, 2018). The N400 component, which was first reported by Kutas and Hillyard (1980) in semantically inconsistent sentences, is typically investigated in semantic studies (Pynte et al., 1996; Tartter et al., 2002; Coulson & Van Petten, 2002; Chen et al., 2013; Metusalem et al., 2016). The brain responses showing N400 appear at around 300ms after the stimuli onset and peaks at around 400ms, which indicates the language-related brain activations. The current study aims to investigate the hemispheric dominance of bilingual metaphor processing in native (Chinese) and foreign or second (English) language, by taking both metaphoricity and familiarity into consideration. An experiment using brain event-related potentials (ERPs) and a divided-visual-field (DVF) paradigm was designed to study the role of metaphoricity (metaphorical vs. literal) and familiarity (familiar vs. unfamiliar) of the expressions on metaphor processing and its hemisphericity. These aspects were first studied separately in the native language (L1Chinese) and second language (L2-English), and then compared between the two languages. The Fine-Coarse Semantic Coding Theory (FCSC Theory; Beeman, 1998) would predict that the N400 response would be smaller (indicating a match with expectations) at the left hemisphere for the familiar literal and metaphorical sentences, especially in the native Chinese language. On the other hand, the Graded Salience Hypothesis (GSH; Giora, 2003) would predict that N400 is larger (that is, more negative) to the unfamiliar than familiar when presented to the left visual field, suggesting that unfamiliar meanings will be predominantly processed in the right hemisphere. In addition, it can be hypothesized that N400 differences in L1 between conditions (that is, familiar vs. unfamiliar and metaphorical vs. literal) are clearest at the left hemisphere, while the corresponding N400 differences in L2 (for which there has been less exposure) will appear predominantly at the right hemisphere. Therefore, we presume that L1 and L2 metaphorical expressions will demonstrate a different pattern in the preponderance of hemispheric involvement, based on the assumption that L2 expressions contain less salient and more coarse semantic meanings and are processed more as distant semantic relations compared to the native language (Giora, 2003; Mashal et al., 2015). 2. Methodology 2.1. Participants A total of 25 native Chinese(L1) speakers (7 males and 18 females) with English as their second language(L2) participated in the current study. The final sample consisted of 20 participants (5 males, 15 females). They ranged from 22 to 28 years of age (M = 23.7, SD = 1.45). All participants were born in China and had started learning English from around the age of 9 and had passed the Test for English Majors-Band 8 (TEM-8), the highest level that can be attained in the English proficiency test in China. Before the formal experiment, they were required to provide self-reported language profiles and take a LexTALE Test (Lemhöfer & Broersma, 2012), a standardized lexical test for language proficiency. The average score for their self-reported language proficiency was 4.467 (on a scale from 1 to 5, where 5 = most proficient, SD = 0.498) and that of the LexTALE Test was 69.79% (range: 50–96.25%, full score = 100%, SD = 0.101). All of the participants were right-handed, measured by the Edinburgh Handedness Inventory (Oldfield, 1971; Cohen, 2008) (M = 84.44, SD = 14.96), with normal or corrected-to-normal vision and no history of neurological disorder or brain injury. ERP data were discarded from 5 participants due to equipment problems or noisy EEG data. 2.2. Materials The stimulus materials consisted of 16 different experimental conditions containing 4 variables: metaphoricity (literal vs. metaphorical), familiarity (familiar vs. unfamiliar), language (Chinese vs. English), and visual field presentation (left-visualfield vs. right-visual-field). The stimulus pool comprised of 400 sentences: 100 familiar metaphorical sentences, 100 unfamiliar metaphorical sentences, 100 familiar literal sentences, and 100 unfamiliar literal sentences, half of which were in Chinese and half in English. 20 anomalous sentences in Chinese and 20 in English served as fillers to make sure that the participants concentrated on the task (not reported here). All sentences were in the form “A is (/are) B.” (for example, “Earth is mother.”) in English and “甲是乙。” (for example, “地球是母亲。”) in Chinese. Some example stimuli are shown in Table 1 below. Metaphoricity Familiarity Language Examples Sentence stem Target word Metaphorical Familiar Chinese 地球是 母亲 English Earth is mother Unfamiliar Chinese 基因是 蓝图 English Genes are blueprints Literal Familiar Chinese 音乐是 艺术 English Music is art Unfamiliar Chinese 镜子是 玻璃 English Mirror is glass Fillers Chinese 老虎是 希望 English Tiger is hope Table 1. Examples of experimental stimuli for each condition (familiar metaphorical sentences, unfamiliar metaphorical sentences, familiar literal sentences, unfamiliar literal sentences, and fillers in both Chinese and English language). 4 pretests were conducted to control the familiarity of the stimuli (a pool of 117 metaphorical sentences and 122 literal sentences in Chinese, and 116 metaphorical sentences and 120 literal sentences in English), in which participants were asked to rate the level of familiarity in the sentences in their native language on a scale of 1 (extremely unfamiliar) to 5 (extremely familiar). The participants in the pretests were college students who would not participate in the formal experiment. For metaphorical materials, there were 34 Chinese participants and 33 native English participants performing the online questionnaires. For literal materials, another 32 Chinese participants and 33 English native speakers participated in the pretests. According to the results from the pretests, the metaphorical sentences with a rating over 4 (Chinese: 4.03–4.87, M = 4.36; English: 4–4.87, M = 4.47) were selected as familiar metaphors, while sentences with a rating below 3.7 (Chinese: 2.11–3.68, M = 3.12; English: 2.09– 3.67, M = 3.01) were selected as unfamiliar metaphors. Similar criteria were applied to choose literal materials. Sentences rated over 4.1 (Chinese: 4.1–4.5, M = 4.24; English: 4.1–4.46, M = 4.23) were regarded as familiar literal sentences, and those rated below 3.9 (Chinese: 2.91–3.9, M = 3.65; English: 2.89–3.8, M = 3.57) were regarded as unfamiliar literals. Independent-sample T-test showed that the familiarity ratings differed between the familiar and unfamiliar stimuli in both languages (p < .001), and no significant difference existed between the familiarity of Chinese and English stimuli for each metaphoricity condition (p > .05). The physical properties of the stimuli were also controlled for the sentence length. Each sentence was exactly 3 words in both Chinese and English. The length of the Chinese target words “乙” was 2 or 3 characters (M = 2.01, SD = 0.12) and the English target words “B” were between 3 and 13 letters (M = 6.27, SD = 1.91) in length. The Chinese targets were counted by using the corpus of Beijing Language and Culture University Corpus (BCC) (Xun et al., 2016), and the English targets were counted by using the British National Corpus (BNC) (The British National Corpus, 2007). The word frequency was controlled according to the data in these corpora. In the experiment, the stimuli were divided into four blocks (two Chinese blocks and two English blocks). In each block, there are 50 sentences of each sentence type and 20 fillers presented to either leftor right-visual-field, making up a stimulus pool of 220 sentences. Chinese and English blocks were presented to participants separately in order to reduce the influence of language switching, and the sequence of blocks was counterbalanced among the participants. In each block, the stimuli were randomly presented to the participants. 2.3. Procedure Prior to the experiment, the participants completed a basic information form detailing their gender, age, and language proficiency. They were required to take the LexTALE Test (Lemhöfer & Broersma, 2012) and the Edinburgh Handedness Inventory (Oldfield, 1971; Cohen, 2008) before the formal experiment. A consent form was obtained from the participants after the experiment. All participants were well instructed and performed a brief practice session, which showed the same procedure as the formal experiment, to ensure that they understood the requirements and process of the experiment. The experiments were conducted in a quiet room using the software Eprime 2.0. Stimuli were presented in yellow against a black background to avoid dazzling the participants, who were seated approximately 50 cm from the monitor screen. They were instructed to refrain from facial, eye, and body movements during the task. Each sentence was presented twice (in the left and right visual fields separately) to each participant, thus the participants saw all stimuli appearing equally in both visual fields. Each participant was presented with all 440 sentences. After the instruction session and a short practice, the formal experiment began with a fixation on the screen for 500ms to focus the participants’ attention. The offset of the fixation cross was followed by the sentence stem (the first two words of the sentence, “甲是” or “A is (/are)”) in the center of the screen for 1500ms, which was followed by another fixation for 300ms. The sentence final word (“乙” or “B”) was presented laterally to the left or right visual field, together with a fixation cross at the center of the screen for 190ms. The lateral position was calculated so that the inside edge of the final words had a visual angle of 3.5° to the left or right of the fixation cross. 800ms after the sentence’s final word appeared on the screen, a plausibility task (“你 能理解这个句子吗？” in Chinese and “Can you understand this sentence?” in English) appeared in the center of the screen, prompting the participants to judge whether the sentence was understandable. The language used in the plausibility question was consistent with the language used in the experimental trial to avoid code-switching. Upon seeing the question, participants were asked to judge whether the sentence was understandable and to indicate their response on the keyboard (“J” for “Yes” and “F” for “No”) with the index fingers of their left and right hands, respectively. The next trial began after the participants gave their judgements. Participants were allowed a short break after 50 trials. Figure 1 illustrates the overall sequence of events for one trial. Lateralization of the stimuli was assured by several aspects of the experimental procedure: (a) the visual angle of the prime was between 3 and 5 degrees; (b) brief presentations of the target words (within 200ms); and (c) the random mixing of right and left presentation. Figure 1. Overall sequence of events for one trial. 2.4. Behavioral task analysis The reaction time (RT) of the participants’ responses was analyzed as part of the behavioral results. The reaction times for correct responses were calculated for each participant in all experimental conditions of interest. Only those responses that judged the stimuli to be understandable were considered correct. The responses given for anomalous sentences that served as fillers were not analyzed. To determine how the hemispheric dominance was influenced by metaphoricity and familiarity in the L1 (Chinese) and L2 (English) blocks, linear mixed models, including random variables and random slopes, were conducted for the RTs recorded in the Chinese and English blocks by SPSS 22.0 (IBM Corp., Armonk, NY). In each block, the effects of metaphoricity and familiarity were analyzed separately with a 2 (Metaphoricity: metaphorical, literal) × 2 (Visual field: left, right) linear mixed model and a 2 (Familiarity: familiar, unfamiliar) × 2 (Visual field: left, right) linear mixed model. 2.5. EEG recording and ERP analysis The electroencephalogram (EEG) was recorded using the BioSemi ActiveTwo system (BioSemi Inc., Amsterdam, The Netherlands), using active Ag/AgCl electrodes (BioSemi ActiveTwo) and 64 scalp sites according to the modified 10-20 System. Each electrode was referred to the average of the left and right mastoids, and re-referenced to average reference later. The EEG signals were digitized at 512 Hz. Data was processed using EEGLAB (Delorme & Makeig, 2004) and ERP_ERO (Zhang et al., 2020) toolboxes under the MATLAB (The MathWorks, Inc.) environment. A high-pass filter of 0.5Hz and a low-pass filter of 30Hz was applied to filter out any noise and unrelated signals, and a notch filter of 50Hz was also applied to eliminate the impact of domestic electricity supply interference. The trials with the amplitudes exceeding ± 80 uV were rejected automatically, and trials with eye movement artifacts were rejected by the Independent Component Analysis (ICA). ERPs were derived by averaging both trials with a Yes and No response for comprehensibility questions on the different conditions for each participant from -100ms to 800ms relative to the onset of the target word (that is, the final word of the sentence). A 100ms pre-target period was used as the baseline. BESA Statistics 2.1 (BESA GmbH, Germany) was applied for statistical analysis with the method of parameter-free cluster permutation statistics (Maris & Oostenveld, 2007; Pernet et al., 2015) and BESA Research 7.0 (BESA GmbH, Germany) and BESA Plot (BESA GmbH, Germany) were used for plotting waveforms and topographies. 3. Behavioral results Overall, participants correctly judged the sentences at an average of 81.12% (range: 62.73–86.60%). 3.1. The Chinese block Table 2 shows the mean values and standard deviations of the reaction times for the Chinese expressions. When the stimuli were presented to the right-visual-field (thereby reflects the preponderance of left-hemispheric processing; RVF/LH), the reaction times for both metaphorical and literal sentences were significantly longer than those presented to the left-visual-field/right hemisphere (LVF/RH) (metaphorical: Estimate = 19.207, S.E. = 9.352, p = 0.046; literal: Estimate = 22.030, S.E. = 10.178, p = 0.031). These results displayed an overall right hemispheric advantage for both metaphoric and literal meanings. For the effect of familiarity, the reaction time of Chinese unfamiliar sentences presented to the RVF/LH were significantly longer than Chinese familiar sentences presented to the RVF/LH (Estimate = 46.298, S.E. = 12.424, p = 0.001) and the reaction times of Chinese unfamiliar sentences presented to the RVF/LH were also significantly longer than Chinese unfamiliar expressions presented to the LVF/RH (Estimate = 41.948, S.E. = 10.692, p = 0.000), leading to a significant interaction of the Familiarity and Visual fields (Estimate = 31.038, S.E. = 14.310, p = 0.030). This result showed that Chinese unfamiliar meanings might be dominantly processed in the right hemisphere, which is consistent with the prediction from the Graded Salience Hypothesis (GSH, Giora, 2003) that the right hemisphere was dominant when processing less salient meanings. Metaphoricity Familiarity Visual Field Value of t Value of p Left visual field Right visual field literal, unfamiliar metaphorical and unfamiliar literal expressions presented to both left and right visual fields for 330–560ms. (c) Cluster-based permutation topographies of the comparisons: Chinese familiar metaphorical vs. literal expressions presented to the LVF, Chinese familiar metaphorical vs. literal expressions presented to the RVF, Chinese unfamiliar metaphorical vs. literal expressions presented to the LVF, and Chinese unfamiliar metaphorical vs. literal expressions presented to the RVF for the following time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Metaphorical–Literal contrast (2 stars = p < .01, 3 starts = p < .001). As for the ERP asymmetry of familiarity effect, it showed overall bilateral results for all the conditions. Chinese familiar metaphors and familiar literal expressions showed a significantly larger N400 effect than unfamiliar ones at the frontal and central regions. When presented to the left visual field, familiar metaphors activated significantly more negative N400 than unfamiliar metaphors at the frontal-central areas bilaterally, with some right hemispheric preponderance at ca. 420ms (cluster 1 at 360– 560ms, max. t-value -6.03, p < .001, at ca. 520ms). The right visual field presentation also generated an N400 effect with larger negativity for familiar metaphors at the bilateral frontal-central regions, which occurred earlier at ca. 360ms (cluster 1 at 330– 400ms, max. t-value -4.84, p < .001, at ca. 330ms). For the literal targets, the left visual field presentation generated a significant cluster with a left hemispheric preponderance at ca. 350–450ms and distributed bilaterally at ca. 460ms (cluster 1 at 330–500ms, max. t-value -4.86, p < .001, at ca. 380ms), showing significantly more negative N400 for familiar targets. There was no significant difference between the brain responses to familiar and unfamiliar literal expressions when presented to the right visual field. The grand average ERP waveforms at the electrodes of interest, and ERP averaged topographies across the time window of 330–560ms are presented in Figure 6(a) and (b) above, while Figure 7 presents a series of cluster-based permutation analysis topographies. Figure 7. Cluster-based permutation topographies of the comparisons: Chinese familiar vs. unfamiliar metaphorical expressions presented to the LVF, Chinese familiar vs. unfamiliar metaphorical expressions presented to the RVF, Chinese familiar vs. unfamiliar literal expressions presented to the LVF, and Chinese familiar vs. unfamiliar literal expressions presented to the RVF for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Familiar–Unfamiliar contrast (3 starts = p < .001). 4.2. ERP results for English expressions 4.2.1. Metaphoricity effects for the combined LVF and RVF presentations in English Contrary to the results of Chinese expressions, English familiar literal expressions showed significantly larger negativity than metaphors at the frontal region bilaterally. The significant cluster emerged at ca. 500ms and lasted until 540ms (max. t-value 3.39, p < .001, at ca. 530ms). Figure 8 presents the grand average ERP waveforms at the electrodes of interest, ERP averaged topographies across the time window of 330– 560ms, and a series of cluster-based permutation analysis topographies. Figure 8. (a) The waveforms of the N400 amplitudes for English familiar metaphorical and literal expressions on the electrodes of interest (F3, C3, P3, F4, C4, and P4). Blue line: ERPs to English familiar metaphorical expressions; red line: ERPs to English familiar literal expressions. (b) ERP topographies of English familiar metaphorical and literal expressions for 330–560ms. (c) Clusterbased permutation topographies between the brain responses for the English familiar metaphorical and literal targets for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Metaphorical– Literal contrast (3 starts = p < .001). For unfamiliar sentences, three significant clusters were observed showing differences between the brain responses to the metaphorical vs. literal expressions (see Figure 9). The first cluster emerged at ca. 410ms and lasted until 560ms, with a lefthemispheric preponderance at the temporo-centro-parietal showing significantly more negative N400 components for the literal targets (max. t-value 7.05, p < .001, at ca. 480 ms). A weaker cluster with a lateral right fronto-temporo-occipital region seemed to appear at the time window of 440–560ms at the edge electrodes (max. t-value 4.16, p < .01, at ca. 560 ms). The third cluster appeared at the time window of 530–560ms, showing significantly more negative N400 components for metaphorical expressions (max. t-value -3.56, p < .001, at ca. 550 ms). Figure 9. (a) The waveforms of the N400 amplitudes for English unfamiliar metaphorical and literal expressions on the electrodes of interest (F3, C3, P3, F4, C4, and P4). Blue line: ERPs to English unfamiliar metaphorical expressions; red line: ERPs to English unfamiliar literal expressions. (b) ERP topographies of English unfamiliar metaphorical and literal expressions for 330–560ms. (c) Cluster-based permutation topographies between the responses for the English unfamiliar metaphorical and literal targets for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Metaphorical–Literal contrast (2 stars = p < .01, 3 starts = p < .001). 4.2.2. Familiarity effects for the combined LVF and RVF presentations in English The investigation of familiarity effect was also performed with combined data of left visual field and right visual field. For metaphorical expression, two significant clusters were observed, showing the differences between the brain responses to the familiar vs. unfamiliar expressions with significantly more negative N400 components for the familiar metaphors (see Figure 10). The first cluster appeared at ca. 450ms and lasted until 560ms with a left hemispheric preponderance at the frontal-temporal areas (max. t-value -5.73, p < .001, at ca. 500ms). The second cluster emerged at ca. 520ms and continued to 560ms, with a right hemispheric preponderance (max. t-value -4.54, p < .01, at ca. 550ms). Figure 10. (a) The waveforms of the N400 amplitudes for English familiar and unfamiliar metaphorical expressions on the electrodes of interest (F3, C3, P3, F4, C4, and P4). Blue line: ERPs to English familiar metaphorical expressions; red line: ERPs to English unfamiliar metaphorical expressions. (b) ERP topographies of English familiar and unfamiliar metaphorical expressions for 330–560ms. (c) Cluster-based permutation topographies between the responses for the English familiar and unfamiliar metaphorical targets for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Familiar–Unfamiliar contrast (2 stars = p < .01, 3 starts = p < .001). The significant cluster of English literal expressions showed differences between the brain responses to the familiar and unfamiliar targets (see Figure 11). It emerged at ca. 510ms and lasted until 560ms showing significantly more negative N400 components for familiar targets (max. t-value -3.53, p < .001, at ca. 540ms). This cluster appeared bilaterally at the frontal and central areas. Figure 11. (a) The waveforms of the N400 amplitudes for English familiar and unfamiliar literal expressions on the electrodes of interest (F3, C3, P3, F4, C4, and P4). Blue line: ERPs to English familiar literal expressions; red line: ERPs to English unfamiliar literal expressions. (b) ERP topographies of English familiar and unfamiliar literal expressions for 330–560ms. (c) Clusterbased permutation topographies between the responses for the English familiar and unfamiliar literal targets for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Familiar– Unfamiliar contrast (3 stars = p < .001). 4.2.3. ERP asymmetry based on the left and right visual field presentations in English The effect of the visual field was further analyzed by comparing brain responses to the target words presented to either the left or the right visual field in each condition (see Figures 12 and 13). When presented to the left visual field, familiar literal expressions activated significantly more negative N400 than familiar metaphors at the frontal-central region bilaterally, with some right-hemispheric preponderance (cluster 2 at 380-470ms, max. t-value 3.53, p < .01, at ca. 410ms). This effect became stronger later on at ca. 510ms (cluster 1 at 500–560ms, max. t-value 4.14, p < .001, at ca. 530ms). However, there was no significant difference between the brain responses to the metaphorical vs. literal contrast with the right visual field presentation. For the unfamiliar expressions, the responses to unfamiliar metaphorical vs. literal expressions presented to the left visual field generated only a narrow significant cluster at the left temporal region at around 510ms (cluster 2 at 460–560ms, max. t-value 4.64, p < .01, at ca. 470ms) and slightly larger negativity at the right frontal-temporo-parietal region for the literal targets at ca. 480ms (cluster 1 at 450–540ms, max. t-value 5.05, p < .001, at ca. 470ms). The right visual field presentation generated significantly more negative N400 for unfamiliar metaphors at the frontal-central region, with a right hemispheric preponderance at ca. 480ms (cluster 1 at 450–560ms, max. t-value -5.20, p < .001, at ca. 490ms), and a narrow significant cluster at ca. 450ms with an opposite topography at the left temporo-parietal region with more negative N400 components for unfamiliar literal targets (cluster 2 at 440–480ms, max. t-value 4.60, p < .001, at ca. 450ms). Figure 12 presents the grand average ERP waveforms at the electrodes of interest, ERPaveraged topographies across the time window of 330–560ms, and a series of clusterbased permutation analysis topographies. Figure 12. (a) The waveforms of the N400 amplitudes for English familiar metaphorical, familiar literal, unfamiliar metaphorical and unfamiliar literal expressions presented to both left and right visual fields on the electrodes of interest (F3, C3, P3, F4, C4, and P4). Blue line: ERPs to English familiar metaphorical expressions; blue dashed line: ERPs to English unfamiliar metaphorical expressions; red line: ERPs to English familiar literal expressions; red dashed line: ERPs to English unfamiliar literal expressions. (b) ERP topographies of English familiar metaphorical, familiar literal, unfamiliar metaphorical and unfamiliar literal expressions presented to both left and right visual fields for 330–560ms. (c) Cluster-based permutation topographies of the comparisons: English familiar metaphorical vs. literal expressions presented to the LVF, English familiar metaphorical vs. literal expressions presented to the RVF, English unfamiliar metaphorical vs. literal expressions presented to the LVF, and English unfamiliar metaphorical vs. literal expressions presented to the RVF for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Metaphorical–Literal contrast (2 stars = p < .01, 3 starts = p < .001). To investigate the ERP asymmetry of familiarity effect, the brain responses to familiar vs. unfamiliar metaphors and familiar vs. unfamiliar literal sentences were compared separately. When presented to the left visual field, unfamiliar metaphors activated significantly more negative N400 than familiar metaphors at the frontalcentral part of the brain at around 420ms (cluster 1 at 380-460ms, max. t-value 3.12, p < .001, at ca. 4100ms). For the target stimuli presented to the right visual field, the responses to familiar vs. unfamiliar metaphorical expressions generated only a narrow significant cluster at the left temporal region at around 510ms (cluster 1 at 450–560ms, max. t-value -3.57, p < .001, at ca. 460ms) and a stronger negativity at the right frontotemporo-parietal region for the familiar targets at ca. 540ms (cluster 2 at 520–560ms, max. t-value -4.95, p < .01, at ca. 540ms). For the literal expressions, there was no significant difference between the responses to familiar vs. unfamiliar literal expressions with left visual field presentation. When presented to the right visual field, the responses to familiar literal targets generated an N400 effect with larger negativity at the right hemisphere at around 480ms and distributed bilaterally later at ca. 540ms (cluster 1 at 450–560ms, max. t-value -4.41, p < .001, at ca. 550ms). Figure 12(a) and (b) above present the grand average ERP waveforms at the electrodes of interest, and ERP-averaged topographies across the time window of 330–560ms, and Figure 13 (below) shows a series of cluster-based permutation analysis topographies. Figure 13. Cluster-based permutation topographies of the comparisons: English familiar vs. unfamiliar metaphorical expressions presented to the LVF, English familiar vs. unfamiliar metaphorical expressions presented to the RVF, English familiar vs. unfamiliar literal expressions presented to the LVF, and English familiar vs. unfamiliar literal expressions presented to the RVF for the time points: 330ms, 360ms, 390ms, 420ms, 450ms, 480ms, 510ms, 540ms, and 560ms. The stars denote significant clusters for the ERP differences between the Familiar–Unfamiliar contrast (2 stars = p < .01, 3 starts = p < .001). 5. Discussion This study investigated the hemispheric dominance for metaphor processing in Chinese (L1) and English (L2) languages in native Chinese speakers who were proficient in English by using experimental stimuli of different metaphoricity (metaphorical, literal) and different familiarity (familiar, unfamiliar). Overall, the results demonstrated quite different processing patterns of L1 and L2. They also revealed that both metaphoricity and familiarity had an influence on semantic processing, with the results showing significant differences between metaphorical and literal expressions, and between familiar and unfamiliar expressions. Furthermore, the results showed slower reaction times and less clear brain response patterns for metaphoricity and familiarity effects in L2, indicating less efficient second-language metaphor processing. The behavioral results showed a right hemispheric advantage (shorter reaction times for the left visual field presentations) for unfamiliar metaphorical meanings. However, the ERP results showed that the brain activation was distributed bilaterally for both familiar and unfamiliar expressions. Less clear hemispheric preponderance of brain response effects in some conditions could be explained by the interhemispheric transfer of the information, which occurs in the brain early on at ca. 160–170ms (Chaumillon, Blouin, & Guillaume, 2018). Therefore, in the time window of N400 typically used in semantic studies (Tartter et al., 2002; Coulson & Van Petten, 2002; Metusalem et al., 2016), which we chose to be 330–560ms based on our data in the current study, the semantic information has already been transferred to the contralateral hemisphere, showing bilateral distribution of the N400 effect. 5.1. Effects of metaphoricity on hemispheric activation Previous studies have broadly investigated the effects of metaphoricity and familiarity (Chen et al., 2013; Mashal et al., 2015; Segal & Gollan, 2018). Consistent with our hypothesis, our results showed that the metaphoric meanings in L1 generated more negative N400 responses than literal meanings at the left hemisphere with the hemifield presentation. For Chinese (L1) expressions, the results showed that N400 was mainly distributed at the frontal and central regions influenced by both metaphoricity and familiarity. The analyses with combined ERPs for different visual fields (see Figures 2 and 3) showed that the Chinese familiar metaphors generated a significantly more negative N400 than the literal expressions, suggesting that the metaphorical meaning processing was more difficult than the literal meanings (Coulson & Van Petten, 2002). The effect was observed predominantly at the right hemisphere at an early portion (ca. 360ms) of N400, whereas with a bilateral distribution at a later time window of 500–560ms. For the unfamiliar targets, the responses to unfamiliar metaphors generated more negative N400 than unfamiliar literal expressions at the left hemisphere at the time window of 340–460ms. This result is consistent with our prediction that the differences of the brain responses to the metaphorical vs. literal targets in L1 are clearest at the left hemisphere, which is supported by studies that showed a left hemisphere advantage for metaphor processing in L1 (Mashal et al., 2015). The analyses separately for the left and right visual fields (see Figure 6) showed that when the stimuli were presented to the left visual field, the responses to familiar metaphors generated more negative N400 than familiar literal expressions bilaterally with some right hemispheric preponderance at the time window of 500–550ms. When presented to the right visual field, familiar metaphors generated more negative N400 than familiar literal expressions bilaterally at ca. 360ms. These results for Chinese familiar expressions indicate the difficulties for metaphorical meaning processing, and that the brain activities were distributed in both hemispheres (Coulson & Van Petten, 2007). For unfamiliar expressions presented to the left visual field, unfamiliar literal expressions generated more negative N400 than unfamiliar metaphors, with a narrow occipital lobe distribution at the left hemisphere at around 540–560ms. When presented to the right visual field, the responses to unfamiliar metaphors generated more negative N400 than unfamiliar literal expressions at the left hemisphere at ca. 360–430ms, which is also consistent with our prediction of L1 metaphor processing. The N400 differences in L1 between conditions (i.e., metaphorical vs. literal) are clearest at the left hemisphere, since L1 expressions contain more salient semantic meanings and processed as close semantic relations compared to second language (Beeman, 1998; Giora, 2003; Mashal et al., 2015). The findings suggest that metaphoricity plays a role during L1 metaphor processing. Similar to the results from a previous study (Coulson & Van Petten, 2007), metaphoric materials generated more negative N400 amplitudes in the frontal and central areas across hemispheres, and the hemifield presentation influenced the brain activations between the left and the right hemispheres. Their results also suggested that both hemispheres were sensitive to semantic metaphoric processing (Coulson & Van Petten, 2007). 5.2. Effects of familiarity on hemispheric activation For Chinese (L1) expressions, the analyses with combined ERPs for different visual field presentations (see Figures 4 and 5) showed that Chinese familiar expressions generated a significantly more negative N400 than unfamiliar expressions in both literal and metaphorical conditions. The results are inconsistent with the Graded Salience Hypothesis (GSH; Giora, 2003) which would predict a larger N400 for unfamiliar expressions than familiar expressions. For metaphorical expressions, the effect was observed predominantly bilaterally at an early portion (ca. 360ms), whereas with a left hemispheric preponderance at a later time window of 450–560ms. For literal expressions, the familiar expressions generated more negative N400 than unfamiliar expressions with a left hemispheric preponderance at the time window of 340–400ms. The results indicate clear differences of brain responses to familiar vs. unfamiliar expressions in L1, which is consistent with our prediction. The hemifield presentation might have an impact on the brain activations in the two hemispheres at different time windows, consistent with the findings in the studies of Coulson and Van Petten (2007) that hemifield presentation shifted the brain activity between the left and the right hemisphere in the N1 and post-N400 stages. The analyses separately for the left and right visual field (see Figure 7) showed that familiar metaphors generated significantly more negative N400 bilaterally than unfamiliar metaphors with some right hemispheric preponderance. When presented to the left visual field, the N400 effect that was larger for familiar metaphors than that for unfamiliar metaphors was observed at the time window of 390–540ms. When presented to the right visual field, the familiar metaphors generated only early effects of more negative N400 than unfamiliar metaphors, at around 330–360ms. For literal expressions, when presented to the left visual field, familiar literal expressions generated more negative N400 than unfamiliar literal expressions. The effect was observed predominantly at the left hemisphere at the time window of 350–450ms, whereas with a bilateral distribution later at ca. 460ms. However, the target stimuli presented to the right visual field did not generate any differences in N400. The current results were partly contrary to several previous ERP studies with similar paradigms (Forgács et al., 2012, 2014; Davenport & Coulson, 2013). Forgács et al. (2012, 2014) carried out studies which presented novel, which could be regarded as unfamiliar, literal word pairs in semantic decision tasks. In their studies, novel literal expressions were processed faster in the RVF/LH (Forgács et al., 2014) and were more strongly activated in the left inferior frontal areas (Forgács et al., 2012). However, the results of the current study showed less activated N400 for unfamiliar literal expressions, which might be influenced by the sentence context and grammatical information that were missing from the processing of word pairs in the above-mentioned studies. These results are inconsistent with the Graded Salience Hypothesis (Giora, 2003), which postulates that less salient meanings, whether metaphorical or literal, are mainly processed by the RH, and would predict that the N400 is larger to the unfamiliar expression than familiar ones when presented on the left visual field. The results of the familiarity effect are also contrary to the hypothesis of the Fine-Coarse Semantic Coding Theory (FCSC Theory; Beeman, 1998), which suggests that words or sentences with more distantly related and subordinate meanings are processed by activating the coarse semantic mechanisms in the RH, and would predict a smaller N400 for familiar literal and metaphorical sentences at the left hemisphere. Since familiarity and cloze probability are strictly controlled in the current study, our results indicate that both metaphoricity and familiarity may influence the semantic processing of metaphors. Both hemispheres are sensitive to the processing of semantic meanings, with different levels of metaphoricity and familiarity in sentence context. The divided-visual-field paradigm employed in this current study showed a hemifield effect on both behavioral and ERP responses. The results in both L1 and L2 showed an interaction of familiarity and visual field presentation that familiar expressions showed clearer patterns when presented to LVF than to RVF, while unfamiliar ones showed clearer patterns with RVF presentation than LVF presentation (see Figure 6 and 12). This interaction is indicated from the results that the Chinese unfamiliar metaphorical vs. literal expressions presented to the LVF generated only odd left occipital lobe responses (see Figure 6c), lacking the frontal-central differences, and the corresponding English contrast of unfamiliar metaphorical vs. literal expression presented to the LVF also generated only narrow temporal lobe responses (see Figure 12c), without the differences in the frontal-central region. Moreover, significant differences appeared earlier with RVF presentation than LVF presentation in L1 (Chinese), showing that the hemispheric dominance might be influenced by the level of familiarity, and the left hemisphere might be more sensitive than the right hemisphere to the metaphor familiarity (Segal & Gollan, 2018). Many studies from the field of psychology have investigated this above-mentioned effect of familiarity on visual field attention (Bradshaw and Gates, 1978; Buttle and Raymond, 2003; Marzi and Viggiano, 2007; Küper and Zimmer, 2018; Angiulli et al., 2020). They have suggested that a supporting speech mechanism located in the right hemisphere was activated by novel or unfamiliar materials (Bradshaw and Gates, 1978), and a superfamiliarity effect was found in the left visual field presentation with face stimuli (Buttle and Raymond, 2003). 5.3. Different processing patterns of L1 and L2 Compared to L1 (Chinese), the results of L2 (English) showed less clear Wang, Q. (2018). Neural mechanism and representation of English and Chinese metaphors of bilinguals with different second language proficiency: An ERP study. Chinese Journal of Applied Linguistics, 41(1), 67–83. https://doi.org/10.1515/cjal2018-0004. Xun, E., Rao, G., Xiao, X., & Zang, J. (2016). Dashuju beijingxia BBC yuliaokude yanzhi [The development of BBC corpus under the background of big data]. Corpus Linguistics, 3(1), 93–109. Yang, J., Li, P., Fang, X., Shu, H., Liu, Y., & Chen, L. (2016). Hemispheric involvement in the processing of Chinese idioms: An fMRI study. Neuropsychologia, 87, 12–24. https://doi.org/10.1016/j.neuropsychologia.2016.04.029. Zhang, G., Li, X., & Cong, F. (2020). Objective Extraction of Evoked Event-related Oscillations from Time-frequency Representation of Event-related Potentials. Neural Plasticity, 2020, 1-20. doi:10.1155/2020/8841354.