Analyzing Persuasive Techniques in TED Talks

Analyzing Persuasive Techniques in TED Talks Brendan Boyle, Thelma Lax, Gregory Pflaumer, and Natchanun Sanitdee Department of Languages, University of Helsinki 10.5.2024 Abstract This report discusses a project involving a persuasive technique annotation scheme applied to a TEDTalk dataset. As TEDTalks are by convention persuasive texts with easily available transcripts, they are apt for a speculative exploration into the design and implementation of a corpus using TED as a dataset and its annotation for persuasive techniques. The 10 most watched TEDTalk videos are selected for the annotation tasks. The annotation is completed in the INCEpTION environment using 12 categories (tags) and 4 annotators. The datasets are categorized based on the persuasive technique framework (ethos, logos, pathos) (Higgins & Walker, 2012). Fleiss’s kagreement coefficient (Fleiss, 1971) is used to measure the inter-annotator agreement (IAA). The score for each category is also reported. The overall score of 0.271 indicates a fair agreement level among annotators with the highest agreement on Call-To-Action (CTA) category at 0.653 and the lowest agreement on Pathos-Emotion (PAT-EMO) and Pathos-Rhetorical Question (PAT-RH) at 0.066 and 0.058, respectively. The categorical scores implies that anecdote-related categories (PAT-ANE, ETH-ANE, and LOG-ANE) are relatively challenging to annotate, and emotion-related categories (especially PAT-EMO and PAT-RH) are the most difficult ones. Keywords: TEDTalks, Persuasive Techniques, Corpus Linguistics, Annotation. 1. Introduction With an unprecedented amount of language data being produced and collected at a rate never seen before in history, online media platforms serve as a lucrative field for scholars to study linguistic phenomena. TEDTalk is a non-profit aiming to spread ideas through a short presentation by different speakers. With top videos having millions of views each, TEDTalk gains its popularity due to its powerful content. TEDTalk presentations, given a limited time of usually under 18 minutes, are packed with different linguistic tools to 1 draw attention from the audience and convince them. Therefore, TEDTalk discourse makes an interesting dataset to study persuasive techniques. To investigate persuasive techniques in TEDTalks in a quantitative, scalable manner, this project has devised a novel annotation scheme for the express purpose of processing TEDTalk transcripts. The scheme is based on classical rhetorical persuasion theory, but adjusted for the multitude of persuasive techniques found in the TEDTalk genre. Annotation was conducted in the INCEpTION environment using 12 categories that were identified as necessary in order capture the depth of persuasive techniques across TEDTalks. 2. Background Theory and Framework 2.1 TED TED Conferences, LLC is an American-Canadian non-profit founded in 1984. TED stands for Technology, Entertainment, and Design, the topics which the content initially covered Ludewige (2017). Later, its emphasis expanded to other fields, and in 2006, TEDTalks were made available online to the public. 2.2 Persuasive Techniques Aristotle’s conception of rhetoric provides the foundation of the three main persuasive techniques (ethos, logos, pathos) in this project. The categorization is mainly inspired by the Elements of Rhetorical Appeals (Higgins & Walker, 2012) as demonstrated in Table 1. In addition, from the initial observation of the transcriptions, an additional persuasive technique, Call-To-Action, is identified. 3. Data and Methods 3.1 Datasets The sample dataset used for the initial construction of the corpus was compiled from the video transcripts of the ten most watch TED and TEDx Talks at the time of collection (March 2024) from https://www.ted.com/talks. The decision to use TED presentation data was inspired by the article from Ludewige (2017). Focusing on the most popular talks narrows down the set of thousands of TED Talks to a handful of speeches that carry the essence of the genre and are considered some of the more successful uses of rhetoric. The use of the corpus for a more in-depth study would require the addition of more transcripts. Given the scope and timeframe for this pilot investigation however, the quantity of data was limited to keep the annotation workload manageable. In addition 2 Appeal Examples of persuasive techniques ETHOS: credibility Similitude Ingratiation Deference Expertise Self-criticism Inclination to succeed Consistency PATHOS: emotion Metaphors Identification, especially through cultural references such as: •Sport •Under-privilege •Health, well-being •Hope, aspiration •Loyalty •Friendship •Sympathy LOGOS: reason Argumentation Logic Warrants/justifications Claims Data Evidence/examples (e.g. historica Table 1: Elements of rhetorical appeals (Higgins & Walker, 2012) to the files annotated by the team for analysis, a second set of transcripts were used as practice data for assembling and refining the codebook. The official transcripts of TED presentations are available freely on the ted.com website. These are all manually created by human transcriptionists, not automatically generated. The transcript texts were collected by the corpus team from the website and compiled in .txt files. To remove metadata such as timestamps and audio descriptions, all files were cleaned simultaneously using OpenRefine (‘OpenRefine’, 2010–2024), a free, open-source data wrangling application. The clustering function was used to merge all instances of timestamps, and then a series of regular expression commands were performed to replace instances of timestamps and audio descriptions (e.g., laughter, applause, cheering) with a single space. Once the data was cleaned, all transcripts were uploaded onto INCEpTION simultaneously, but as separate files. To investigate persuasive techniques in TEDTalks in a quantitative, scalable manner, this project has devised a novel annotation scheme for the express purpose of processing TEDTalk transcripts. The scheme is based on classical rhetorical persuasion theory, but adjusted for the multitude of persuasive techniques found in the TEDTalk genre. 3 3.2 Annotation Process This first round of annotation also acted as a calibration stage where all annotators discussed any disagreements and built a common understanding upon how to annotate the transcripts. Annotation is done using the INCEpTION annotation environment. After logging in to the webpage, navigate to the project you wish to work on. Then select the Annotation tab from the right-hand toolbar. This will open a list of documents. Click on the assigned document. 3.3 Mitigating Biases Selection biases: TEDTalks videos are selected from the top10 mostly watched videos. The transcriptions are assigned equally to each annotator based on a number of words. Label biases: To minimize label biases, after the annotation guideline is created, annotators practice annotating in practice transcriptions and note the confusing labels before discussing and polishing the guideline. 3.4 Methods Before starting, be sure that the Persuasive Techniques Layer is selected from the menu in the top left-hand corner. To create an annotation, double click on the target sentence. The entire sentence will automatically be highlighted. Select the desired tag from the left-hand side toolbar or use the provided hotkeys. The chosen tag will appear above the target sentence in the document. Make sure this tag is correct before moving on. If incorrect, the tag can be removed using the undo or delete buttons or hotkeys. If correct, double click on the next sentence to continue annotation. Upon the end of the document, check to make sure that every sentence has a tag. Once verified that the annotations are correct and complete, click on the lock button in the upper toolbar. This will lock the document and finish the annotation. The application methods of the annotation scheme on the transcriptions are described in the succeeding steps below. 1. Select desired project. (Figure1) 2. Select Annotation from right-hand toolbar. (Figure2) 3. Select an assigned document. (Figure3) 4. Select Persuasive Techniques layer from top left corner menu. (Figure4) 5. Double click on sentence to create annotation. (Figure5) 6. Select appropriate tag from left-hand toolbar or using hotkeys. (Figure6) 4 Figure 1 Figure 2 Figure 3 Figure 4 7. Confirm tag is correct. 8. Double click on next sentence to continue annotating. 9. Verify that every sentence has been tagged. 5 Figure 5 Figure 6 10. Click on lock icon in upper toolbar to complete the document. (Figure7) Figure 7 6 4. Results and Analysis 4.1 Annotation Reliability Analysis 4.1.1 Inter-Annotator Agreement (IAA) Inter-annotator agreement is measured based on the transcription of the TEDTalk "Your body language may shape who you are" by Dr. Amy Cuddy (2012) annotated by 4 annotators. The transcriptions consist of 274 sentences. Because the task involves categorical annotations by 4 annotators, Fleiss’s kagreement coefficient (Fleiss, 1971) is used to assess the annotation reliability. The annotations completed on the INCEpTION tool are populated into a .csv file in a tabular format. Tags (categories) are converted into numbers (1-12). The annotations for the first 10 sentences are shown in the Table 2. Sentence Annotator1 Annotator2 Annotator3 Annotator4 1 11 11 2 10 2 11 11 11 10 3 3 12 3 5 4 12 12 2 10 5 12 12 2 10 6 12 12 2 10 7 2 12 2 1 8 11 11 11 10 9 1 4 2 10 10 1 2 2 10 Table 2: Annotations of the first 10 sentences Fleiss’ kappa score is calculated by dividing the subtraction of the observed agreement Powith the expected agreement Peby the subtraction of 1with the expected agreement Peas demonstrated in the formula below. Table 3from Landis & Koch (1977) shows a guide for interpreting the Kappa coefficient. κ= ¯ Po −¯ Pe 1−¯ Pe Kappa Agreement Strength > 0.8 Almost Perfect > 0.6 Substantial > 0.4 Moderate > 0.2 Fair 0 - 0.2 Slight < 0 Poor Table 3: Interpretation of Kappa coefficient 7 Using kappam.fleiss function in the irr library in R, the agreement scores between 4 annotators for each of the 12 categories, are shown in the Table 4. •Subjects = 274 •Raters = 4 •Kappa = 0.271 Category Tagset* Kappa 1 PAT-ANE 0.261 2 PAT-EMO 0.066 3 PAT-RH 0.058 4 LOG-LOG 0.32 5 LOG-RH 0.545 6 LOG-STA 0.366 7 ETH-ANE 0.201 8 ETH-ETH 0.302 9 ETH-RH 0.132 10 LOG-ANE 0.171 11 CTA 0.653 12 NONE 0.124 Table 4: Fleiss’ Kappa score of 12 categories *The description of each tagset can be found in the 6.1.3 Subtask and Tags. According to the score table, the overall kappa score of 0.271 indicates a fair interannotator agreement. The kappa scores of each category reveal the highest agreement in category 11 Call-To-Action (CTA) at 0.653 followed by Logos-Rhetorical Question (LOGRH) at 0.545. Logos-Statistics (LOG-STA), Logos-Logos (LOG-LOG), and Ethos-Ethos (ETH-ETH) receive a similar agreement level at 0.366, 0.32, and 0.302, respectively. Pathos-Anecdote (PAT-ANE), Ethos-Anecdote (ETH-ANE), and Logos-Anecdote (LOGANE) occupy somewhere in the middle range at 0.261, 0.201, and 0.171, respectively. Ethos-Rhetorical Question (ETH-RH) and None-Persuasive (NONE) see a relatively low agreement (0.132 and 0.124, consecutively). The least agreed on categories include PathosEmotion (PAT-EMO) and Pathos-Rhetorical Question (PAT-RH) at only 0.066 and 0.058, respectively. An interesting observation can be made for the Call-To-Action category (CTA), which receives the highest agreement score. This implies that this category is easiest to be identified, and there was the least confusion among annotators. In addition, the scores of anecdote-related categories (PAT-ANE, ETH-ANE, and LOG-ANE) signal that annotators find it challenging to categorize anecdotes. Last but not least, emotion-related categories, especially PAT-EMO and PAT-RH, seem to be the trickiest task to handle. This urges for a more thorough revision of the codebook and a better understanding alignment process before real annotation. 8 5. Conclusion This report has discussed the design and implementation a novel annotation schema for annotating persuasive techniques in TEDTalk presentations. The annotation schema is speculative as required of its purpose. The requirements toward success in annotator agreement were thus lax, and focus was placed on a thoughtful, cooperative learning process instead of producing a readily applicable scheme. This processed proved fruitful, and though a modest reliability score was eventually achieved, improvements to the scheme and overall reliability success can be made using a sufficient annotator practice, testing, and repeated codebook revisions. In addition, a more comprehensive calculation of the agreement among subgroups of annotators should be made to get a more nuanced reliability assessment (Artstein, 2017). This project, despite its need for further development, may therefore provide any intrepid corpus design projects with an insight into design strategies and the challenges a corpus annotation project may face. 6. Appendix 6.1 Codebook 6.1.1 General Rules for Annotators •The focus of the annotation task is on a sentence level. •If a sentence contains more than one persuasive technique, the sentence needs to be annotated with only one technique that comes first in the annotation scheme table. For example, the following sentence “Businesses can invest in better health care, better education and a higher minimum wage so that they create a group of people who are hopeful about the future and less vulnerable to the calls by extremists to burn the system down.” contains PAT-EMO and LOG-LOG. LOG-LOG comes before PAT-EMO in the scheme table; therefore, it will be annotated as LOG-LOG. 6.1.2 Annotation Scheme Tagset Persuasive Technique Ethos ETH-ETH (credibility building statement) ETH-ANE (credibility building using anecdote) ETH-RH (credibility building using rhetorical question) Logos LOG-LOG (logic and argumentation statement) LOG-STA (argumentation using statistics or data) LOG-ANE (argumentation using anecdote) LOG-RH (argumentation using rhetorical question) 9