ValiText - Application Example Open-Ended Questions on European Identity

Indicators and Metrics of Data Quality: Gesis Open-Ended Questions on Europeanness Isabela Zeberio Data Set: Gesis - Perspectives on Europe The dataset consists of open-ended text responses collected through the GESIS Panel Population Sample, a mixed-mode panel survey conducted both online and via postal mail. The GESIS Panel is a probabilistic sample of approximately 5,000 individuals representing the German population. These participants are surveyed quarterly on various topics, providing both cross-sectional and longitudinal data, but the data is not publicly available yet, so I just got access to the item. As part of the Call for Item Submission: Perspectives on Europe, I submitted two open-ended questions to explore the concept of European identity in the context of the 10th EU parliamentary elections. The submitted questions focused on respondents’ perceptions of what defines a European identity and what characteristics or behaviors might exclude someone from being considered European. Specifically, participants were asked: 1. What does it take to be European? please describe the characteristics and behavior of a person you would describe as European (Was braucht es, um Europäerin oder Europäer zu sein?Bitte beschreiben Sie, welche Eigenschaften und welches Verhalten eine Person hat, die Sie als europäisch bezeichnen würden) 2. And what excludes someone from being European? Please describe the characteristics and behavior of a person you would describe as non-European (Und was schließt aus, dass jemand Europäerin oder Europäer ist? Bitte beschreiben Sie, welche Eigenschaften und welches Verhalten eine Person hat, die Sie als nicht-europäisch bezeichnen würden) The dataset consists entirely of qualitative, unstructured text data derived from these open-ended responses. The responses vary in length and detail, offering rich insights into individual perspectives on European identity. Since the data was collected within a structured panel survey, it benefits from high methodological rigor, ensuring data quality and representativeness within the German population. The GESIS Panel includes a wide range of other variables, however, at this stage, I have only access to the responses from this submitted item—the two open-ended questions on European identity. Measurement Bias For this assignment, I am focusing on how we measure the complex social construct of European identity— specifically, people’s subjective perceptions of “Europeanness.” This means looking at how respondents interpret and define European identity content, which traits and behavior people associate with being European, as well as the boundaries they draw between who belongs (ingroup) and who does not (outgroup). Traditional surveys often use closed questions, which can limit how people express their views. With this project I want to overcome this limitation by using open-ended questions, so people can express in their own words what they think about. Since this approach is different, it offers an opportunity to gain new insights. However, I also need to validate the measurement and assess whether it effectively captures the content of European identity. To do this, I 1 will use ValiText and TextPrep. ValiText Validating text-based measures of social constructs is crucial to ensuring that what we measure truly reflects the construct being studied. ValiText provides both conceptual and practical guidance for executing and documenting validation (Birkenmaier, Wagner, and Lechner 2023). ValiText requires three types of validation evidence: • Substantive Evidence: Establishing the theoretical foundation of the measure. • Structural Evidence: Evaluating the properties of the model and its measures. • External Evidence: Testing how the measure relates to independent criteria or information. For this assignment, I will present the substantive evidence and continue working with this framework for the structural and external evidence. I am really pleased to have learned about this framework—it is extremely helpful. It allows us to reflect on the validity of how we try to measure this complex social construct, enhance transparency and help us better understand both the strengths and limitations of this approach. Substantive Evidence: Establishing the theoretical foundation of the measure. Figure 1: Construct Definition and Operationalization 2 Figure 2: Design Decision Non response # Clean data # Remove puntuaction and special characters clean_text <- function(data) { data %>% mutate(across(c(european, non_european), ~{ .%>% str_to_lower() %>% str_replace_all("[[:punct:]]","")%>% # Remove punctuation str_replace_all("[[:symbol:]]","")%>% # Remove symbols str_squish() })) } data_clean <- clean_text(data_gesis) # Structure of the data str(data_clean) ## tibble [5,238 x 3] (S3: tbl_df/tbl/data.frame) ## $ id : num [1:5238] 1 2 3 4 5 6 7 8 9 10 ... ## $ european : chr [1:5238] "in europa geltende werte zu akzeptieren und zu leben" "weiß nicht" "99" "99" ... ## $ non_european: chr [1:5238] "werteregelungen zu ignorieren" "weiß nicht" "99" "99" ... summary(data_clean) ## id european non_european ## Min. : 1 Length:5238 Length:5238 3 ## 1st Qu.:1310 Class :character Class :character ## Median :2620 Mode :character Mode :character ## Mean :2620 ## 3rd Qu.:3929 ## Max. :5238 While reviewing the text responses, I found several variations of non-responses, such as: • Nichts/Nix • n/a - nö • xxx • Weiß ich nicht • Keine Ahnung • Keine antwort • Keine meinung • Keine Angabe To handle these, I created a regular expression to identify and classify these responses as non-meaningful: # Create two additional columns to identify meaningful and non-meaningful text entries meaningful_columns <- function(data, columns) { # Define patterns for non-meaningful answers patterns <- c( "^(99|66)$", "^(nichts|nix|n/a|no|x{1,3}|k\\.?a\\.?)$", paste0( "(wei(s|ss|ß)\\s*(ich|es)?(\\s*leider)?", "\\s*(ni[cx]ht|nicht|nich)|keine?\\s*", "(ahnung|antwort|meinung|angaben|kommentar?))" ), paste0( "kann\\s*(ich|man)\\s*(.*?\\s*)?", "(ni[cx]ht|nicht|nich)\\s*", "(beschreiben|sagen|beantworten|beurteilen)" ), paste0( "das\\s*(wei(s|ss|ß)\\s*(ich|man)|", "kann\\s*(ich|man))\\s*(.*?\\s*)?", "(ni[cx]ht|nicht|nich)" ), "sehr\\s*schwer\\s*zu\\s*sagen", "das\\s*ist\\s*mir\\s*zu\\s*kompliziert", paste0( "^\\s*wei(s|ss|ß)\\s*(ich|es|man)?", "\\s*(ni[cx]ht|nicht|nich)\\s*$" ) ) # Combine patterns into a single regex non_meaningful_regex <- paste(patterns, collapse = "|") 4 data %>% mutate(across({{columns}}, ~case_when( str_detect(., "^-[0-9]+$")~FALSE, str_detect(str_to_lower(str_trim(.)), non_meaningful_regex) ~FALSE, .== "" ~FALSE, TRUE ~TRUE ), .names = "{.col}_meaningful")) } data_meaningful <- meaningful_columns(data_clean, c(european, non_european)) Interestingly, some responses indicate an inability to answer while still offering valuable insights, such as: data_meaningful %>% filter(id %in% c(499,2763,3787,4143)) %>% select(id, european, non_european) %>% kable() id european non_european 499 die frage kann ich nicht beantworten da jedes europäische land in prinzip eigene gesellschaftliche strukturen gesetze ökonomische interessen hat und fast jedes land außer deutschland seine nationalen interessen in den vordergrund stellt hierbei spielen verhalten und eigenschaften einer einzelnenn person keine rolle wenn diese person keine staatsbürgerschaft eines europäischen landes hat 2763dazu kann ich nichts sagen weil europa aus vielen nationen besteht die alle ihre eigenheiten haben ich weiß nicht was einen europäer ausmacht radikale ansichten und verhaltensweisen 3787das kann ich nicht beantworten für mich keine bestimmten eigenschaften kann ich nicht beantworten bzw beurteilen 4143boa was ist das denn für eine schwierige frage keine ahnung jedes land in europa ist anders deswegen gibt es hier keine antwort ja wenn man halt nicht aus europa ist The way I coded it is producing few false negatives—cases where the response is categorized as non-meaningful when it actually contains relevant information — that I need to correct. For instance: data_meaningful %>% filter(id %in% c(4694,5107,5223)) %>% select(id, european, non_european) %>% kable() id european non_european 4694 ich darf in die eu länder frei reisen gesundheitsversorgung und noch viel mehr das kann ich nicht alles aufschreiben kann ich nicht alles schreiben 5107 keine ahnung jedenfalls nicht unschuldige menschen töten personen die auf grund religiöser herkunft morde begehen 5223 es braucht eine einheitliche sprache ich plädiere für spanisch europäische eigenschaften kann ich nicht beschreiben vielleicht offenheit personen die die religion in den vordergrund stellen 5 Data quality is crucial for ensuring validity. One of the first aspects to examine is the non-response rate, as incomplete, ambiguous, or misrepresented text data makes it difficult to draw accurate conclusions about social phenomena. While the non-response rate is relatively high, the large sample size provides a sufficient number of valid responses. # Non repose percentages data_meaningful %>% summarise(across(c(european_meaningful, non_european_meaningful), ~sum(. == FALSE,na.rm = TRUE)/n() *100, .names = "{str_remove(.col, '_meaningful')}")) %>% pivot_longer(everything(), names_to = "column", values_to = "percent_non_response") ## # A tibble: 2 x 2 ## column percent_non_response ## <chr> <dbl> ## 1 european 34.3 ## 2 non_european 38.7 For responses like these, I recognize that the questions may be too complex for some respondents, or they might experience survey fatigue. Some may not want to take the time to write out their answers, may not think in these terms, or might even perceive the question as unimportant. We need to consider these factors carefully when assessing the validity of how we measure European identity content. # People who did not like the question or did not think is these terms data_meaningful %>% filter(id %in% c(419,580,834,1086,1248,1293,1363,2433)) %>% select(id, european, non_european) %>% kable() id european non_european 419 keine besonderen eigenschaften kein bestimmtes verhalten kein ausschluss 580 jeder soll sein wie er möchte komische frage 834 den begriff europäerineuropäer lehne ich generell ab da ich mehrfache staatsbürgerschaften für eine person ablehne kann ich mit dem begriff europäerineuropäer wenig anfangen 1086 schwachsinnige frage ebenso schwachsinnig 1248 das ist blödsinn das ist blödsinn 1 1293 für mich sind alle menschen gleich dito 1363 was soll das für eine frsge sein unbeantwortbar 2433 das kann ich nicht sagen auch das kann ich so nicht sagen mich stören diese beiden fragen sehr sie gehen mir sozusagen gegen den strich # People who though the question was difficult to answer data_meaningful %>% filter(id %in% c(2276,3128,5032)) %>% select(id, european, non_european) %>% kable() 6 id european non_european 2276 demokratisch gemeinsame werte schwer zu beantworten 3128 gute frage aber sehr schlecht zu beantworten wenn sich jemand nicht integriert 5032 das ist mir zu kompliziert siehe vorgehende antwort Although the non-response rate is high, we have gain valuable perspectives from those who believe European identity either does not exist or is irrelevant, those who believe everyone is equal and many other nuances that we would not have been able to obtain with closed-ended questions. data_meaningful %>% filter(id %in% c(1714,1739,1806,1819)) %>% select(id, european, non_european) %>% kable() id european non_european 1714ich habe noch keine person kennen gelernt die ihrem verhalten nach europäisch ist ich glaube dieses denken ist in den meisten köpfen noch nicht angekommen wenn es überhaupt einmal dazu kommt die beschlüsse die aus brüssel kommen sind nicht immer dazu angetan sich als europäer zu fühlen das kann ich nicht beschreiben weil ich alles aus deutscher sicht betrachte aber ich denke es geht den meisten menschen der anderen europäischen staaten ebenso 1739eine identität ist geprägt durch umgebung erziehung und erfahrung es ist schwierig das an einer region festzumachen demokratische grundwerte toleranz und humanismus würde ich als grundpfeiler der europäischen länder gerne sehen auch wenn das leider nicht immer so ist jeder mensch und damit auch jeder europäer ist anders kann man ihm aufgrund von eigenschaften oder verhalten daher sein europäer sein absprechen 1806ich würde das nicht verallgemeinern das selbe wie vorher 1819habe ich mir noch nie gedanken darüber ge macht für mich sollten alle menschen friedlich miteinander auskommen und jeden achten das ist für mich als euro pär erin wichtig gute zusammenarbeit mit allen menschen egal welcher hautfarbe keine ahnung ist mir egal ob er für mich europäisch ist oder nicht TextPrep I will know go through some of the options TextPrep offers to deal with the pre-processing of the text data before the analysis. In the TextPrep tutorial they use skim for an overview of the various variables of the data set. While it is great to get an overview it does not account for all the special cases I just presented above of missing data skimr::skim(data_meaningful) Table 6: Data summary Name data_meaningful Number of rows 5238 Number of columns 5 _______________________ Column type frequency: character 2 logical 2 numeric 1 ________________________ 7 Group variables None Variable type: character skim_variable n_missing complete_rate min max empty n_unique whitespace european 1 1 0 590 62 3396 0 non_european 2 1 0 582 93 3162 0 Variable type: logical skim_variable n_missing complete_rate mean count european_meaningful 0 1 0.66 TRU: 3443, FAL: 1795 non_european_meaningful 0 1 0.61 TRU: 3212, FAL: 2026 Variable type: numeric skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist id 0 1 2619.5 1512.22 1 1310.25 2619.5 3928.75 5238 ����� Machine Translation A really helpful function is machine translation, especially since I am working with German text responses and I do not speak the language. For this assignment I will show how it works for a subset of 100 entries due to the computational time this process takes. # Working with only the meaningful answers subset_data <- data_meaningful %>% filter(european_meaningful == TRUE &non_european_meaningful == TRUE)%>% slice_sample(n = 100) # Translation of German meaningful answers and # creation of translated dataframe using Google Translate translation_google_de <- polyglotr::google_translate(c(subset_data$european, subset_data$non_european), target_language = "en", source_language = "de") translation_google_de <- sapply(translation_google_de, function(x) x[[1]]) subset_data_google <- subset_data # Split the translations back into two separate vectors n<- nrow(subset_data) subset_data_google$european <- translation_google_de[1:n] subset_data_google$non_european <- translation_google_de[(n+1):(2*n)] subset_data_google %>% select(id, european) ## # A tibble: 100 x 2 ## id european ## <dbl> <chr> ## 1 1264 Mutual respectable behavior doesn't go a long way ~ 8 ## 2 3261 understanding of freedom and democracy ## 3 5197 Commitment to values, honesty, solidarity ## 4 3107 He must identify with European culture and believe ## 5 400 democratic and social action, mutual fair support ## 6 625 A peaceful coexistence regardless of origin and re~ ## 7 4803 democratic understanding of tolerance ## 8 5121 Christian values attitude and belief in short Ch~ ## 9 3682 born in Europe, European culture, speaks European ~ ## 10 2988 native Europeans who are willing to find solutions~ ## # i 90 more rows subset_data_google %>% select(id, non_european) ## # A tibble: 100 x 2 ## id non_european ## <dbl> <chr> ## 1 1264 someone who is disrespectful to others ## 2 3261 terror and hate ## 3 5197 disloyalty, discrimination, ruthlessness ## 4 3107 if he doesn't integrate culture belief ## 5 400 no compromise, no democracy, ruthlessness and ther~ ## 6 625 Discrimination against women or degradation of wom~ ## 7 4803 dictatorial attitudes radical Islamism feudal rule ## 8 5121 the attitude of appreciation and belief ## 9 3682 not white Muslim oa typical non-European religion ~ ## 10 2988 Not being born in Europe and or thinking and actin~ ## # i 90 more rows Stopwords Stopwords are commonly understood to as frequently used words that add only little or no meaning for interpretation. In the tutorial they compare three commonly used and general stopword lists of the stopwords package: NLTK, SMART and Stopwords ISO. They do and show the difference in output so let’s do that with this data. # To have all the text in one column subset_data_long <- subset_data_google %>% pivot_longer( cols = c(european, non_european), names_to = "source", values_to = "text" )%>% mutate(source = ifelse(source == "european","european","non_european")) %>% select(id, source, text, everything()) subset_nltk <- subset_data_long subset_smart <- subset_data_long subset_iso <- subset_data_long #function to extract stopwords extract_stopwords <- function(text, source) { # Get stopwords for the specified source stops <- stopwords::stopwords(language = "en",source = source) # Split into words words <- text %>% strsplit("\\s+")%>% 9 labs(title = "What excludes someone?", x = "Number of Words",y = "Frequency")+ theme_minimal() + xlim(0,50) grid.arrange(p1, p2, ncol = 2) 0 100 200 300 400 500 0 10 20 30 40 50 Number of Words Frequency What makes someone European? 0 200 400 0 10 20 30 40 50 Number of Words Frequency What excludes someone? This approach combines theory-driven and data-driven coding strategies. This two-step process allows to build upon existing theoretical frameworks while remaining open to emergent themes in the data. The deductive part started with categories from existing literature on European and national identity (ancestry, birthplace, religion, language, etc.). The inductive phase involved generating new codes based on patterns that emerged directly from the data. I read all responses multiple times, summarizing each one and noting key points to identify potential themes. Through this iterative process, I developed codes from specific text segments, capturing meaningful insights not covered by the predetermined deductive codes. This turned out to be really valuable because I found several categories that have been largely overlooked in previous research. Through this hybrid process, I identified twenty distinct codes grouped into seven thematic categories. These categories showed up consistently across both questions and often mirrored each other structurally. Traits used to affirm European identity (like “Being born in Europe” or “Holding a European passport”) were commonly negated when describing exclusion (“Not being born in Europe” or “Not having a European passport”). The coding process involved two research assistants, both native German speakers, who initially coded 15% of both datasets. After achieving Cohen’s Kappa above 0.80 for all categories, demonstrating that the categories are reliably measurable, each rater was assigned one complete dataset to code. These 20 binary indicators (present/absent for each category) constitute the measurement of European identity content. European Identity Inclusion Categories The following table presents the 20 categories used to code responses to “What does it take to be European?” 16 Table 11: Coding Categories for European Identity Inclusion Code Description Example Physical appearance Physical characteristics associated with European appearance Light skin color Ancestry Having European heritage, ancestors, or family origins A European is someone whose parents are from Europe Birthplace Being born in a European country Born in Europe Culture Engaging with European traditions, customs, lifestyle, and shared cultural practices Adhering to the European way of life, culture, traditions Integration Adapting to or/and assimilating into European society One should integrate into society, not force one’s beliefs, lifestyle, etc. History Awareness of and connection to shared European history and collective memory Having a shared past Knowledge Being informed about European affairs Being well-informed about the European Union and its member states European feeling Collective sense of belonging, European identity, shared goals beyond national borders Europe as a political community with a ‘we-feeling’ Values Abstract principles that guide ethical or moral frameworks such as equality, freedom and solidarity Respect for individual dignity Traits and behavior Observable actions or character traits that reflect personal behaviors such as being diligent or charismatic Being reliable and punctual Democratic attachment Support for democratic institutions, process, and commitment to democratic attitudes and values Acting democratically and defending democracy (Euro)nationalismSense of national identity and pride Love one’s own country and people above all else Rights and obligations Fundamental rights and duties Adhering to law and order Legal citizenship Fulfilling formal requirements for EU citizenship or passport Possessing citizenship of an EU country Personal contribution Individual responsibility to contribute economically or/and socially to society Ready to earn one’s own living, not living at the expenses of the state Economic security Economic benefits and protections of being in the EU Free trade and movement in Europe Place of residence Living, working, or having grown up in Europe Someone who lives and works in Europe Religion References to religious beliefs and practices, primarily Christianity or opposition to Islam Being Christian, not wearing a headscarf Language Speaking or actively learning European languages as marker of identity Speaking a European language European identity rejection Rejection of European identity as meaningful or relevant social category There is no European identity European Identity Exclusion Categories The following table presents the 20 categories used to code responses to “What excludes someone from being European?” 17 Table 12: Coding Categories for European Identity Exclusion Code Description Example Physical appearance Having dark skin or other physical features not perceived as European Being black Ancestry Not having European heritage, ancestors, or family origins Not having European roots Birthplace Not being born in Europe Not born in Europe Culture Not being part of European culture; having a different cultural background or lifestyle Not part of European culture, different way of life Integration Refusing to integrate to European society People who do not want to integrate History Lack of knowledge or connection to European history No historical awareness Knowledge Disinterest in the EU, its member states, or European affairs No knowledge about the EU European feeling Rejecting European unity, anti-European sentiment, strong nationalism, or not identifying as European Nationalist thinking and behavior, does not identify with Europe Values Rejection of what are considered European values Rejects European values; not tolerant or respectful Traits and behavior Exhibiting negative or anti-social behaviors such as intolerance, selfishness, lack of empathy, persistent dishonesty, or violence Intolerant, selfish, violent, persistent liar, lack of empathy Democratic attachment Not supporting democracy and/or supporting dictatorships or autocracies People who rejects democracy or even fights against it (Euro)nationalismNot having national pride, not thinking nationally or being hostile to the nation Not thinking nationally Rights and obligations Violating human rights, breaking laws, refusing to comply with European and/or national laws Anyone who does not abide by rules and laws Legal citizenship Not having European citizenship, passport, or other formal legal status No European passport Personal contribution Not willing to contribute economically or socially, abusing welfare or not providing for oneself Only benefit from the social system without giving anything in return, a welfare abuser Economic security Not recognizing benefits of being in the EU, supporting withdrawal from the euro, no interest in freedom of movement Someone who wants to close the borders within Europe Place of residence Not living, working, or having resided in Europe Lives outside Europe Religion Not being Christian, being Muslim, or holding religious beliefs perceived as non-European Being Muslim, veiled women Language Not speaking any European languages If he does not speak a European language fluently European identity rejection Denying the existence of a European identity or any defining European characteristics There is no such thing as a European identity These categories demonstrate the multifaceted nature of how Germans conceptualize European identity boundaries, incorporating both inclusive criteria (what makes someone European) and exclusive criteria (what excludes someone from being European). 18 Conclusion In this project, I explored a novel approach to measuring the complex social construct of European identity using open-ended survey questions within the GESIS Panel. This method allows for more nuanced responses than traditional closed questions but also requires careful validation. To ensure rigor, I applied the ValiText framework, focusing on substantive evidence to establish a strong theoretical foundation. Additionally, I used TextPrep to tackle text preprocessing challenges, such as handling non-responses and selecting the right stopword lists. Despite the high non-response rate, the depth of the remaining answers provides a solid foundation for further analysis. Moving forward, I will complete the ValiText framework by collecting structural and external evidence to strengthen the validation process. Future steps include examining how these identity categories cluster within individuals through latent class analysis, exploring how different conceptualizations of Europeanness correlate with sociodemographic characteristics and political attitudes once the full GESIS dataset becomes available, and assessing the predictive validity of this measures against established indicators of European identification and EU support. This comprehensive approach will provide deeper insights into how German citizens construct and contest the boundaries of European belonging in an era of increasing political polarization around questions of identity and migration. References Birkenmaier, Lukas, Claudia Wagner, and Clemens Lechner. 2023. “ValiText – a Unified Validation Framework for Computational Text-Based Measures of Social Constructs.” Roberts, Margaret E., Brandon M. Stewart, and Dustin Tingley. 2019. “Stm: An R Package for Structural Topic Models.” Journal of Statistical Software 91 (2). https://doi.org/10.18637/jss.v091.i02. 19