Refining Audience Engagement: Leveraging Text-based Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits

RAIS Journal for Social Sciences | VOL. 9, No. 2, 2025 ISSN 2574-0245 (Print) | ISSN 2574-1179 (Online) | DOI: 10.5281/zenodo.17468861 ! 60! ! Refining Audience Engagement: Leveraging Textbased Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits ! Wen Chen DENQ1, Wen Chin HSU2 1Faculty School of Management, California Lutheran University, USA, [email protected] 2Associate Professor, Computer Science Department, California State University, Northridge, USA hsu.wen-[email protected] ! Abstract: This study examines how small, Los Angeles-based, advocacy-focused nonprofit organizations (NPOs) use Facebook to engage audiences through information sharing, calls for action, and community-building. More importantly, it explores text-based audience comments—a novel metric for measuring audience engagement in social media studies in advocacy-focused nonprofit organizations (advocacy-NPOs). Rather than relying solely on traditional metrics (e.g., frequency of “likes,” “comments,” and “shares” of NPO-generated posts), we analyzed usergenerated content such as text-based audience comments to capture nuanced engagement as well as audience preferences and interactions. The findings reveal that NPO-generated posts prioritized Information (92%) and Action (71.3%) categories, with limited focus on the Community. By contrast, text-based audience comments were most common in the Community category (80.3%), with frequent replies to NPOs including expressions of gratitude and demonstrations of support. This discrepancy between NPOs’ organizational posts emphasis on Information and audiences’ preference for community underscores the importance of peer interaction and social bonding as primary motivators for engagement. Additionally, both NPO-generated posts and text-based audience comments contained advocacyfocused content, so this pattern reflects alignment between organizational missions and audience interests. These insights can help advocacy-NPOs develop practical strategies to enhance their social media impact and strengthen audience engagement. Keywords: Social Media Engagement, Advocacy Nonprofits, Facebook, Audience Comments ! Introduction Social media has revolutionized how nonprofit organizations (NPOs) engage with their audiences. It not only provides an efficient channel for exchanging information and mobilizing collective action but also helps boost NPOs’ reputation (Namisango et al., 2019). Aligned with NPOs’ core mission to address social objectives, these interactive functions have driven the widespread adoption of social media among NPOs. Social media, especially for smaller NPOs, is a cost-effective communication tool for expanding outreach and maintaining meaningful engagement with supporters (Campbell et al., 2013; Guo & Saxton, 2014; Zhang et al., 2013). According to the 2023 Nonprofit Tech for Good Report, approximately 91% of NPOs in the United States maintain official websites, which illustrates their widespread digital presence. Among social media platforms, Facebook is the most popular and is used by 96% of surveyed NPOs, followed by Instagram (73%), Twitter (59%), LinkedIn (49%), and YouTube (44%; 2023 Nonprofit Tech for Good Report, 2023). DENQ & HSU: Refining Audience Engagement: Leveraging Text-based Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits ! 61! Research indicates that 55% of people who engage with NPOs on social media subsequently take action to support those NPOs’ missions (NP Source, 2022). Social media has reshaped the nature of interactions between NPOs and their audiences by helping to deepen existing relationships, and simultaneously reach new audiences (Schmitz et al., 2020). Online fundraising has also emerged as a crucial revenue source, particularly for smaller NPOs, which raised 18% of their total funds from digital channels—this applies even more to larger organizations (Blackbaud Institute, 2022). Beyond Audience Comments While most social media studies in the nonprofit sector have focused on quantifying engagement to measure audience interactions by counting the number of “likes”, “comments”, and “shares” of posts, scholars have increasingly recognized that these metrics only touch the surface of audience interaction (Appleby, 2016). Although qualitative analysis offers a deeper understanding of NPO-generated posts, most research to date has only focused on analyzing messages from X (formerly Twitter; Anagnostopoulos et al., 2017). We believe user-generated content, such as text-based audience comments, could provide deeper insights into how individuals (collectively, the audience) perceive, respond to, and participate in advocacy efforts with NPOs online. Therefore, further exploration of the qualitative dimension, particularly audience interactions, beyond traditional metrics, is essential for a comprehensive understanding of social media effectiveness of advocacyNPOs (i.e., nonprofit organizations that have advocacy as their core mission). By analyzing these comments, advocacy-NPOs can gain a nuanced understanding of what content captures their audience's attention, how messages resonate with their audience, and how they motivate community members to act. This understanding is especially critical for advocacy-NPOs, where community building plays a vital role in achieving organizational goals. Advocacy organizations are entities dedicated to shaping public policy and community opinion on chosen issues, interests, or causes (Hojnacki et al., 2012; Schlozman et al., 2015). These groups use a range of tactics, including lobbying, public campaigns, and litigation, to achieve their social, political, environmental, and economic goals. In this study, we focused on advocacy-NPOs that are registered as 501(c)(3) organizations, as defined by the Internal Revenue Service (IRS), and excluded 501(c)(4) and 501(h) organizations. Social media has transformed how these advocacy-NPOs communicate. With social media platforms, these advocacy-NPOs can interact with both their members and the public in real time (Bürger, 2015; Guo & Saxton, 2014; Saxton & Wang, 2014). The decentralized nature of social media platforms facilitates two-way communication, promotes deeper audience interaction, and amplifies the impact of advocacy initiatives (Seelig et al., 2019). Additionally, they can enhance public discourse, mobilize supporters, and transform traditional strategies for public engagement. Literature Review Although the value of social media in NPOs is widely acknowledged, few studies have explored how text-based audience comments shape social media engagement in the context of NPO advocacy (Appleby, 2016). This study helps fill this gap by systematically analyzing both NPO-generated posts and the associated text-based audience comments to better understand how information, action, and communities interact in digital spaces. Evolution of Engagement Metrics The evolution of measurement engagement within NPOs has shifted from relying on quantitative metrics (e.g., frequencies of “likes,” “comments,” and “shares”) toward RAIS Journal for Social Sciences | VOL. 9, NO. 2, 2025 ! 62! adopting more nuanced and interpretive frameworks. Sentiment analysis, one example of quantitative analysis, has emerged as a data-driven approach that categorizes messages as positive, neutral, or negative feedback and adds interpretive dimensions to engagement assessment (Balfour, 2020; Bhati & McDonnell, 2024). Although traditional metrics remain dominant, these approaches have been criticized for their inability to capture the depth of stakeholder interactions and audience preferences. Lovejoy and Saxton’s (2012) pioneered qualitative content analysis to examine nonprofits communication strategies. Building on this, Campbell and Lambright (2020) argued that effective engagement analysis should transcend mere numerical counts and include qualitative analysis of posts that reflect community and stakeholder relationships. Ksiazek et al. (2016) highlighted the value of analyzing user-generated comments as qualitative analysis of audience comments can generate deeper insights into user preferences and organizational engagement practices. The Emergence of Community-Building as a Strategic Focus for Advocacy-NPOs Despite the growing recognition of community building as critical for online engagement, limited research has explored how advocacy-NPOs use social media to build communities with their supporters. Lovejoy and Saxton (2012) introduced the “information-communityaction” framework for NPOs’ social media strategies. They emphasized that communitybuilding messages are crucial for deepening engagement, especially in an advocacy context. Guo and Saxton (2014) subsequently found that advocacy-NPOs used social media not only to share information or solicitations for action but also to build communities of supporters around shared causes. Further studies suggest that community-related content typically generates higher audience engagement than information-only posts (Maxwell & Carboni, 2016; Xue et al., 2024), because audiences tend to show stronger preferences for content that fosters a sense of community over purely informational messaging (Campbell et al., 2013; Ksiazek et al., 2016). Morgan et al. (2024) examined how NPOs leverage the social media platform TikTok to enhance their missions, as the fast-changing nature of TikTok challenges NPOs’ traditional long-term planning models. Text-based Audience Comments as an Authentic Measure of Engagement Text-based audience comments offer richer and more authentic insights into engagement than quantifiable metrics represented as numerical totals associated with a post, and even NPO-generated posts. Traditional metrics are critiqued for providing a surface-level assessment of engagement by overlooking the qualitative dimensions inherent in user interaction (Baccarella et al., 2021). Furthermore, NPO-generated posts tend to be creatorcentered rather than audience focused, so they could not directly reflect the genuine audience preferences. Xue et al. (2024) emphasized the significance of user-generated content (e.g., text-based audience comments) in promoting authentic connections between NPOs and their audiences; their exploration supported analyzing text-based audience comments as a means of engagement. Empirical inquiries demonstrate this distinction as Young et al. (2020) suggested that the comments could serve as indicators of deeper engagement on health promotion interventions. Liadeli et al. (2022) observed that users’ comments may indicate a higher level of user involvement compared to quantitative metrics and provide richer insights into preferences and the development of community. DeMasters et al. (2024) stated that NPOs’ TikTok content, which is community-oriented, does not align with their information-sharing strategies. This discrepancy suggests that audiencegenerated comments rather than NPO-generated content more accurately represent authentic community engagement. DENQ & HSU: Refining Audience Engagement: Leveraging Text-based Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits ! 63! Research Questions According to social presence theory, the more interactive and social a platform is, the greater its engagement levels tend to be, as users are encouraged to express themselves, share experiences, build relationships, and receive feedback (Men et al., 2018). Therefore, these concepts were applied to measure engagement levels and identify relevant content by focusing on higher levels of engagement. Based on those theoretical foundations, we develop the following questions: RQ1: How do small, Los Angeles-based NPOs engage their audiences on Facebook to share information, call for action, and build communities? RQ2: To what functions or purposes (information sharing, action mobilization, or community building) do text-based audience comments respond in Facebook posts by small, Los Angeles-based NPOs? Methodology Data Collection and Reasoning The dataset was created by selecting NPOs from the most current version (April 2022) of Nonprofit Explorer. The selection criteria for the NPO population and final samples are detailed in Appendix Table A1. A total of 505, a verified dataset size, organizational posts from seven NPOs were collected over a three-month period (July 1 to September 30, 2022) using web scraping tools. Of the 505 posts, 150 were randomly sampled for data analysis, and 76 of the 149 audience comments corresponded to the 150 NPO-generated posts employing standard data science techniques such as descriptive statistics (e.g., measures of central tendency). The sample sizes of 150 posts and 76 corresponding text-based audience comments were selected for two main reasons. First, the basic unit of analysis was the content of a Facebook message, not an individual NPO. Prior research on NPO communication and public marketing has demonstrated that social media messages, especially Facebook posts, which do not have character limits, are information-rich and suitable for both qualitative and quantitative analyses (Dolan et al., 2019). When content analysis is employed, the basic unit is a message (e.g., a Facebook post) rather than an organization. Because each message can be represented as a data point, a sufficiently large sample of messages that are rich in content and engagement cues is likely to generate meaningful insights (Dolan et al., 2019). Second, two preliminary pilot studies were conducted to refine the criteria for assessing social media content between NPOs and their audiences (Appendix Table A2). The pilot findings confirmed that small-scale analyses of social media posts can generate critical insights by identifying practical details such as appropriate research methods, optimal periods for data collection, and the most suitable social media platform (Julious, 2005). Data Analysis and Coding Procedure Our analysis followed a mixed-methods approach, a descriptive quantitative analysis of engagement metrics as well as a qualitative thematic analysis of the 150 NPO-generated posts utilizing web scraping tools, and the 76 text-based audience comments. Then, the results were compared and examined by using data science classification models to avoid human bias. Specifically, we conducted a descriptive quantitative analysis of engagement metrics to provide summary insights including central tendency and spread, comparative summaries and percentages. We then performed a qualitative analysis to categorize the messages into three labels (i.e., information, community, and action), using MAXODA, a content coding framework. The coded messages were further examined and compared using large language model (LLM) systems from the machine learning field. RAIS Journal for Social Sciences | VOL. 9, NO. 2, 2025 ! 64! For the deductive analysis, all NPO-generated posts and audience comments were categorized into three areas (information, community, and action) and coded based on the 11 subcategories from Lovejoy and Saxton (2012) along with two additional subcategories (mission and organizational information) from Campbell and Lambright (2018). An inductive analysis was used to identify new subcategories within each category (information, action, and community). Each Facebook message, all NPO-generated posts and audience comments, was coded by segment, allowing for multiple codes per message, thereby capturing the complexity of audience engagement with the content (Denq, 2024). Results Qualitative analysis of NPO-generated posts We classified NPO-generated posts on Facebook into three categories (Information, Action, or Community) and further broke down accompanying subcategories to catch more nuanced meaning. Information category posts (with added advocacy–related subcategory) During the coding process, one subcategory—advocacy—was identified inductively for information-related messaging (Table 1). Advocacy-NPOs primarily focused on reinforcing their organizational mission, vision, and values, with raising awareness of their advocacy initiatives and brand identity as the top priority. With 39 posts, event-related content constituted a significant proportion, including detailed (particularly advocacy-oriented) events of NPO communication that often combined calls for volunteers or other actions (Denq, 2024). Advocacy-related Information focused on relevant legislation or regulations, providing a link to contact legislators or local policymakers, which ranked among the top three content categories. This subcategory identified through inductive analysis is proposed as an addition to the Information category within our framework, especially for advocacyNPOs. Table 1. Subcategories of Information Category Posts Subcategory Key contents #Post As a % of all informationrelated posts (138 posts/ 175 codes) Mission Information related to its specific mission to raise awareness of the organization’s brand 88 63.77%/50.29% Events Information to promote organizational events/activities 39 22.29%/28.26% Advocacy (New) Information-related advocacy and lobbying such as policies or petitions 25 6.8%/18.12% Organizational Information related to organizational programs, services, board members, or photos posts for events 23 6.3%/16.67% Total Single, double, or triple codes 175 DENQ & HSU: Refining Audience Engagement: Leveraging Text-based Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits ! 65! Action category posts For the 107 action-related messages, the analysis presented is in Table 2. The most common goal was the promotion of an event and inviting the audience to join. Action-related posts were divided into several subcategories, based on specific NPO goals. The primary goal was to promote an event and invite an audience to join. This content category was followed by action posts designed to empower audience members in specific ways by “learning how to help.” The requests to “learn how to help” entailed specific actions in support of the mission, especially for organizations with more complicated social issues such as homelessness. The third most important category was lobbying and advocacy, which often included requests to contact public officials or local representatives regarding relevant policies. Table 2. Subcategories of Action Category Posts Subcategory Key contents # Posts As a percentage 107 posts/150 codes Promoting an event Invite audience to attend the event or activities 41 27.32%/27.33% Learn how to help Request to learn how to help NPOs 38 25.33%/25.33% Lobbying and advocacy Request that the audience contact public officials to lobby for policies or petitions 21 14.00%/14.00% Following the organization on social media Invite the audience to follow NPOs on official websites or social media sites 15 10.02%/10.00% Donation appeal Request monetary and in-kind donations 14 13.08%/9.33% Know the NPOs better (new) Request to read/learn more information about organizations such as missions 10 10.35%/6.67% All others (miscellaneous) Recruit volunteers and employees; join another site or vote for the organization; posted positive feedback 11 10.33%/7.28% Total Double or triple codes 150 Community category posts The community-related messages served to interact, share, and communicate with audiences (or stakeholders) to eventually facilitate online and offline community-building. Table 3 provides a breakdown of these messages based on three major subcategories. The first subcategory was “mentions and acknowledgments” of current and local events that were not initiated by the organization itself as well as giving thanks and recognition. The second subcategory, “giving recognition and thanks,” represents acknowledging the contributions of supporters, volunteers, and followers. This type of message received the largest number of comments, with most thank you messages related to direct services, rather than financial support. Community-related messages reflect NPOs’ broad RAIS Journal for Social Sciences | VOL. 9, NO. 2, 2025 ! 66! perspectives on social issues as they engage larger audiences through their organizational or personal affiliations. Table 3. Subcategories of Community Category Posts Subcategory Key contents # of Posts As a percentage of (37 posts/47 codes) Acknowledgment of current and local events (by affiliated organizations) Events not initiated by the organization 25 67.57%/53.19% Giving recognition and thanks Recognition of individuals or groups 19 51.35%/41.43% Responses to replied to messages Responses as interactive communication 3 8.11%/6.38% Total Double and triple codes 47 Qualitative Analysis of Text-based Audience Comments and Their Engagement Levels The audience comments were analyzed and separated into three categories—Information, Community, and Action—based on the framework from Lovejoy and Saxton (2012). In Tables 4, 5, and 6, text-based audience comments displayed a dominant interest in Community-related content, followed by Action and Information categories. Information category text-based audience comments In Table 4, Information-related comments (a total of thirty-eight comments) from the audience were also classified under each of the four sub-categories: mission, events, advocacy, and organization-related. Among audience comments, information-related interactions primarily focused on mission-related discussions and highlighted significant engagement with narratives about homelessness and organizational missions. Despite being the least frequent category, mission-focused content elicits substantial audience interaction. Table 4. Text-based Audience Comments Across Information Subcategories Key contents # of subcategories coded As a % of total information contents messages (37 comments/37 codes) Mission Comments on organizational mission 24 63.89% Event Comments about advocacy 6 16.67% Advocacy Comments on events 4 11.11% Organization Comments on organization 3 8.33% Total Single, double, and triple codes 37 DENQ & HSU: Refining Audience Engagement: Leveraging Text-based Audience Comments as an Evolution Metric in Small 501(c)(3) Advocacy-Focused Nonprofits ! 67! Action-related text-based audience comments As Table 5 shows, Action-related comments accounted for a notable portion of audience responses and emphasized support for NPO activities and community advocacy. This category exhibited dynamic engagement in which audiences expressed support, provided positive feedback, and participated in advocacy efforts. Table 5. Text-based Audience Comments Across Action Subcategories Key contents # of subcategories coded As a % of total information contents messages (44 comments/63 codes) Help NPOs and their audience Comments about the actions the audience did voluntarily to support organizations and their community 21 33.33%/47.73% Other-replied with positive feedback Comments to provide positive feedback to organization and other users’ posts 14 22.22%/31.82% Advocacy and lobbying Comments to advocate for NPOs and request to vote or contact local representative for NPOs 9 14.29%/20.45% Questions about how to help Comments to inquire about how to support NPOs and audience 9 14.29%/20.45% Knowing the organization better Comments to express their intention to better know the organization by acting as the audience suggested 8 12.70%/18.18% Donation appeal Comments in response to donation request 2 3.17%/4.55% Total 63 Community-related text-based audience comments Based on Table 5, the most engaged category—Community-related comments— demonstrated a vibrant interaction among the audience, with discussions centering on public replies, expressions of gratitude, and shared support for organizational endeavors. RAIS Journal for Social Sciences | VOL. 9, NO. 2, 2025 ! 68! Table 6. Content Comments Across Community Subcategories Key contents # of subcategories coded As a % of total information content messages 62 comments/97 codes Public replies The audience replied to other audiences’ posts (or NPOs’ posts) 30 48.39%/30.9% Giving recognition and thanks The audience expressed appreciation for the organizational works 26 41.94%/26.8% Demonstrate the support The audience validated the relationship with the NPOs 23 37.10%/23.71% Sharing current news and local events The audience shared current and local events related to NPOs (not initiated by NPOs) 18 29.03%/18.56% Total 97 ! The Comparison of NPO-generated Posts and Accompanying Text-based Audience Comments by Content Category Table 7 provides the classification of the 150 NPO-generated samples and accompanying 76 text-based audience comments according to the content focused on information, action, or community. The findings revealed that Information distribution was the primary purpose in 92% of NPO-generated posts, compared with 48% of text-based audience comments had this focus. Calls for action appeared in 71.3% of NPO posts but in only 57.9% of text-based audience comments. Community-building was discovered in 24.7% of NPO posts, while a substantial 80.3% of text-based audience comments focused on this category. Table 7. Comparing Organizational and User-generated Foci Community Action Information Organizational posts (n=150) 24.7% (37) 71.3% (107) 92% (138) Audience comments (n=76) 80.3% (62) 57.9% (44) 48% (37) Discussion & Conclusion Similarity in Prevalence of Advocacy between NPO-generated Posts and Text-based Audience Comments The advocacy-related content categorized under Information and Action may have gained popularity for two reasons: the high alignment between NPOs’ missions and their audience’s interests, and the expanded criteria helped to identify advocacy-related content (Tables 1, 2, 3, 4, and 5). We found that the higher the alignment between NPOs’ mission