In March 2020, the World Health Organization (WHO) declared COVID-19 a pandemic, underscoring the critical need for effective public health measures, including vaccination, to control the spread of the virus [1]. Vaccination has historically been one of the most effective interventions in preventing infectious diseases and protecting population health [2]. Despite the availability and widespread administration of COVID-19 vaccines, many individuals remain hesitant to receive them, raising serious public health concerns [3]. WHO identified vaccine hesitancy as one of the top 10 global health threats in 2019, even before the onset of COVID-19 [4]. This persistent hesitancy has been driven by factors, including vaccine safety concerns, fears of side effects [1], mistrust in the health care system, various socioeconomic and demographic influences [5,6], and the usage of social media platforms for gathering information [7,8].

These factors do not affect all communities equally. Prior research has shown higher vaccine hesitancy in some ethnic minority populations [9]. Studies examining racial and ethnic disparities in COVID-19 vaccination found that people with lower income or education, women, and some minority groups, including Black and Hispanic populations, were more likely to delay or avoid vaccination [10,11]. Such findings underscore the importance of localized analysis, since state- and national-level patterns may mask community-specific concerns.

This is particularly relevant in Texas. During the 2021 legislative session, vaccine hesitancy discussions reflected concerns about medical freedom, vaccine effectiveness, and vaccine safety [12]. While these concerns overlap with national patterns, they may not fully capture barriers affecting smaller communities, including Hispanic populations. National-level data have shown that Hispanic individuals were less likely than White individuals to receive an initial vaccine dose [13]. Other research has suggested that messaging emphasizing family protection and directly addressing mistrust may be more effective for Latino communities [14]. Together, these findings suggest that interventions may need to be tailored to not only demographic groups but also local settings.

Social media has been widely used to study COVID-19 vaccine hesitancy [15-18]. Some studies focused on sentiment [19-25], whereas others examined misinformation [16,26,27]. Prior work has analyzed English-only [28,29], Spanish-only [30-33], and multilingual datasets [34-37]. For example, one study examined vaccine misinformation in English and Spanish [34], and another showed that multilingual discussion around AstraZeneca and Omicron underscored the need for culturally informed public messaging [35]. However, prior research has rarely linked bilingual social media discourse to real-world vaccination uptake at the county level.

This study addresses that gap by analyzing English and Spanish Facebook posts from public pages and groups in Tarrant County, Texas, across 2021 and 2022. Tarrant County is a large, diverse urban county with a substantial Hispanic population (30.5%) [38], making it a useful setting for examining cultural influences on vaccine uptake. By combining bilingual social media data with vaccination data disaggregated by time and ethnicity, this study provides a more granular view of how online discourse relates to population-level vaccination behavior.

The study addresses three research questions:

This analysis is informed by two complementary frameworks: (1) the health belief model [39] and (2) the agenda-setting theory [40]. The health belief model suggests that health behaviors are shaped by perceived risk, benefits, barriers, and cues to action [39]. In this context, social media posts may act as cues to action by shaping how people think about vaccine safety, effectiveness, and accessibility. The agenda-setting theory emphasizes the role of media in influencing which issues are seen as important [40]. Applied here, it suggests that frequent discussion of issues, such as side effects, mandates, and misinformation, may increase their salience and thereby shape public attitudes and behaviors. Together, these frameworks provide a useful basis for understanding how vaccine-related discourse on Facebook may be associated with vaccine uptake.

In collaboration with the Tarrant County Public Health Department, this study used monthly vaccination records that included ethnicity, enabling the analysis of both total and Hispanic vaccination uptake. The study also introduced a scalable approach for annotating vaccine-related posts using GPT-4. Post-level variables were defined through iterative manual coding of 200 English and 200 Spanish posts, and these were then used to construct prompts for labeling the full dataset across 16 variables. By examining stance, misinformation, and a wide range of vaccine-related topics, this study has identified message types associated with higher vaccination uptake and highlighted cultural differences that may help guide future public health communication.

Figure 1 presents the study framework, which included the following 4 stages: Facebook data collection, data filtering, post classification, and statistical analysis.

Many studies have leveraged social media platforms to assess vaccine hesitancy and vaccination acceptance. For example, studies have examined sentiments and misinformation about vaccines on platforms like Twitter, Facebook, YouTube, and WhatsApp [29,59-61]. This study collected data from one of the most widely used social media platforms, Facebook [62]. Facebook was considered a suitable platform for this study, as a previous study has reported that 66% of the Hispanic population uses this platform [63]. CrowdTangle application programming interface [64], a social media listening tool from Meta, was leveraged to query the data based on the selected keywords, location (Tarrant County), and period (January 1, 2021, to December 31, 2022). It is important to note that the data provided by CrowdTangle originate from public pages and public groups, not individual users, while the selected location yields posts relevant to that region based on the location distribution of page/group followers or participants. The keywords used for collecting the English dataset were adopted from the CoVaxxy dataset [65], which is an extensive collection of COVID-19 vaccine-related tweets collected with a unique keyword list created by using the snowballing technique. Some examples include pfizervaccine, covax, and getvaccinated. The keywords for collecting the Spanish dataset were adopted from another related study analyzing Spanish Facebook posts in Texas [30], where the keyword list was also created by incorporating a snowballing technique to identify relevant keywords specific to Spanish COVID-19 vaccine-related posts. Some examples include vacuna covid, segunda dosis, and efecto secundario covid. Finally, 21,737 English posts and 1596 Spanish posts were collected. There may be a few reasons for the lower number of Spanish posts. First, the percentage of the Hispanic population in Tarrant County is around 30%. Therefore, the percentage of Spanish versus English posts is expected to be lower. Second, some Hispanic users might still post in English. While this study aimed to compare social media post categories between Hispanic and non-Hispanic populations, a key limitation is the inability to definitively determine the ethnicity of users based on post language. The assumption that Spanish-language posts originate from Hispanic individuals and English-language posts originate from non-Hispanic users may not always be accurate. This introduces potential biases in the analysis, as users may post in a language that does not reflect their ethnicity. This limitation affects the validity of the conclusions drawn about how different demographic groups express attitudes toward vaccination. Despite this, the findings remain relevant because they identify key post categories associated with vaccination uptake in each language group. These insights suggest that the sample is still broadly representative, even if individual user ethnicity cannot be verified with certainty.

The study was reviewed by the University of Texas at Arlington Institutional Review Board (protocol #2022‐0072), which determined that it did not meet the definition of human subjects research under 45 CFR 46.104 (d)(2i), Revised 2018 [66]. As such, the study was classified as non–human subjects research and did not require Institutional Review Board approval. The project used publicly available data and was consistent with public health surveillance activities, as it involved the systematic collection and analysis of population-level information to monitor health-related behaviors and information patterns, without interaction with individuals or access to identifiable private information, which has been outlined in Centers for Disease Control and Prevention Policy 557 [67]. All Facebook data analyzed in this study were obtained from public Facebook pages and groups only. No private or identifiable user information was collected, and data were analyzed in aggregate and were fully deidentified prior to analysis, ensuring the protection of user privacy and compliance with ethical research standards for the secondary use of data.

Although querying based on vaccine-related keywords yielded many Facebook posts, an essential step before further data processing involved ensuring that the posts obtained were related to COVID-19 vaccines. Initially, a random sample of 1000 English posts and 500 Spanish posts was extracted from the whole dataset. Then, 2 pairs of annotators (2 annotators for the English posts and 2 for the Spanish posts) manually labeled the samples by indicating whether each post was related to COVID-19 vaccines. The annotators labeling the English dataset were English speakers, with one annotator having a computer science background and the other having a public health background. The annotators labeling the Spanish dataset were Spanish and English speakers, with one annotator having a computer science background and the other having a public health background. The annotators provided a simple binary label for each post, with “1” used to indicate a relation with COVID-19 vaccines and “0” used to indicate no relation. Once the labeling was completed, the annotators met to discuss and resolve any disagreements in labeling. The Cohen κ scores for the labeling of the English and Spanish datasets were 0.76 (substantial agreement) and 0.14 (slight agreement), respectively. The disagreements in the Spanish sample were manually checked, and the low Cohen κ was found to be mainly due to one of the annotators suggesting that posts of job postings mentioning vaccination requirements in the application process should be considered vaccine-related. However, the other annotators did not agree, as these posts did not provide any further discussion regarding the vaccines. Hence, the annotators decided that these posts should not be classified as vaccine-related. In the English dataset, 60.5% (605/1000) of posts were related to COVID-19 vaccines, while in the Spanish dataset, 83.2% (416/500) of posts were related to the vaccines. By manually examining non–vaccine-related posts, lists of keywords relevant to such posts were created.

Some manually identified keywords specific to non–vaccine-related posts were dog, heartworm, hiring, music, etc, corresponding to posts related to animals, events that required COVID-19 vaccination, job openings, or document services. After stemming the texts of the posts and the keywords, posts containing any of the keywords from the corresponding lists were classified as non–vaccine-related. Using this methodology, the percentages of misclassified posts when comparing the obtained labels to the ground-truth dataset were 5.9% (59/1000) and 5.6% (28/500) for the English and Spanish datasets, respectively. Therefore, we filtered the whole dataset using these keywords (ie, discarded posts irrelevant to COVID-19 vaccination). We eventually classified 13,819 English posts and 1249 Spanish posts as relevant.

Previous literature has suggested many reasons for vaccine hesitancy, such as the fear of potential vaccine side effects, efficacy issues, mistrust toward the government or health system, misinformation spread, and others [11,14,26,68]. Thus, this study assessed English and Spanish Facebook posts to investigate how stance, misinformation, informative versus noninformative posts, and post categories/topics (such as vaccine side effects, government, health system, and others) were associated with vaccination rates in Tarrant County in 2021 and 2022. In more detail, the goal was to identify the potential reasons behind vaccine hesitancy in social media posts and examine the association between online activity and actual vaccination uptake. A summary of relevant variables related to social media posts is illustrated in Figure 2.

This study attempted to cover all the reasons for vaccine hesitancy and label them in Facebook posts. However, it is important to note that direct measurement of vaccine trust is challenging, despite its significant role in vaccine reluctance [11]. Therefore, the most prevalent reasons for vaccine hesitancy were incorporated as topics in this study.

Additional classifiers were required to annotate each post with the correct labels for each variable. Initially, random samples of 200 posts were extracted from both the English and Spanish datasets. Then, 2 English speakers labeled the English sample (one with a computer science background and the other with a background in statistics), while 2 Spanish speakers (with public health and computer science backgrounds) and 1 English speaker (with a computer science background), who translated the posts, labeled the Spanish sample using the variables described in Figure 2. The labeling codebook is provided in Multimedia Appendix 1. It was used to train the annotators in performing the labeling. The codebook was initially created by incorporating the knowledge from previous literature, which also helped in defining variables in the Variables section. Manual labeling and defining the codebook were iterative processes. Definitions had to be adjusted multiple times to ensure accuracy and interannotator consistency, and some categories were added by open coding, as discussed in the Independent Variables section. The following are 2 examples from the codebook, including variable definitions and, where applicable, representative examples or supplementary materials:

If the post was irrelevant to COVID-19 vaccination (as some posts remained after keyword filtering), the annotators did not annotate the post. In some instances, annotators had to open the post. They read the title of the article shared within the post, as sometimes the text of the post did not explicitly mention the vaccine, but it still commented on the article about it. Such posts were considered relevant. After the labeling was completed, the annotators met to resolve any conflicts that were raised in the labeling process to create final labels. However, as 3 annotators labeled the Spanish dataset, the final values were computed by a majority vote.

As the goal of this study was to provide an understanding of the differences in COVID-19 vaccine hesitancy between the English and Spanish datasets, the differences in the observed categories found in these datasets have been further discussed. The percentage of posts in each variable category is reported in Table 1. Moreover, examples of posts in each category can be found in Multimedia Appendix 2.

Interestingly, statistics suggested a substantially greater number of posts in the English manually labeled dataset than the Spanish dataset that either encouraged (41/185, 22.2% vs 14/189, 7.4%) or discouraged (6/185, 3.2% vs 1/189, 0.5%) vaccination. This indicates that the majority of posts in the Spanish manually annotated ground-truth dataset were neutral in their stance toward vaccination, which is also implied by the larger percentage of informative posts in this dataset. Furthermore, a larger percentage of Spanish posts discussed vaccine side effects and safety compared with English posts, which tended to discuss matters related to the educational system, government, policy, and health system. This potentially indicates that the Spanish dataset included posts that mainly focused on the vaccine and its relevant information, while the English dataset included many posts relevant to the vaccine but also had posts on other aspects connected to its administration. An additional analysis is required to understand how these relevant categories have been discussed over time and what their associations are with vaccination rates in Tarrant County.

Once the ground-truth dataset was created, the next step involved obtaining labels for the entire dataset. Large language models (LLMs) have been widely used in the natural language processing literature for text classification tasks, while ChatGPT (OpenAI) has been used for data annotation [70-73]. Moreover, GPT-4 has been shown to be a great choice for detecting COVID-19 vaccine self-report and vaccine chatter [74], reaching accuracies of approximately 81% and 93%, respectively, without any additional prompting or fine-tuning. Therefore, it was a suitable candidate for annotating the vaccine-related dataset, considering its powerful ability to understand the context of the provided input and generate correct labels. Thus, the first step involved formulating a prompt that provided a detailed definition of each variable and corresponding examples, which would enable the model to accurately classify each post. Prompt formulation required several iterations, and the initial prompts were modified so that GPT-4 (OpenAI Azure endpoint was used) could better understand the task and generate labels that aligned more with the ground-truth dataset. By adopting a chain-of-thought [75] approach, larger prompts were broken down into multiple smaller, focused prompts, enabling the model to process each variable in steps. Hence, the labeling codebook was used as a starting point, and the same human thought process was followed to create the prompts that could accurately label the data. In more detail, each prompt included the definition of the variable, an example post belonging to the positive class, and a task to classify the post as related to that variable or not, without explanation. In some cases, the prompt had to explicitly tell the model which posts should not fall under the positive class. For example, the following prompt asked GPT-4 to classify a post as policy/mandate-related or not (the post included a definition of the variable, examples of posts that should be labeled as belonging to the positive class, and exceptions, ie, posts that were previously labeled as belonging to the positive class but should belong to the negative class):

This example illustrates the need to provide a very specific prompt to the model to obtain annotations that closely match human labels and do not alternate over multiple iterations. Thus, it was necessary to present the model with a simple task (ie, to provide binary labels) and explain the reasoning aspects that should lead the model to the correct class inference. This sequential breakdown improved accuracy (from 81% to 90% in the case of the policy/mandate variable), as the model could more effectively reason through each part of the task to produce consistent labels. The list of final prompts used to obtain labels for each post in the ground-truth dataset can be found in Multimedia Appendix 3. It is important to acknowledge that this annotation task was not simple even for human annotators, and it required multiple discussions and changes in the codebook. This shows the importance of providing very detailed and precise guidelines to both human annotators and LLMs to obtain highly accurate labels.

Once again, GPT-4 was asked to classify posts as “related” or “not related” to the COVID-19 vaccine as an additional filtering step to remove any noise remaining after the initial keyword filtering. In addition to the post text, the titles of the websites shared in the posts were passed to the prompt along with article/site descriptions to provide additional context that annotators could see during the labeling process. This approach was a more accurate method to detect vaccine-related posts, as in some instances, the post text did not directly mention the COVID-19 vaccine but rather commented on an article about the vaccine. Therefore, such posts were also labeled by annotators as vaccine-related. By providing these inputs, GPT-4 could detect such posts with higher accuracy. The accuracies for detecting vaccine-related and non–vaccine-related posts in the English and Spanish datasets were 99% and 96%, respectively. Before calculating the accuracies for other variables, detected non–vaccine-related posts were removed from the English and Spanish ground-truth datasets, and they remained with 185 and 189 posts, respectively. In addition, some posts that were not accurately detected by the classifier as non–vaccine-related (1 post in the English dataset and 3 posts in the Spanish dataset) were also removed when computing the final model accuracy for the rest of the variables, as such posts were not annotated in the manual labeling process. Moreover, the process of obtaining GPT-4 labels for all variables was repeated thrice to ensure that the labels were consistent, as the “hallucination” of LLMs (generating plausible yet nonfactual content) is a known problem in the literature [76]. The majority vote of 3 labels was used to compute the final labels, as the goal was to check how consistent the output was across multiple iterations. The highest percentages of posts in the English dataset that received different labels in these iterations were found to belong to the vaccine availability (16/200, 8.0%) and government (13/200, 6.5%) categories, while for all the other labels, the value did not exceed 6%, with the lowest being around 1% for COVID-19 illness experience, side effects, and religion. On the other hand, for all the variables in the Spanish dataset, the value did not exceed 6%. When obtaining the labels for the entire datasets, the labels were obtained only once, as a limited number of posts received different labels in multiple iterations.

After the final labels were computed, they were compared with the manually labeled sample. As demonstrated in Table 2, GPT-4 performed very well in labeling the data with the discussed variables. In the English dataset, the highest accuracy of 99% was obtained for detecting vaccine side effect posts, while the lowest accuracy of 89% was obtained for detecting health system–related posts. However, the classification accuracies in the Spanish dataset were slightly different. In some instances, GPT-4 performed better in the Spanish dataset than in the English dataset (eg, education and statistics), but it did not perform as well in detecting vaccine availability. A possible reason for these results is that the Spanish dataset did not contain as many categories as the English dataset. In some cases, this made the categories harder to detect, while in certain instances, the model could easily recognize the absence of categories, yielding a higher accuracy. It is important to acknowledge that most state-of-the-art classifiers for different text classification tasks have certain limitations; hence, it is not surprising that this classifier performs differently in detecting different categories. As the study’s goal did not involve enhancing the state-of-the-art model in detecting the mentioned variables, the classifier’s performance will need to be improved in future work.

After labels were obtained for the entire dataset, a small number of posts did not have the expected output, and the model could not infer the corresponding label. This issue mainly occurred for very short posts or posts only containing URLs. These posts were discarded from the datasets (62 posts from the English dataset and 23 from the Spanish dataset).

In generating labels for the entire dataset, the English dataset involved 35.2 million input tokens (prompts and posts) and 307,000 output tokens, while the Spanish dataset involved 2.3 million input tokens (prompts and posts) and 37,000 output tokens. The estimated cost of obtaining the annotations for the entire English dataset was US $1074.8, while the cost for the Spanish dataset was US $71.5. The rate limit allowed sending 10 requests per minute; thus, 14,400 prompts could be sent a day. Considering that 11 different prompts were used in this study (with approximately 15,000 posts), approximately 12 days were required to obtain all the annotations. Considering that the posts were collected over a 2-year period, the cost could be considered acceptable in emergency scenarios where a better understanding of reluctance toward interventions could potentially prevent a public health crisis. Moreover, GPT-4 was the latest model released at the time of this study. However, other open-source models could be explored if enough data are available, and this remains a promising area for future work.

As noted in Table 1, the distribution of posts for each variable was highly imbalanced. In more detail, most of the time, posts belonging to a negative class for each variable were prevalent in the dataset, which could have led to the misleading accuracies in Table 2. Thus, after completion of labeling of the entire dataset using GPT-4, a random sample of 25 posts belonging to each positive class and 25 posts belonging to each negative class was extracted to verify the accuracy of the annotations. A single annotator labeled 50 posts for each variable (where each class of a multicategorical variable was treated as a binary variable) and compared the labels with the annotations from GPT-4. The results suggested that for most of the variables, GPT-4 annotated 40 or more posts with the accurate label when, in certain cases, GPT-4 labeled all 50 posts correctly (COVID-19 illness experience and religion in the Spanish dataset). In the Spanish dataset, GPT-4 had low accuracy in detecting misinformation (0.70), postvaccination advice (0.60), and discouraging, informative, and community-specific advice (0.76). In the English dataset, the model had high accuracy in detecting religion (0.98), debunking (0.96), statistics (0.96), and vaccine safety (0.94), and had low accuracy in detecting postvaccination advice (0.62), misinformation (0.74), encouraging advice (0.78), and community-specific advice (0.80). For the remaining variables, the model had accuracies higher than 0.80. Therefore, future work should aim to improve the performance of the classifier for these specific annotations. While the use of GPT-4 for annotating the dataset is highly innovative, the lower accuracies observed for some categories suggest room for improvement, which can be addressed in future work. The performances reported in this study are similar to those reported in related work [74], where for detecting COVID-19 vaccine chatter and self-reported vaccination using GPT-4, the accuracies were nearly 93% and 81%, respectively.

Furthermore, another recent study reported that GPT-4 had an average accuracy of 79.2% in various manually labeled datasets [70]. Hence, it is evident that the model performs better for certain variables than others, where fine-tuning might be required. Further labeling of high-quality datasets and ensuring consistency in the labeling process through prompt tuning might be needed to enhance model performance.

This study aimed to explore how different characteristics of social media posts are associated with vaccination rates in Tarrant County and compare the trends observed in total vaccination rates and Hispanic vaccination rates. In more detail, the analysis focused on the relationship between different social media post features in English and Spanish and vaccination rates at the daily level throughout 2021 and 2022.

For the analysis, English and Spanish posts were merged into a single dataset (containing a total of 13,518 data points), as there might be Hispanic individuals who post in English rather than Spanish. Merging the English and Spanish datasets required careful attention to both linguistic and cultural nuances in vaccine-related discourse. While the same annotation framework and prompts were applied to both datasets using GPT-4, the way topics are framed can differ significantly across languages and communities. For example, the expression of vaccine skepticism in Spanish posts might often include religious or familial references that could be less common in English posts, requiring more context-aware labeling. Additionally, certain categories, such as government trust or perception of the health care system, may carry different connotations based on cultural background, which could influence how posts are interpreted and categorized. To mitigate these challenges, bilingual annotators reviewed posts and labels in both languages to account for this contextual variation. Nevertheless, we acknowledge that achieving complete semantic and cultural equivalence is challenging, and residual bias in cross-language classification remains a limitation. The dataset was then split into posts originating in 2021 and 2022. For each year, 2 linear regression models corresponding to 2 dependent variables were used by clustering SEs per day, as posts shared on the same day might not be independent of each other. Linear regression was selected for this analysis because the outcome variables (ie, daily total vaccinations and daily Hispanic vaccinations) are continuous, allowing us to estimate the linear relationship between social media content features and vaccination uptake. Since each post is assigned the same vaccination count for its corresponding day, multiple posts share the same value for the dependent variable. This creates “within-day clustering,” where residuals of observations from the same day may be correlated, violating the independence assumption of standard linear regression. To address this, we used “clustered SEs at the day level,” which provide robust inference by correcting for intracluster correlation and ensuring more accurate CIs and P values. This approach allows for a valid statistical inference even when the number of posts varies substantially across days.

Furthermore, each data point represented a Facebook post and included the binary categorical variables describing the post. Other variables included in each observation were the date when the post was shared, new Hispanic vaccinations, new total vaccinations on that particular day, and the language of the post. All 4 models included all the independent and corresponding control variables.

This section presents the findings of the 4 linear regression models where SEs were clustered per day. It is important to note that similar models were leveraged, but they used the weekly vaccination data rather than the daily data, and SEs were clustered per week. The results of this analysis closely resemble the findings discussed in this section, showing the robustness of the models. Moreover, the models were tested by using a dependent variable, where new vaccinations per day and week were divided by the population available to vaccinate, and the results remained similar. Table 5 presents the significant results of the 4 models in the analysis (variables with nonsignificant findings have been excluded for readability). Complete data are provided in Multimedia Appendix 4.

Regression analysis yielded multiple relevant findings. As demonstrated for models 1 and 2 in Table 5, posts discussing vaccine safety were associated with a higher daily increase in total vaccination (P=.006) and vaccination of the Hispanic population (P=.002). This might indicate that demonstration of vaccine safety once vaccine administration begins is crucial for ensuring a successful intervention. Furthermore, posts mentioning the government were associated with a lower daily increase in both total (P=.04) and Hispanic vaccinations (P<.001), potentially suggesting mistrust toward the government at that time. In addition, posts providing details about vaccine availability increased the likelihood of a higher number of newly vaccinated individuals (P=.01) and Hispanic individuals (P=.04). Interestingly, posts about religion showed a statistically significant positive relationship with new Hispanic vaccinations (P=.007), indicating that such posts might increase the probability of new Hispanic individuals obtaining the vaccine. It is important to note that when running the models where vaccination increase was calculated per week, all variables that showed statistical significance at the daily level remained significant. However, posts regarding the health system showed a negative statistically significant correlation with weekly new total vaccinations (P=.04) and new Hispanic vaccinations (P=.03), suggesting potential mistrust toward the health system in this period. Additionally, posts shared in Spanish showed a positive statistically significant correlation with weekly new Hispanic vaccinations (P=.02), illustrating that posts shared in Spanish were associated with a higher vaccination increase in the Hispanic population compared with posts shared in English.

Models 3 and 4 (Table 5) from 2022 had different results compared with the models from 2021. For example, posts debunking misinformation showed a statistically significant relationship with both new total vaccinations (P<.001) and new Hispanic vaccinations (P=.04), demonstrating that such posts are associated with a higher number of new vaccinations compared with nondebunking posts. It is important to note that the difference in findings could be due to the size of the dataset in 2022 compared with the dataset in 2021. In summary, the results (Table 5) indicate that the stance of social media posts, posts debunking misinformation, and some other informational categories, such as vaccine availability and safety, play significant roles in influencing the number of newly vaccinated individuals.

This study examined bilingual Facebook discourse about COVID-19 vaccines in Tarrant County and linked it to daily vaccination uptake in the total and Hispanic populations. Three main findings stand out.

First, vaccine-related discourse differed substantially by language and year. English and Spanish posts did not emphasize the same concerns, and the mix of topics shifted from 2021 to 2022. English discouraging posts were dominated by government and policy concerns, whereas Spanish discouraging posts more often emphasized side effects, safety, and the health system. This suggests that vaccine hesitancy was not uniform across language groups and that communication strategies should not assume the same drivers across populations.

Second, some predictors of uptake were common across groups. Posts discussing vaccine availability and vaccine safety were positively associated with vaccination in 2021, and posts debunking misinformation were positively associated with vaccination in 2022. These findings are consistent with the view that practical access information, trust-building content, and correction of false claims are important components of effective vaccine communication.

Third, some predictors were specific to Hispanic vaccination uptake. Encouraging posts and religion-related posts were positively associated with new Hispanic vaccinations in 2021, but not with total vaccinations in the same way. This is particularly important because it suggests that culturally grounded content may matter above and beyond general informational messaging.

The significance of religion-related posts is consistent with prior literature showing that religiosity can shape health behavior [57]. In some communities, faith leaders and faith-based institutions serve as highly trusted intermediaries. In such settings, messages framed around protecting family, stewardship, and community responsibility may resonate more strongly than messages framed only in technical or institutional terms. These findings suggest that partnerships with churches and faith-based organizations may be especially valuable in public health outreach to Hispanic communities.

The findings of this study provide several actionable recommendations for public health officials. Social media interventions targeting vaccine-hesitant populations, particularly in the Hispanic community, should be designed with cultural sensitivity in mind [58]. For example, incorporating bilingual content and collaborating with trusted community figures—such as religious leaders or local health care providers—could enhance the credibility and reach of these interventions. Furthermore, posts that emphasize family protection, community well-being, and the dispelling of myths related to vaccine safety and efficacy have been shown to resonate well with Hispanic audiences [77]. By focusing on culturally appropriate messaging and leveraging trusted networks within these communities, public health efforts can more effectively counter vaccine misinformation and build confidence in the vaccination process.

In summary, the results underscore the importance of bilingual communication. Because language may shape how health information is understood and trusted, simply translating English-language messages may not be sufficient. Effective outreach may require culturally aligned framing, not just linguistic conversion.

This study has several limitations. First, the dataset included only posts from public Facebook pages and groups, which may not represent the views of people using private groups, other platforms, or no social media at all. Second, language is not a perfect proxy for ethnicity, and thus, it cannot be assumed that Spanish-language posts come only from Hispanic users or English-language posts come only from non-Hispanic users. Third, because multiple posts shared the same daily vaccination outcome, the observational structure limits causal interpretation even though clustered SEs were used. Fourth, GPT-4 annotation, while strong overall, was not perfect and may have introduced measurement errors in some categories. Finally, the models did not fully capture broader cultural, structural, or offline influences on vaccine behavior.

This study examined how stance, misinformation, and topic content in English and Spanish Facebook posts were associated with new daily vaccinations in Tarrant County, Texas, during 2021 and 2022. The findings showed that social media discourse differed meaningfully across languages and over time and that some message types were associated with higher vaccination uptake. Across both populations, posts addressing vaccine availability, safety, and misinformation debunking were associated with higher uptake. For the Hispanic population specifically, encouraging posts and religion-related posts were also significant predictors, suggesting culturally specific dynamics.

Taken together, these findings support the use of targeted, culturally responsive, and bilingual public health messaging in future emergencies, and they suggest that social media analysis can help identify which kinds of messages may be most useful for reaching specific communities.