Comparison of Artificial Intelligence Tools With Human Coding for Sentiment, Topic, and Thematic Analysis Tasks of Public Health Datasets During the COVID-19 Pandemic in Australia: Case Study

Introduction

In the context of health, public opinions can change over time, vary across the population, and are often influenced by factors such as personal experiences and media exposure [1]. Public opinion can impact the course of an epidemic through impacting levels of compliance with public health measures (PHMs) such as vaccines, mask-wearing, and social distancing [2]. For example, Yu et al [3] used agent-based modeling to describe the relationship between the spread of COVID-19 and opinion dynamics in 15 different countries and found that public opinion on preventive interventions impacted the cumulative number of cases, particularly in the early stages of an epidemic. The World Health Organization has identified disinformation, the intentional spread of misleading information, as a threat to public health [4]. This may occur by changing the opinions and, therefore, the behaviors of populations through the creation of uncertainty about PHMs [4,5]. Therefore, the collection of public opinion data is important to measure public acceptance of PHMs and to monitor changes over time, thereby serving as a tool to combat the impact of disinformation and promote compliance with public health advice. There has been a proliferation of published research using artificial intelligence (AI) tools to analyze these data [6]. However, there is limited understanding of the accuracy of these readily available AI tools in conducting sentiment, topic, or thematic analysis on datasets of public opinion in real-world scenarios.

Public opinion data provide an important feedback mechanism during health emergencies such as the COVID-19 pandemic [7]. Qualitative research methods, such as analysis of data collected from focus groups and individual interviews, are recommended to better understand community opinions about disease perception and preventive behaviors to inform response efforts during health emergencies [8-10]. However, barriers to rapid qualitative research in the context of an infectious disease outbreak may include difficulties in conducting focus groups and interviews due to exposure risks, and participants may be reluctant to participate in the study due to the impact of the disease or the public health response [8,10]. In emergency situations, it is important to share findings in almost real time; therefore, the time taken to conduct qualitative research is a barrier, particularly during health emergencies [8,10]. Rapid data analysis techniques may deliver some time savings; however, data collection, interpretation, and write-up of results remain time intensive [11], with the World Health Organization European guidelines suggesting that the entire process takes 4 to 6 weeks, and Dong et al [12] classified rapid qualitative methods as those taking less than 6 months from conception to reporting of results [9,12]. Comparatively, data collection and analysis using AI can take as little as a few minutes [13]. Other barriers may include the capacity to form a research team of available staff with the necessary expertise to undertake qualitative analysis of community opinion to inform health policy [8].

Health emergencies such as the COVID-19 pandemic have led to the consideration of new methodologies for the collection and use of evidence to inform policy decision-making [14]. Previous studies have proposed the use of AI tools, such as natural language processing (NLP) and large language models (LLMs), to reduce the workload and provide real-time insights to help inform public health decision-making [15-19]. AI tools can be used to complement traditional qualitative data analysis methods in public health through sentiment analysis, topic modeling, and thematic analysis and have been shown to be less time-consuming and resource-intensive for these tasks [13,20-23]. Access to platforms to perform social media analysis, as well as improved training and capacity to conduct this type of research, was identified as key areas for improvement in a global survey of public health professionals [24]. Our study has specifically chosen tools that are readily available and do not require in-depth training for a public health professional to use.

Despite the proliferation of published research on public opinion data, collected via surveys and social media, toward PHMs during the COVID-19 pandemic, it is unclear whether these data are used as part of evidence-based policy decision-making [25].

This study adopted an applied informatics perspective, focusing on tools and data pipelines readily available to Australian public health analysts. The study aimed to evaluate the feasibility and performance of commonly available AI text analysis tools in analyzing public health datasets from social media (X) and news media (Google Alerts [GA]). Specifically, the study examined how traditional NLP methods and LLM tools perform when analyzing online public opinion data about PHMs during the COVID-19 pandemic in Australia.

The study addressed two research questions:

1.  How accurately and consistently do AI-based tools classify sentiment and extract relevant topic and thematic content from public health–related online text?
2.  What role does human oversight play in ensuring the validity and reliability of AI-driven sentiment and thematic analysis in public health informatics workflows?

First, we hypothesized that AI tools would demonstrate generally limited accuracy for broad sentiment classification and that LLMs would perform better than NLP-based text analysis tools in identifying and summarizing thematic content. Second, we hypothesized that human review and interpretive oversight will remain necessary to ensure data quality and contextual understanding of online public health datasets, underscoring the continuing importance of human oversight in public health informatics workflows.

Methods

Data Sources and Sampling

This study used open-source online data to evaluate the feasibility of automated methods for analyzing public opinion about PHMs during the COVID-19 pandemic in Australia. Two platforms were selected: GA and X (formerly known as Twitter), representing traditional and social media sources, respectively, and the most commonly used sources in infodemiology research [26].

GA is a change detection and notification system that automatically monitors multiple websites for mentions of a textual string and allows the user to select the frequency of monitoring, the source, language, and region [27,28]. GA was configured to retrieve Australian news articles that included key COVID-19–related terms (Table 1) once daily [28]. Each alert provided the publication date, publisher, article title, and 2-line summary, which were collected and stored in a Microsoft Excel spreadsheet. Data collection occurred from December 19, 2022, to February 19, 2023, during the fourth Omicron COVID-19 wave in Australia [29].

The social media platform X is one of the most widely used platforms for health research and was freely available with an academic license at the time of data collection [30,31].

A Python app was used to access the X academic application programming interface (API) and to search for tweets geolocated in Australia using identical keyword combinations and date ranges. Specifically, we used a custom script to handle query construction, data collection, and filtering by keywords, language, and location. Separate searches were conducted for each keyword. All English-language tweets posted in Australia on these days were retrieved and included in the relevance assessment. Tweets with geolocation outside Australia and non–English-language tweets were excluded. Tweet ID and date of publication, tweet content, region, and location were collected and stored in a Microsoft Excel spreadsheet. User information was not collected to maintain anonymity.

A random sample of 800 items (400 GA articles and 400 tweets) was drawn using the RAND() function in Microsoft Excel (version 2306; Microsoft 365) to produce a manageable dataset for manual validation and automated analysis [32]. This dataset size allowed comparison across multiple sentiment and thematic analysis tools while maintaining a feasible human coding time for the manually coded reference set.

Data Cleaning and Preprocessing

Text data from both platforms were preprocessed to standardize the format and remove noise before analysis. Duplicates were removed from the GA dataset by checking for duplicate URLs, thereby retaining articles with similar content published across different days or in different publications (eg, syndicated articles). Within the X dataset, duplicate tweets were removed, while quote tweets and retweets were preserved for analysis. This allowed for analysis of the volume of public interest across news media and social media regarding the topics of interest. Data preprocessing included the removal of Twitter handles, URLs, stop words, and punctuation. Individual GA articles and tweets that discussed more than one PHM (eg, both mask-wearing and vaccination) were disaggregated and analyzed separately for each relevant PHM category.

All collected GA results and tweets were assessed for relevance to PHMs as per the inclusion and exclusion criteria (Textbox 1) by 1 reviewer (DH). A subset of 100 tweets and 100 news articles was assessed by a second reviewer (HS), and interrater reliability was assessed using the Cohen κ coefficient [33].

Human-Coded Sentiment Analysis

Human-coded sentiment analysis involves individuals manually reviewing textual data and assigning a sentiment label to each text based on its content [34]. A data subset comprising 400 GA and 400 tweets was randomly selected using the RAND() function in Microsoft Excel and reviewed by 2 reviewers (DH and LL). As only the first 2 sentences of the news article were included in the GA dataset, the search term was also provided to assist the reviewer to determine sentiment. Sentiment toward the PHM was assessed and assigned a positive, negative, or neutral label based on the interpretation of the text. If the sentiment toward the PHM was positive, but the sentiment of the whole tweet was negative, it was assigned a positive value.

The reviewers used a rule-based process with the development of annotation guidelines (Textbox 2) and met after the first 50 items to review the guidelines and discuss any clarifications needed. Reviewers met again at the completion of the analysis to address any ambiguous cases. Interrater reliability was calculated using the Cohen κ coefficient [33]. Disagreements between human raters were resolved through discussion and consensus to ensure consistency in the final labeled dataset used for AI comparison. One reviewer (DH) completed the sentiment annotation of the remaining relevant GA and X datasets. GA results and tweets that did not contain sufficient information to assign sentiment were removed from the datasets. GA results and tweets that discussed more than one PHM were analyzed separately for each relevant PHM category, allowing for assessment of sentiment associated with specific PHMs within the same source text. GA results and tweets labeled positive, negative, and neutral were aggregated for analysis. Search terms relating to vaccination and mask-wearing (Table 1) were combined for further analysis; for example, the results for “vax,” “jab,” and “vaccine” were collapsed under the umbrella term “vaccine.” The proportion of sentiment, expressed as a percentage of the total number of GA results and tweets, was calculated with 95% CIs.

AI Tools Evaluated for Sentiment Analysis Task

A total of 5 AI-based text analysis tools were evaluated to compare the performance of traditional NLP techniques and LLM-based methods, which were validated against human-coded reference data. Tools were selected based on their accessibility for public sector analysts and compatibility with existing Australian government data science infrastructure [35-38]. R (version 4.3.1; R Foundation for Statistical Computing) within RStudio (version 2023.06.1; Posit Software), sentiment analysis tools (Valence Aware Dictionary and Sentiment Reasoner [VADER], SentimentGI, and SentimentQDAP), Microsoft Azure Machine Learning, and ChatGPT-4 represent platforms that are free, low cost, or integrated within standard analytics environments used by public health agencies [39-43]. Each text entry was assigned a compound sentiment score ranging from −1 (most negative) to +1 (most positive), which was then categorized as positive, neutral, or negative using default thresholds (≥0.05 positive; ≤−0.05 negative).

NLP Tools in R

R is used extensively in public health research and includes packages specifically designed for cleaning and visualizing large public health datasets [35].

VADER is a lexicon- and rule-based sentiment analysis tool optimized for social media text. It was implemented in R using the vader package [44].

The SentimentAnalysis package (SentimentGI and SentimentQDAP) in R extends lexicon-based sentiment scoring by incorporating valence shifters (eg, negators and amplifiers) that adjust for linguistic nuance, which makes it particularly suited to conversational or social media–style text [45].

Microsoft Azure

Microsoft Azure Machine Learning Text Analytics (version 3.1; 2023) was used to assess sentiment through integration into the Microsoft Excel spreadsheet environment via the Power Query function, with results returned as sentiment probabilities for positive, neutral, and negative categories [46]. The Microsoft suite is widely used across Australian government agencies and was therefore included in this study [38]. API version and date of access were documented to ensure reproducibility.

OpenAI ChatGPT (GPT-4)

ChatGPT was tested as an emerging LLM approach for qualitative text classification. Using OpenAI’s API (March 2024) [47], each text item was submitted with a structured zero-shot prompt (Table S1 in Multimedia Appendix 1). Responses were parsed programmatically for sentiment and theme. This approach reflects the real-world use of generative AI in rapid public health analysis while maintaining reproducibility through prompt documentation.

Topic and Thematic Analysis

To demonstrate how these data could be used in the public health domain for decision-making and evaluation, topic summary and thematic analyses were performed. Topic summary analysis was performed on the entire dataset by identifying key phrases and trends manually by highlighting recurring words, phrases, or patterns that may represent shared topics within the datasets [48]. For comparison, topic modeling was conducted on the GA and X datasets using latent Dirichlet allocation (LDA), a probabilistic modeling technique used to identify topics occurring in a textual dataset [22]. This was done using the “topicmodels” package in R [49], and the code is included in Multimedia Appendix 2. ChatGPT-4 was used to replicate the manual topic selection using a zero-shot prompting technique outlined in Table S1 in Multimedia Appendix 1 [50]. Results were collated, and the top 5 topics from each method were compared.

For the thematic analysis, the positive and negative results in the GA and X datasets were extracted from the entire annotated dataset and put into 4 separate files. Neutral sentiment results were excluded. A qualitative thematic analysis explored the opinions expressed toward PHMs within the Australian news media and among X users. This was done using Braun and Clarke’s [51] 6-step process, coding each data line prior to identifying common themes across the dataset. Examples from the datasets were collated, with paraphrasing of tweets to maintain user anonymity. For comparison, ChatGPT-4 was used to replicate the manual thematic analysis [52] on the 4 datasets (GA positive, GA negative, X positive, and X negative results) using the zero-shot prompting technique outlined in Table S1 in Multimedia Appendix 1 [50]. The thematic analysis results were presented with subthemes and paraphrased representative sentences for each dataset.

Statistical Analysis

Concordance of each AI tool with human sentiment analysis was calculated by comparing the rating of a subset of tweets and articles with the human-rated sentiment analysis. Interrater reliability was calculated using the Cohen κ coefficient [33]. The κ statistic was used to assess interrater agreement, and results were classified as 0 to 0.5 weak, 0.51 to 0.8 moderate, and 0.81 and above strong [33]. Statistical analysis was performed using R, where a 2-sided P value of <.05 was considered statistically significant [49].

Validation and Comparison

All tool outputs were compared against the manually coded dataset. Descriptive and statistical analyses were performed on the results of the sentiment analysis process using Microsoft Excel [32]. For each sentiment (eg, “positive”), we identified all cases where the human rater assigned that sentiment and then calculated the percentage of those cases in which the AI tool assigned the same sentiment (Table S2 in Multimedia Appendix 1). This process was repeated for each sentiment category (positive, neutral, and negative) and each tool.

All statistical analyses were performed using R [53].

The accuracy of the generative AI results for topic modeling was calculated using the number of agreements in each dataset [52]. This was done using a cross-matching rubric (Table 2).

A comparative analysis of the human and LLM thematic analyses was conducted by 1 author (DH), who reviewed each output of the LLM and assessed whether the theme matched the human thematic analysis. A cross-matching rubric, similar to the one used for topic analysis agreement, was developed to classify the LLM-generated output as “proficient,” ”partially proficient,” or ”not proficient” in capturing themes from the data that matched the human coding (Table 2) [50,52]. For all LLM outputs, regardless of whether they matched the human output, a score for how “reasonable” it was to derive the theme from the dataset was given, using a scale of 0 to 2, as has been used in similar studies [52]. The scoring was completed by 1 author, a subject matter expert (DH; Table 2).

Ethical Considerations

Nonidentifiable data from online news media and social media were collected during this study. We did not analyze individual accounts and have not published any identifiable information or individual quotes. The LLM used in this study, ChatGPT-4, was used in a strictly limited capacity for analyzing nonsensitive, deidentified text. The research team ensured that no identifiable data were shared with the AI platform, and all use complied with institutional research integrity and data privacy guidelines. All data were deidentified (including the removal of Twitter handles) prior to input, and chat history was disabled to prevent storage or reuse of content by the model provider. Chats were deleted when the session was complete. The study was approved by the UNSW Human Research Ethics Committee (approval number HC230028).

Results

Overview

The results of this study are presented as follows. First, a description of the dataset is given. Second, the results of the comparison of human-coded and machine-coded sentiment analysis on the data subset are outlined. Third, a summary of the sentiment analysis of the entire dataset is provided. Finally, human-generated and machine-generated topic modeling and thematic analysis results of the entire dataset are compared to demonstrate the feasibility of using AI tools to assist public health analysts in assessing public opinion during health emergencies.

Description of the Dataset

Overall, 2761 GA articles and 3495 tweets were collected during the study period. Following removal of duplicates, 2227 GA articles and 3484 tweets were included in the relevance screening, with 57.6% (n=1283) of GA articles and 71% (n=2473) of tweets assessed as relevant. There were moderate (GA: 88% concordance, ĸ=0.76) and strong (X: 94% concordance, ĸ=0.86) levels of agreement for relevance between 2 analysts in the data subsets of 100 (88%) GA articles and 100 (100%) tweets.

During the sentiment analysis process, further GA articles (n=154) and tweets (n=120) were removed from the dataset, as there was not enough information to determine sentiment (Figure 1). There were moderate levels of agreement for sentiment scores between the 2 reviewers in both the GA dataset (κ=0.69; P<.001) and the X dataset (κ=0.75; P<.001; Table S2 in Multimedia Appendix 1).

‎

Sentiment Classification Performance of AI Tools

The results of sentiment classification performance are presented as proportion (%) of agreement with the human rater for each tool and sentiment category (Figures 2 and 3). The performance of the AI tools was generally low and varied across sentiment categories and data sources. Agreement was highest for positive sentiment and lowest for neutral sentiment across both platforms. In the GA dataset, SentimentQDAP had the highest agreement with the human rater for negative sentiment (n=28, 62.2%), while SentimentGI had the highest agreement for positive sentiment (n=36, 62.1%), and VADER performed best for neutral sentiment in this dataset, with an agreement rate of 42.9% (n=120). In the X dataset, VADER achieved the highest agreement for negative sentiment (n=78, 54.9%), while ChatGPT-4 demonstrated the highest agreement for both neutral (n=48, 38.2%) and positive (n=92, 55.1%) sentiments. An interrater reliability analysis was performed between the dependent samples of reviewer 1 and each AI tool. For this purpose, Cohen κ was calculated, and in all cases, no agreement was found (Table S3 in Multimedia Appendix 1).

‎

‎

Manual Sentiment Analysis of Entire Datasets

Total Sentiments Across the Datasets

Positive, negative, and neutral sentiment results were collated (Table 3), with differences in the distribution of sentiment observed between the 2 platforms. Although the relevant datasets comprised 1283 (57.6%) GA articles and 2473 (%) tweets, a higher number of text segments were analyzed due to articles and tweets that referred to multiple PHMs. These were coded and analyzed separately for each PHM to measure sentiment patterns. In the analysis of the GA platform (n=1587), the majority of the sentiment was neutral (n=1021, 64.3%, 95% CI 62%‐66.7%). Negative sentiments accounted for 18.1% (n=287, 95% CI 16.2%‐20%), while positive sentiments accounted for 17.6% (n=279, 95% CI 15.7%‐19.5%). In the analysis of text segments on the X platform (n=3124), sentiment demonstrated a higher degree of polarity, with negative sentiments comprising 40% (n=1248; 95% CI 38.2%‐41.7%) and positive sentiments comprising 39.5% (n=1233; 95% CI 37.8%‐41.2%) of the total. In the X dataset, 20.6% (n=643; 95% CI 19.2%‐22%) of sentiments were neutral. These results indicate fewer opinions and more neutral reporting of events in the GA dataset, while there was a higher degree of both positive and negative emotional expression found within the X dataset.

Sentiment Analysis of Each Search Term

Sentiment analysis results were collated for each search term in each dataset (Table S4 in Multimedia Appendix 1) to investigate sentiment expressed toward specific topics and specific PHM. As described earlier, the various PHMs were discussed in different tones across the 2 platforms, with GA results more likely to be classified as neutral sentiment, and X results more likely to show polarity for each search term. Neutral GA sentiments were expected due to the reporting of events in the news media. For search terms relating to pharmaceutical companies in the X dataset, reports on the search term “Moderna” (n=30), sentiment was more evenly distributed (negative: n=11, 36.7%; neutral: n=10, 33.3%; and positive: n=9, 30%) when compared to sentiment for the search term “Pfizer” (n=222), which was mostly negative (n=135, 60.8%), 30.6% neutral (n=68), and 8.6% positive (n=19). When comparing the search term “booster” (n=187) in X dataset, sentiment was broken down as 40.1% (n=75) negative, 16% (n=30) neutral, and 43.9% (n=82) positive; however, sentiment for the search term “vaccine” (n=860) was mostly negative (n=378, 44%), 28.4% neutral (n=244), and 27.7% positive (n=238). Search terms relating to lockdown (n=23, 54.8%) and mandates (n=85, 66.4%) were associated with negative sentiment in the X dataset, while isolation was associated with positive sentiment (n=17, 85%).

Sentiment Analysis of Vaccination and Mask Search Terms

Results for search terms related to COVID-19 vaccination and mask-wearing were combined (Table S5 in Multimedia Appendix 1). Vaccine results were analyzed with and without the inclusion of results of the search term “booster,” as it was shown to have opposing polarity when compared to “vaccine” (Table S5 in Multimedia Appendix 1). Analysis of the combined vaccine search terms continues to show the same pattern of distribution between the 2 datasets, with sentiment in the GA dataset predominantly neutral (65.8%) and lower proportions of negative (18.1%) and positive (16.1%) sentiment. In contrast, the X dataset was more polarized and critical of vaccines, with nearly half (49.5%) of all results expressing negative sentiment and lower levels of neutral (26.9%) and positive (23.6%) sentiment (Table S5 in Multimedia Appendix 1). When booster-related results were included in the analysis, these patterns were retained.

The pattern was markedly different for sentiment analysis results of reports collected via mask-related terms. The GA results were also mostly neutral (61.4%), and 26.4% expressing positive sentiment and only 12.3% expressing negative sentiment. In the X dataset, there was a much higher proportion of positive sentiment (66.8%) than negative (21.5%) or neutral (11.7%) sentiments expressed.

Topic Summary Analysis

LDA topic modeling of the GA and X datasets (Tables S6 and S7 in Multimedia Appendix 1) and the LLM results were compared to the human rating of the top 5 most discussed topics and are presented in Tables 4 and 5, including alignment scores from the cross-matching rubric (Table 2).

GA Dataset

Mask use and mask mandates, as well as vaccine mandates, are discussed in the GA dataset within the context of easing COVID-19 restrictions (Table 4). Across the 5 cases, the RStudio LDA output topics fully aligned with 1 of 5 cases and partially aligned with the manual coding in 2 of 5 instances, while the LLM output fully matched in 2 of 5 instances and partially matched in 3 of 5 cases. The LLM outputs show a deeper understanding of the context but are often too generalized, while some of the machine-generated outputs were related but not specific to the manually generated topics.

X Dataset

Topics discussed in the X dataset included announcements concerning the easing of restrictions, and opinions both supporting and opposing the dropping of mask mandates and prolonged isolation following a positive COVID-19 test (Table 5). Vaccination campaigns and safety concerns were also discussed. Results from the cross-matching rubric of the X dataset showed that the LLM output more closely aligned with the manually generated topic, fully matching in 3 of 5 cases and partially matching in 1 of 5 cases. The LDA output fully matched in 2 of 5 instances and partially matched in 2 of 5. The LLM output is more descriptive of the topics.

Human-Coded Thematic Analysis

Thematic analysis was performed on the positive- and negative-assigned GA articles and tweets (Tables S8a-S8d in Multimedia Appendix 1). Many tweets were of the account owners reporting their choice to engage or not engage with particular PHMs (eg, “My wife and I wear masks when we go out unless we are eating or drinking or outside. There is no way I am having that experimental vaccine”). Several other themes emerged and are summarized in Table 6.

In the positive GA dataset, themes included masks being recommended in particular contexts (eg, “...masks are recommended in health care settings, on public transport, in crowded indoor settings”) and encouraging vaccination in at-risk groups (eg, “Aside from the government and GPs, family members have an important role in encouraging senior citizens to get vaccines or booster shots against COVID-19”).

In the negative GA dataset, themes included reports of the COVID-19 vaccines causing injuries (eg, “Dr [name] has broken her silence about a ‘devastating’ COVID vaccine injury, slamming regulators for ‘censoring’ public discussion, and Thousands of Australians suffering from COVID-19 vaccine injury feel they are ‘not being heard’ or treated fairly by the government”). Occupational vaccine mandates were also discussed with negative sentiment expressed (eg, “Over 200 firefighters in New South Wales and Victoria are being forced to ‘stay away from saving lives’ because of ongoing vaccine mandates, and Coles is the only major supermarket in Australia that continues to use discriminatory COVID vaccination mandates for workers”).

In the positive X dataset, these included masks being protective for the wearer, with reasons of vulnerability or framing it as “smart” behavior (eg, “My daughter works with COVID patients and we wear a mask when she visits. My partner has cancer, and Lots of people coughing on the train and I’m the only one smart enough to wear a mask”). People also discussed the wearing of masks to protect others in the community (eg, “It is selfish not to wear a mask to protect the health of our most vulnerable”). There were many posts expressing the opinion that the benefits of vaccines outweigh the risks (eg, “I’m sorry for people injured by the vaccine but the risk is insignificant compared to complications from Covid”). When mentioning boosters, the sentiment tended to be favorable, wanting access to updated boosters (eg, “I saw on the news that we might get 5th jab in February – can’t come soon enough!”).

In the negative X dataset, a strong theme emerged about COVID-19 vaccines causing injuries and deaths. Many tweets included statistics regarding vaccine deaths and anecdotal reports of people they know or had heard of dying suddenly after being vaccinated, or reporting lived experience of having a vaccine injury (eg, “Vaccine injury and deaths outnumber actual deaths FROM Covid,” and “My neighbour told me that her son’s friend went down to the local shopping center to get vax, dies 15 minutes later”). There was also a theme that COVID-19 vaccines are experimental, and people who take the vaccine are “brainwashed” and “sheep,” positioning those who had not had the vaccine as “smart” (eg, “Scientists came up with this vaccine in 10 weeks, and people still believe it’s safe, talk about being brainwashed, and The sheep are rolling up their sleeves for their 5th jab, and I don’t inject poison into my body, unlike the vax junkies”).

Thematic Analysis Results From LLM

The LLM-generated summaries of the thematic analysis of the positive- and negative-assigned GA and X datasets are presented in Tables 7-10. Results of LLM thematic analysis, including theme descriptions and the proficiency and reasonableness scores benchmarked against human-coded thematic analysis, are presented in Table 2. When compared to the human-coded thematic analysis, the results for the LLM proficiency were 13 of 20 proficient and 7 of 20 partially proficient, showing that the LLM produced themes that were relevant to the dataset and closely matched with the human-coded themes. All themes, whether fully or partially proficient at matching human-generated themes, were rated “very reasonable” (the themes had high relevance to the dataset and would likely be generated by human coding), suggesting that the results may be useful to support human coding of large datasets.

Discussion

Principal Findings

Our study compared the results of AI analysis of PHM-related datasets with human-coded analysis for common tasks used in the context of health emergencies, such as sentiment analysis (to explore public opinion of PHMs), topic modeling (to identify what is being discussed in online news and social media), and thematic analysis (for a more in-depth analysis of how PHMs are being discussed in the public domain). AI tools were deliberately selected to reflect those accessible to public health professionals in Australia that require minimal technical expertise.

Overall, AI tools performed inconsistently across tasks. All models showed poor performance for sentiment analysis. ChatGPT-4 was found to demonstrate stronger alignment with human raters for both the topic modeling and thematic analysis tasks. These findings highlight both the potential and the limitations of AI tools to complement traditional methods of analysis for public health professionals by providing rapid insights while still requiring human interpretation and oversight [26].

Sentiment Analysis Task

To evaluate the accuracy of accessible AI sentiment analysis tools, a subset of Australian English-language GA and X datasets was analyzed by 2 human raters and 5 AI tools. Sentiment was poorly detected by all AI tools, with no agreement between the human rater and either the GA or X datasets (Table 3). In particular, accuracy was less than 30% for neutral sentiment in the GA dataset and under 50% for positive or negative sentiment in the X dataset. These findings align with previous studies showing that off-the-shelf sentiment analysis tools perform poorly when applied to complex, health-related discourse [54,55].

The human-coded sentiment analysis showed that the GA dataset had a majority of neutral sentiment in discussion about PHMs, while the X dataset was evenly split between positive and negative. These findings highlight differences between the data sources, with the prevalence of neutral sentiment in the GA dataset, suggesting predominantly neutral reporting of current stories and events in online news media. The X dataset showed greater polarity of sentiment, which may indicate that users of X have stronger opinions and are more likely to express them online. Our study demonstrates that in Australia during the study period, mask use and isolation were associated with more positive sentiment, while vaccines, lockdowns, and mandates attracted more negativity. These results indicate that sentiment analysis may give useful high-level insights regarding public opinion for public health decision-makers while highlighting the need for contextual interpretation by human analysts.

Topic Analysis Task

In our study, topic modeling identified overlapping areas of discussion between datasets, with GA focusing on mask use and vaccine mandates in the context of easing restrictions and X focusing on vaccination campaigns, mask mandates, and isolation requirements. LDA partially aligned with human-coded topics, while the LLM output provided fuller contextualization but was overly generalized. Recent advances in generative AI have shown improved topic matching with human annotators of health-related datasets using LLMs [52]. This may assist in public health responses during health emergencies by improving the understanding of topics that are being discussed in the news and on social media and addressing misunderstandings or concerns with public health messaging [56].

Thematic Analysis Task

Qualitative research methods are promoted as the most suitable approach to gain an understanding of the experiences of individuals during health emergencies, which can be used to inform local public health policy decisions and implementation [10]. Manual interpretation of data, including thematic analyses, presents an extensive time and resource burden [52]. Generative LLMs can analyze and interpret vast amounts of text and have shown good accuracy in generating themes when compared to human analysts, with adequate depth of explanations of themes and inclusion of appropriate quotations, with time savings of several hours or even days [13,52,57]. Some studies have suggested that thematic analysis results generated by LLMs may be best used in collaboration with human coders with domain-specific knowledge [14,58,59].

The human-coded thematic analysis of the GA dataset (Table S8a-d in Multimedia Appendix 1) revealed the ways in which PHMs were being discussed in the news media, including strong support for vaccination of at-risk groups and the use of masks in specific high-transmission contexts, while also reporting on vaccine injuries. Themes that emerged from the X dataset revealed polarized views on the uptake of PHM, with the positioning of compliance with vaccines and mask-wearing as “smart” by supporters of those PHM. From the opposing view, there was much discussion about vaccine harms, including injuries and deaths, and the positioning of compliance with PHMs as a result of “brainwashing.” There were also reports of sudden death from the “experimental vaccine,” which is useful for public health professionals to be aware of when planning campaigns to combat misinformation [60].

In our study, the LLM-generated themes were relevant to the dataset, and the majority were closely matched with the human-coded themes. Even when the themes were only partially matched, they rated high on how reasonable it was for the LLM to generate the theme from the dataset, suggesting that thematic analysis may provide valuable input to inform public health decision-making in a timely way. The summaries generated by the LLM provided an overview of the AI-generated thematic analysis; however, they lacked the specific insights and understanding of the social context of the human-generated summaries.

Error and Bias Analysis

Errors in sentiment classification and theme generation were consistent across tools and datasets in this study, reflecting known limitations of NLP and LLM models when applied to public health discourse [55,61,62]. Misclassification of neutral sentiment was frequent, which may result from narrow sentiment thresholds and training data that are not designed for health-related news and social media [54]. Inability to detect sarcasm is another known limitation of AI tools for sentiment and thematic analysis tasks, when the textual data include positive words to express negative sentiment, highlighting another area where human oversight for contextual nuance is important for accurate analysis [63,64]. While LLMs have demonstrated superior accuracy over other AI tools for tasks such as sentiment analysis, topic modeling, and thematic analysis of public health datasets, they do not match human raters for interpretation and depth of analysis [13,52,55,65]. Fine-tuned or domain-specific LLMs trained on health-related text may improve accuracy, but these models are underrepresented, and substantial annotation and validation efforts will be required to develop appropriately trained models for public health contexts [63-65].

LLM reproducibility presents another challenge, as outputs may vary over time as model parameters and training data are updated by developers. This is known as “model drift” and may complicate longitudinal comparisons and replication of results [66]. Researchers should record model versions and exact prompt wording (as in Table S1 in Multimedia Appendix 1) to enhance transparency and allow future verification.

Manual analysis of online public opinion data may be open to bias through the interpretation of the researcher [67]; however, automated methods lack the capacity to clarify the results of analysis, as may be possible with more traditional methods [66]. While social media can give voice to more marginalized groups, health inequities can be amplified if unrepresentative data are used for analysis [68]. Social bots, which are computer algorithms designed to mimic human interactions on social media, can be used to manipulate public opinion and therefore skew sentiment data [69].

Limitations

There were several limitations of our study. First, regarding the data sources used, GA retrieved the first 2 sentences of the article, which may not give an accurate representation of the sentiment of the article. While the anonymity of data from online social networking sites may have benefits over qualitative or survey data by reducing the impact of social desirability bias [70,71], the awareness of the post being observed by others may make the user less likely to publish unpopular opinions [72]. The anonymity of social media data also impacts the ability to collect demographic information, which can impact how generalizable the results of the analysis [73]. The use of social media data for research is becoming increasingly challenging due to restrictions on access by commercial owners of the platform [74]. The use of untrained NLP sentiment analysis tools, which were unable to correctly identify both neutral sentiment and sarcasm, was a further limitation of this study. While AI can offer a timely way to provide real-time data, public health professionals may be skeptical of the results without understanding the process and how to interpret the output; therefore, education on its effective use will be necessary for future implementation [75]. Inequities and bias that are present in the training data may be replicated in AI outputs [75]; for example, LLMs trained primarily on content originating from North America and the United Kingdom may misrepresent Australian cultural and linguistic nuances, potentially skewing thematic outputs in subtle ways [76]. As this study was conducted on an English-language dataset, the results of this analysis do not capture sentiment and thematic perspectives from culturally and linguistically diverse communities [1]. This limitation is particularly important in multicultural settings, such as Australia, where attitudes toward PHMs may differ across language groups [1]. For AI techniques to inform public health policies in Australia, they must support an understanding of the actual sentiment in diverse communities. While automatic translation and multilingual models can be used for analysis of text in multiple languages, there is a dearth of available data for analysis in languages other than English in the Australian context.

Conclusions

During health emergencies, there is a need to balance rapid analysis of data with accuracy to support public health decision-making. This study examined the accuracy of 5 AI tools in performing tasks designed to measure public opinion in Australia toward PHMs, such as vaccines, mask mandates, and lockdowns. AI tools were chosen that are widely available across government agencies in Australia. All AI tools were found to perform poorly in a sentiment analysis task of the GA and X datasets when compared to a human rater. AI-generated topic modeling and thematic analysis were conducted using the LLM ChatGPT-4 and compared to human-generated responses. The LLM topic modeling outputs showed a high level of alignment with the human-generated topics, and while understanding of the context of the results was indicated, it was often overly generalized. The LLM output of the thematic analysis task was found to be highly relevant and well matched to the human-generated analysis. Even when the themes were only partially matched to the human-generated themes, the results were classified as reasonable, relevant to the data, and likely to be generated by a human analyst. These findings suggest that AI tools, particularly LLMs, may serve as a rapid triage tool to surface emergent themes from large-scale public datasets, which could then be reviewed or refined by human analysts in time-sensitive policy settings. It is unlikely that AI tools will replace traditional research methods used to investigate attitudes and opinions to epidemic PHMs in Australia; however, there remains an opportunity to use this technology to complement qualitative research techniques used by public health professionals in a cost-effective and timely way in the context of health emergencies.