Annice Kim
Research Subjects
Research Interests
Annice Kim has a total of 91 co-authors. Here are the top co-authors by number of shared publications:
- James Nonnemaker
RTI International
11 shared publications - Robert Chew
Social, Statistical, and Environmental Sciences, RTI International, Research Triangle Park, North Carolina (S.K., K.C., M.V., G.B., R.H., R.C., M.P., L.C.K.)
9 shared publications - Jamie Guillory
RTI International, Research Triangle Park, North Carolina
7 shared publications - Lauren McCarl Dutra
Policy and Regulatory Science Program, RTI International, Research Triangle Park, North Carolina, USA
6 shared publications - Nathaniel Taylor
KTH Royal Institute of Technology, Stockholm, Sweden
6 shared publications - Brian Bradfield
Social Sciences, Statistics, and Environmental Sciences, RTI International, 3040 East Cornwallis Ave, Research Triangle Park, NC, 27709, USA
5 shared publications
Latest Publications
- Top Cited PapersOpen Access
LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding
Deductive coding is a widely used qualitative research method for determining the prevalence of themes across documents. While useful, deductive coding is often burdensome and time consuming since it requires researchers to read, interpret, and reliably categorize a large body of unstructured text documents. Large language models (LLMs), like ChatGPT, are a class of quickly evolving AI tools that can perform a range of natural language processing and reasoning tasks. In this study, we explore the use of LLMs to reduce the time it takes for deductive coding while retaining the flexibility of a traditional content analysis. We outline the proposed approach, called LLM-assisted content analysis (LACA), along with an in-depth case study using GPT-3.5 for LACA on a publicly available deductive coding data set. Additionally, we conduct an empirical benchmark using LACA on 4 publicly available data sets to assess the broader question of how well GPT-3.5 performs across a range of deductive coding tasks. Overall, we find that GPT-3.5 can often perform deductive coding at levels of agreement comparable to human coders. Additionally, we demonstrate that LACA can help refine prompts for deductive coding, identify codes for which an LLM is randomly guessing, and help assess when to use LLMs vs. human coders for deductive coding. We conclude with several implications for future practice of deductive coding and related research methods.
Top Cited PapersOpen AccessLLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding
Deductive coding is a widely used qualitative research method for determining the prevalence of themes across documents. While useful, deductive coding is often burdensome and time consuming since it requires researchers to read, interpret, and reliably categorize a large body of unstructured text documents. Large language models (LLMs), like ChatGPT, are a class of quickly evolving AI tools that can perform a range of natural language processing and reasoning tasks. In this study, we explore the use of LLMs to reduce the time it takes for deductive coding while retaining the flexibility of a traditional content analysis. We outline the proposed approach, called LLM-assisted content analysis (LACA), along with an in-depth case study using GPT-3.5 for LACA on a publicly available deductive coding data set. Additionally, we conduct an empirical benchmark using LACA on 4 publicly available data sets to assess the broader question of how well GPT-3.5 performs across a range of deductive coding tasks. Overall, we find that GPT-3.5 can often perform deductive coding at levels of agreement comparable to human coders. Additionally, we demonstrate that LACA can help refine prompts for deductive coding, identify codes for which an LLM is randomly guessing, and help assess when to use LLMs vs. human coders for deductive coding. We conclude with several implications for future practice of deductive coding and related research methods.
- Open Access
- 8 June 2022
- Published by RTI Press
Weed Light: An Exploratory Study of Delta-8 THC Conversations on Reddit
Delta-8 tetrahydrocannabinol (THC) is a newly popular cannabinoid that can be purchased easily at gas stations, vape shops, and online retailers in most regions of the United States. The US Food and Drug Administration (FDA) has expressed the need for novel data sources to understand the use of products like Delta-8 and rapidly detect potential safety issues. Using social media platform Reddit, we examined users’ reported experiences with Delta-8. We collected publicly available Reddit posts of Delta-8 from January 2020 through April 2021 using Brandwatch, a commercially available social media listening platform. We identified the names of popular subreddits, trends in post volume, and frequent topic phrases for Delta-8. Approximately 57,913 Reddit posts were collected from 11,333 unique users across 708 subreddits. Most frequently, discussions were about available products, locations for purchase, overall safety and legality, reactions to the product, and general experiences. Given the lack of regulation and testing, questionable legality, and the product’s wide availability at retail and online locations, Delta-8 products should be a concern for health officials. Monitoring social media platforms like Reddit can help FDA achieve its regulatory and consumer protection goals for emerging cannabis products.
- 8 June 2022
- Published by RTI Press
Open AccessWeed Light: An Exploratory Study of Delta-8 THC Conversations on Reddit
Delta-8 tetrahydrocannabinol (THC) is a newly popular cannabinoid that can be purchased easily at gas stations, vape shops, and online retailers in most regions of the United States. The US Food and Drug Administration (FDA) has expressed the need for novel data sources to understand the use of products like Delta-8 and rapidly detect potential safety issues. Using social media platform Reddit, we examined users’ reported experiences with Delta-8. We collected publicly available Reddit posts of Delta-8 from January 2020 through April 2021 using Brandwatch, a commercially available social media listening platform. We identified the names of popular subreddits, trends in post volume, and frequent topic phrases for Delta-8. Approximately 57,913 Reddit posts were collected from 11,333 unique users across 708 subreddits. Most frequently, discussions were about available products, locations for purchase, overall safety and legality, reactions to the product, and general experiences. Given the lack of regulation and testing, questionable legality, and the product’s wide availability at retail and online locations, Delta-8 products should be a concern for health officials. Monitoring social media platforms like Reddit can help FDA achieve its regulatory and consumer protection goals for emerging cannabis products.
- Open Access
- 18 January 2022
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 24 (1) , e30257
Identifying Electronic Nicotine Delivery System Brands and Flavors on Instagram: Natural Language Processing Analysis
Background: Electronic nicotine delivery system (ENDS) brands, such as JUUL, used social media as a key component of their marketing strategy, which led to massive sales growth from 2015 to 2018. During this time, ENDS use rapidly increased among youths and young adults, with flavored products being particularly popular among these groups. Objective: The aim of our study is to develop a named entity recognition (NER) model to identify potential emerging vaping brands and flavors from Instagram post text. NER is a natural language processing task for identifying specific types of words (entities) in text based on the characteristics of the entity and surrounding words. Methods: NER models were trained on a labeled data set of 2272 Instagram posts coded for ENDS brands and flavors. We compared three types of NER models—conditional random fields, a residual convolutional neural network, and a fine-tuned distilled bidirectional encoder representations from transformers (FTDB) network—to identify brands and flavors in Instagram posts with key model outcomes of precision, recall, and F1 scores. We used data from Nielsen scanner sales and Wikipedia to create benchmark dictionaries to determine whether brands from established ENDS brand and flavor lists were mentioned in the Instagram posts in our sample. To prevent overfitting, we performed 5-fold cross-validation and reported the mean and SD of the model validation metrics across the folds. Results: For brands, the residual convolutional neural network exhibited the highest mean precision (0.797, SD 0.084), and the FTDB exhibited the highest mean recall (0.869, SD 0.103). For flavors, the FTDB exhibited both the highest mean precision (0.860, SD 0.055) and recall (0.801, SD 0.091). All NER models outperformed the benchmark brand and flavor dictionary look-ups on mean precision, recall, and F1. Comparing between the benchmark brand lists, the larger Wikipedia list outperformed the Nielsen list in both precision and recall. Conclusions: Our findings suggest that NER models correctly identified ENDS brands and flavors in Instagram posts at rates competitive with, or better than, others in the published literature. Brands identified during manual annotation showed little overlap with those in Nielsen scanner data, suggesting that NER models may capture emerging brands with limited sales and distribution. NER models address the challenges of manual brand identification and can be used to support future infodemiology and infoveillance studies. Brands identified on social media should be cross-validated with Nielsen and other data sources to differentiate emerging brands that have become established from those with limited sales and distribution.
- 18 January 2022
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 24 (1) , e30257
Open AccessIdentifying Electronic Nicotine Delivery System Brands and Flavors on Instagram: Natural Language Processing Analysis
Background: Electronic nicotine delivery system (ENDS) brands, such as JUUL, used social media as a key component of their marketing strategy, which led to massive sales growth from 2015 to 2018. During this time, ENDS use rapidly increased among youths and young adults, with flavored products being particularly popular among these groups. Objective: The aim of our study is to develop a named entity recognition (NER) model to identify potential emerging vaping brands and flavors from Instagram post text. NER is a natural language processing task for identifying specific types of words (entities) in text based on the characteristics of the entity and surrounding words. Methods: NER models were trained on a labeled data set of 2272 Instagram posts coded for ENDS brands and flavors. We compared three types of NER models—conditional random fields, a residual convolutional neural network, and a fine-tuned distilled bidirectional encoder representations from transformers (FTDB) network—to identify brands and flavors in Instagram posts with key model outcomes of precision, recall, and F1 scores. We used data from Nielsen scanner sales and Wikipedia to create benchmark dictionaries to determine whether brands from established ENDS brand and flavor lists were mentioned in the Instagram posts in our sample. To prevent overfitting, we performed 5-fold cross-validation and reported the mean and SD of the model validation metrics across the folds. Results: For brands, the residual convolutional neural network exhibited the highest mean precision (0.797, SD 0.084), and the FTDB exhibited the highest mean recall (0.869, SD 0.103). For flavors, the FTDB exhibited both the highest mean precision (0.860, SD 0.055) and recall (0.801, SD 0.091). All NER models outperformed the benchmark brand and flavor dictionary look-ups on mean precision, recall, and F1. Comparing between the benchmark brand lists, the larger Wikipedia list outperformed the Nielsen list in both precision and recall. Conclusions: Our findings suggest that NER models correctly identified ENDS brands and flavors in Instagram posts at rates competitive with, or better than, others in the published literature. Brands identified during manual annotation showed little overlap with those in Nielsen scanner data, suggesting that NER models may capture emerging brands with limited sales and distribution. NER models address the challenges of manual brand identification and can be used to support future infodemiology and infoveillance studies. Brands identified on social media should be cross-validated with Nielsen and other data sources to differentiate emerging brands that have become established from those with limited sales and distribution.
- Open Access
- 26 May 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Towards More Meaningful Social Media Analysis: Case Study of Utilizing an Age Prediction Algorithm to Identify and Code Reddit Posts about E-cigarettes by Youth vs. Adults (Preprint)
BACKGROUND Previous qualitative studies and data science studies using Reddit for tobacco research are limited by the lack of available demographic information. Social media investigations are often limited to manual qualitative coding or machine learning classification in isolation. OBJECTIVE This study combines both machine learning methods and manual qualitative coding to provide contextual age nuance to social media analysis. By being able to predict a Redditor’s age using publicly available data, the most popular posts can be analyzed and qualitatively coded to provide nuanced comparisons on thematic topics by age group. METHODS The current study combines these two methods to 1) predict Reddit users’ age into two categories (13-20, 21-54) and 2) qualitatively code Electronic Nicotine Delivery System [ENDS] related Reddit posts within the two age groups. We identified Reddit posts on three topics: Vaping in General, Tobacco 21 Minimum Age Laws, and Flavor Restriction Policies. An age algorithm was used to predict Reddit users’ ages (13-20 or 21-54 year old users). The 25 posts with the highest karma score (number of upvotes minus number of downvotes) for each query and each predicted age group were qualitatively coded. RESULTS The top three, two of which were part of the query, out of nine, topics that emerged were “Flavor Restriction Policies”, “Tobacco 21 Policies”, and “Use”. Tobacco 21 and Flavor Restriction Policy posts were prominent coding categories. Opposition to flavor restriction policies was a prominent sub-category for both groups, but more common in the 21-54 group. The 13-20 group was more likely to discuss opposition to minimum age laws as well as access to flavored ENDS products. The 21-54 group more commonly mentioned general vaping use behavior. CONCLUSIONS Users predicted to be in the 13-20 age group posted about different ENDS-related topics on Reddit than users predicted to be in the 21-54 age group. Future studies could use these complementary methods with social media data to gain insights from target audiences.
- 26 May 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessTowards More Meaningful Social Media Analysis: Case Study of Utilizing an Age Prediction Algorithm to Identify and Code Reddit Posts about E-cigarettes by Youth vs. Adults (Preprint)
BACKGROUND Previous qualitative studies and data science studies using Reddit for tobacco research are limited by the lack of available demographic information. Social media investigations are often limited to manual qualitative coding or machine learning classification in isolation. OBJECTIVE This study combines both machine learning methods and manual qualitative coding to provide contextual age nuance to social media analysis. By being able to predict a Redditor’s age using publicly available data, the most popular posts can be analyzed and qualitatively coded to provide nuanced comparisons on thematic topics by age group. METHODS The current study combines these two methods to 1) predict Reddit users’ age into two categories (13-20, 21-54) and 2) qualitatively code Electronic Nicotine Delivery System [ENDS] related Reddit posts within the two age groups. We identified Reddit posts on three topics: Vaping in General, Tobacco 21 Minimum Age Laws, and Flavor Restriction Policies. An age algorithm was used to predict Reddit users’ ages (13-20 or 21-54 year old users). The 25 posts with the highest karma score (number of upvotes minus number of downvotes) for each query and each predicted age group were qualitatively coded. RESULTS The top three, two of which were part of the query, out of nine, topics that emerged were “Flavor Restriction Policies”, “Tobacco 21 Policies”, and “Use”. Tobacco 21 and Flavor Restriction Policy posts were prominent coding categories. Opposition to flavor restriction policies was a prominent sub-category for both groups, but more common in the 21-54 group. The 13-20 group was more likely to discuss opposition to minimum age laws as well as access to flavored ENDS products. The 21-54 group more commonly mentioned general vaping use behavior. CONCLUSIONS Users predicted to be in the 13-20 age group posted about different ENDS-related topics on Reddit than users predicted to be in the 21-54 age group. Future studies could use these complementary methods with social media data to gain insights from target audiences.
- Open Access
- 9 May 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Identifying Electronic Nicotine Delivery System Brands and Flavors on Instagram: Natural Language Processing Analysis (Preprint)
BACKGROUND Electronic nicotine delivery system (ENDS) brands, such as JUUL, used social media as a key component of their marketing strategy, which led to massive sales growth from 2015 to 2018. During this time, ENDS use rapidly increased among youths and young adults, with flavored products being particularly popular among these groups. OBJECTIVE The aim of our study is to develop a named entity recognition (NER) model to identify potential emerging vaping brands and flavors from Instagram post text. NER is a natural language processing task for identifying specific types of words (entities) in text based on the characteristics of the entity and surrounding words. METHODS NER models were trained on a labeled data set of 2272 Instagram posts coded for ENDS brands and flavors. We compared three types of NER models—conditional random fields, a residual convolutional neural network, and a fine-tuned distilled bidirectional encoder representations from transformers (FTDB) network—to identify brands and flavors in Instagram posts with key model outcomes of precision, recall, and F1 scores. We used data from Nielsen scanner sales and Wikipedia to create benchmark dictionaries to determine whether brands from established ENDS brand and flavor lists were mentioned in the Instagram posts in our sample. To prevent overfitting, we performed 5-fold cross-validation and reported the mean and SD of the model validation metrics across the folds. RESULTS For brands, the residual convolutional neural network exhibited the highest mean precision (0.797, SD 0.084), and the FTDB exhibited the highest mean recall (0.869, SD 0.103). For flavors, the FTDB exhibited both the highest mean precision (0.860, SD 0.055) and recall (0.801, SD 0.091). All NER models outperformed the benchmark brand and flavor dictionary look-ups on mean precision, recall, and F1. Comparing between the benchmark brand lists, the larger Wikipedia list outperformed the Nielsen list in both precision and recall. CONCLUSIONS Our findings suggest that NER models correctly identified ENDS brands and flavors in Instagram posts at rates competitive with, or better than, others in the published literature. Brands identified during manual annotation showed little overlap with those in Nielsen scanner data, suggesting that NER models may capture emerging brands with limited sales and distribution. NER models address the challenges of manual brand identification and can be used to support future infodemiology and infoveillance studies. Brands identified on social media should be cross-validated with Nielsen and other data sources to differentiate emerging brands that have become established from those with limited sales and distribution.
- 9 May 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessIdentifying Electronic Nicotine Delivery System Brands and Flavors on Instagram: Natural Language Processing Analysis (Preprint)
BACKGROUND Electronic nicotine delivery system (ENDS) brands, such as JUUL, used social media as a key component of their marketing strategy, which led to massive sales growth from 2015 to 2018. During this time, ENDS use rapidly increased among youths and young adults, with flavored products being particularly popular among these groups. OBJECTIVE The aim of our study is to develop a named entity recognition (NER) model to identify potential emerging vaping brands and flavors from Instagram post text. NER is a natural language processing task for identifying specific types of words (entities) in text based on the characteristics of the entity and surrounding words. METHODS NER models were trained on a labeled data set of 2272 Instagram posts coded for ENDS brands and flavors. We compared three types of NER models—conditional random fields, a residual convolutional neural network, and a fine-tuned distilled bidirectional encoder representations from transformers (FTDB) network—to identify brands and flavors in Instagram posts with key model outcomes of precision, recall, and F1 scores. We used data from Nielsen scanner sales and Wikipedia to create benchmark dictionaries to determine whether brands from established ENDS brand and flavor lists were mentioned in the Instagram posts in our sample. To prevent overfitting, we performed 5-fold cross-validation and reported the mean and SD of the model validation metrics across the folds. RESULTS For brands, the residual convolutional neural network exhibited the highest mean precision (0.797, SD 0.084), and the FTDB exhibited the highest mean recall (0.869, SD 0.103). For flavors, the FTDB exhibited both the highest mean precision (0.860, SD 0.055) and recall (0.801, SD 0.091). All NER models outperformed the benchmark brand and flavor dictionary look-ups on mean precision, recall, and F1. Comparing between the benchmark brand lists, the larger Wikipedia list outperformed the Nielsen list in both precision and recall. CONCLUSIONS Our findings suggest that NER models correctly identified ENDS brands and flavors in Instagram posts at rates competitive with, or better than, others in the published literature. Brands identified during manual annotation showed little overlap with those in Nielsen scanner data, suggesting that NER models may capture emerging brands with limited sales and distribution. NER models address the challenges of manual brand identification and can be used to support future infodemiology and infoveillance studies. Brands identified on social media should be cross-validated with Nielsen and other data sources to differentiate emerging brands that have become established from those with limited sales and distribution.
- Open Access
- 30 April 2021
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 7 (4) , e30017
Correction: Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation
[This corrects the article DOI: 10.2196/25807.].
- 30 April 2021
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 7 (4) , e30017
Open AccessCorrection: Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation
[This corrects the article DOI: 10.2196/25807.].
- Open Access
- 28 April 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Correction: Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
- 28 April 2021
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessCorrection: Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
- Open Access
- 16 March 2021
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 7 (3) , e25807
Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation
Background: Social media are important for monitoring perceptions of public health issues and for educating target audiences about health; however, limited information about the demographics of social media users makes it challenging to identify conversations among target audiences and limits how well social media can be used for public health surveillance and education outreach efforts. Certain social media platforms provide demographic information on followers of a user account, if given, but they are not always disclosed, and researchers have developed machine learning algorithms to predict social media users’ demographic characteristics, mainly for Twitter. To date, there has been limited research on predicting the demographic characteristics of Reddit users. Objective: We aimed to develop a machine learning algorithm that predicts the age segment of Reddit users, as either adolescents or adults, based on publicly available data. Methods: This study was conducted between January and September 2020 using publicly available Reddit posts as input data. We manually labeled Reddit users’ age by identifying and reviewing public posts in which Reddit users self-reported their age. We then collected sample posts, comments, and metadata for the labeled user accounts and created variables to capture linguistic patterns, posting behavior, and account details that would distinguish the adolescent age group (aged 13 to 20 years) from the adult age group (aged 21 to 54 years). We split the data into training (n=1660) and test sets (n=415) and performed 5-fold cross validation on the training set to select hyperparameters and perform feature selection. We ran multiple classification algorithms and tested the performance of the models (precision, recall, F1 score) in predicting the age segments of the users in the labeled data. To evaluate associations between each feature and the outcome, we calculated means and confidence intervals and compared the two age groups, with 2-sample t tests, for each transformed model feature. Results: The gradient boosted trees classifier performed the best, with an F1 score of 0.78. The test set precision and recall scores were 0.79 and 0.89, respectively, for the adolescent group (n=254) and 0.78 and 0.63, respectively, for the adult group (n=161). The most important feature in the model was the number of sentences per comment (permutation score: mean 0.100, SD 0.004). Members of the adolescent age group tended to have created accounts more recently, have higher proportions of submissions and comments in the r/teenagers subreddit, and post more in subreddits with higher subscriber counts than those in the adult group. Conclusions: We created a Reddit age prediction algorithm with competitive accuracy using publicly available data, suggesting machine learning methods can help public health agencies identify age-related target audiences on Reddit. Our results also suggest that there are characteristics of Reddit users’ posting behavior, linguistic patterns, and account features that distinguish adolescents from adults.
- 16 March 2021
- Published by JMIR Publications Inc. in JMIR Public Health and Surveillance
- Vol. 7 (3) , e25807
Open AccessPredicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation
Background: Social media are important for monitoring perceptions of public health issues and for educating target audiences about health; however, limited information about the demographics of social media users makes it challenging to identify conversations among target audiences and limits how well social media can be used for public health surveillance and education outreach efforts. Certain social media platforms provide demographic information on followers of a user account, if given, but they are not always disclosed, and researchers have developed machine learning algorithms to predict social media users’ demographic characteristics, mainly for Twitter. To date, there has been limited research on predicting the demographic characteristics of Reddit users. Objective: We aimed to develop a machine learning algorithm that predicts the age segment of Reddit users, as either adolescents or adults, based on publicly available data. Methods: This study was conducted between January and September 2020 using publicly available Reddit posts as input data. We manually labeled Reddit users’ age by identifying and reviewing public posts in which Reddit users self-reported their age. We then collected sample posts, comments, and metadata for the labeled user accounts and created variables to capture linguistic patterns, posting behavior, and account details that would distinguish the adolescent age group (aged 13 to 20 years) from the adult age group (aged 21 to 54 years). We split the data into training (n=1660) and test sets (n=415) and performed 5-fold cross validation on the training set to select hyperparameters and perform feature selection. We ran multiple classification algorithms and tested the performance of the models (precision, recall, F1 score) in predicting the age segments of the users in the labeled data. To evaluate associations between each feature and the outcome, we calculated means and confidence intervals and compared the two age groups, with 2-sample t tests, for each transformed model feature. Results: The gradient boosted trees classifier performed the best, with an F1 score of 0.78. The test set precision and recall scores were 0.79 and 0.89, respectively, for the adolescent group (n=254) and 0.78 and 0.63, respectively, for the adult group (n=161). The most important feature in the model was the number of sentences per comment (permutation score: mean 0.100, SD 0.004). Members of the adolescent age group tended to have created accounts more recently, have higher proportions of submissions and comments in the r/teenagers subreddit, and post more in subreddits with higher subscriber counts than those in the adult group. Conclusions: We created a Reddit age prediction algorithm with competitive accuracy using publicly available data, suggesting machine learning methods can help public health agencies identify age-related target audiences on Reddit. Our results also suggest that there are characteristics of Reddit users’ posting behavior, linguistic patterns, and account features that distinguish adolescents from adults.
- Open Access
- 17 November 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Predicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Social media are important for monitoring perceptions of public health issues and for educating target audiences about health; however, limited information about the demographics of social media users makes it challenging to identify conversations among target audiences and limits how well social media can be used for public health surveillance and education outreach efforts. Certain social media platforms provide demographic information on followers of a user account, if given, but they are not always disclosed, and researchers have developed machine learning algorithms to predict social media users’ demographic characteristics, mainly for Twitter. To date, there has been limited research on predicting the demographic characteristics of Reddit users. Objective: We aimed to develop a machine learning algorithm that predicts the age segment of Reddit users, as either adolescents or adults, based on publicly available data. Methods: This study was conducted between January and September 2020 using publicly available Reddit posts as input data. We manually labeled Reddit users’ age by identifying and reviewing public posts in which Reddit users self-reported their age. We then collected sample posts, comments, and metadata for the labeled user accounts and created variables to capture linguistic patterns, posting behavior, and account details that would distinguish the adolescent age group (aged 13 to 20 years) from the adult age group (aged 21 to 54 years). We split the data into training (n=1660) and test sets (n=415) and performed 5-fold cross validation on the training set to select hyperparameters and perform feature selection. We ran multiple classification algorithms and tested the performance of the models (precision, recall, F1 score) in predicting the age segments of the users in the labeled data. To evaluate associations between each feature and the outcome, we calculated means and confidence intervals and compared the two age groups, with 2-sample t tests, for each transformed model feature. Results: The gradient boosted trees classifier performed the best, with an F1 score of 0.78. The test set precision and recall scores were 0.79 and 0.89, respectively, for the adolescent group (n=254) and 0.78 and 0.63, respectively, for the adult group (n=161). The most important feature in the model was the number of sentences per comment (permutation score: mean 0.100, SD 0.004). Members of the adolescent age group tended to have created accounts more recently, have higher proportions of submissions and comments in the r/teenagers subreddit, and post more in subreddits with higher subscriber counts than those in the adult group. Conclusions: We created a Reddit age prediction algorithm with competitive accuracy using publicly available data, suggesting machine learning methods can help public health agencies identify age-related target audiences on Reddit. Our results also suggest that there are characteristics of Reddit users’ posting behavior, linguistic patterns, and account features that distinguish adolescents from adults.
- 17 November 2020
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessPredicting Age Groups of Reddit Users Based on Posting Behavior and Metadata: Classification Model Development and Validation (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Social media are important for monitoring perceptions of public health issues and for educating target audiences about health; however, limited information about the demographics of social media users makes it challenging to identify conversations among target audiences and limits how well social media can be used for public health surveillance and education outreach efforts. Certain social media platforms provide demographic information on followers of a user account, if given, but they are not always disclosed, and researchers have developed machine learning algorithms to predict social media users’ demographic characteristics, mainly for Twitter. To date, there has been limited research on predicting the demographic characteristics of Reddit users. Objective: We aimed to develop a machine learning algorithm that predicts the age segment of Reddit users, as either adolescents or adults, based on publicly available data. Methods: This study was conducted between January and September 2020 using publicly available Reddit posts as input data. We manually labeled Reddit users’ age by identifying and reviewing public posts in which Reddit users self-reported their age. We then collected sample posts, comments, and metadata for the labeled user accounts and created variables to capture linguistic patterns, posting behavior, and account details that would distinguish the adolescent age group (aged 13 to 20 years) from the adult age group (aged 21 to 54 years). We split the data into training (n=1660) and test sets (n=415) and performed 5-fold cross validation on the training set to select hyperparameters and perform feature selection. We ran multiple classification algorithms and tested the performance of the models (precision, recall, F1 score) in predicting the age segments of the users in the labeled data. To evaluate associations between each feature and the outcome, we calculated means and confidence intervals and compared the two age groups, with 2-sample t tests, for each transformed model feature. Results: The gradient boosted trees classifier performed the best, with an F1 score of 0.78. The test set precision and recall scores were 0.79 and 0.89, respectively, for the adolescent group (n=254) and 0.78 and 0.63, respectively, for the adult group (n=161). The most important feature in the model was the number of sentences per comment (permutation score: mean 0.100, SD 0.004). Members of the adolescent age group tended to have created accounts more recently, have higher proportions of submissions and comments in the r/teenagers subreddit, and post more in subreddits with higher subscriber counts than those in the adult group. Conclusions: We created a Reddit age prediction algorithm with competitive accuracy using publicly available data, suggesting machine learning methods can help public health agencies identify age-related target audiences on Reddit. Our results also suggest that there are characteristics of Reddit users’ posting behavior, linguistic patterns, and account features that distinguish adolescents from adults.
- Open Access
- 28 May 2020
- Published by RTI Press
Visual Attention to Tobacco-Related Stimuli in a 3D Virtual Store
We used eye tracking to measure visual attention to tobacco products and pro- and anti-tobacco advertisements (pro-ads and anti-ads) during a shopping task in a three-dimensional virtual convenience store. We used eye-tracking hardware to track the percentage of fixations (number of times the eye was essentially stationary; F) and dwell time (time spent looking at an object; DT) for several categories of objects and ads for 30 adult current cigarette smokers. We used Wald F-tests to compare fixations and dwell time across categories, adjusting comparisons of ads by the number of each type of ad. Overall, unadjusted for the number of each object, participants focused significantly greater attention on snacks and drinks and tobacco products than ads (all ). Adjusting for the number of each type of ad viewed, participants devoted significantly greater visual attention to pro-ads than anti-ads or ads unrelated to tobacco (). Visual attention for anti-ads was significantly greater when the ads were placed on the store’s external walls or hung from the ceiling than when placed on the gas pump or floor (). In a cluttered convenience store environment, anti-ads at the point of sale have to compete with many other stimuli. Restrictions on tobacco product displays and advertisements at the point of sale could reduce the stimuli that attract smokers’ attention away from anti-ads.- 28 May 2020
- Published by RTI Press
Open AccessVisual Attention to Tobacco-Related Stimuli in a 3D Virtual Store
We used eye tracking to measure visual attention to tobacco products and pro- and anti-tobacco advertisements (pro-ads and anti-ads) during a shopping task in a three-dimensional virtual convenience store. We used eye-tracking hardware to track the percentage of fixations (number of times the eye was essentially stationary; F) and dwell time (time spent looking at an object; DT) for several categories of objects and ads for 30 adult current cigarette smokers. We used Wald F-tests to compare fixations and dwell time across categories, adjusting comparisons of ads by the number of each type of ad. Overall, unadjusted for the number of each object, participants focused significantly greater attention on snacks and drinks and tobacco products than ads (all ). Adjusting for the number of each type of ad viewed, participants devoted significantly greater visual attention to pro-ads than anti-ads or ads unrelated to tobacco (). Visual attention for anti-ads was significantly greater when the ads were placed on the store’s external walls or hung from the ceiling than when placed on the gas pump or floor (). In a cluttered convenience store environment, anti-ads at the point of sale have to compete with many other stimuli. Restrictions on tobacco product displays and advertisements at the point of sale could reduce the stimuli that attract smokers’ attention away from anti-ads.