Daniel L. Oberski

Artificial intelligence based tools are being developed for decision support in healthcare, however, they are frequently found to lack the required functionality to achieve the clinical goals for which they were built. This results in wasted time, money and resources for hospitals attempting to implement and operate such tools. To determine how functionality issues can be resolved prior to tool implementation, it is necessary to understand why such tools are being designed and then built. Our research focuses on clinical decision support tools with functionality issues arising from target variable invalidity. In this paper, we analyze published articles which present clinical decision support tool designs related to clinical goals. These tools use machine learning models trained on electronic health record data. We find that design decisions driven by data availability can introduce construct invalidity in clinical decision support tool designs, leading to an inability of the tool to address the clinical goal. We observe that alternative goals to the main clinical goal are used to justify continued development. We show that functional limitations of the tool related to the clinical goal can be obscured by imprecise terminology in the model’s stated functionality. Finally, we highlight the need for reconsidered approaches to dataset creation, defining success criteria, and the reporting and transparency of research outcomes as they relate to clinical goals.

Clustering is a set of statistical techniques widely applied in the social sciences. While an important and useful tool, traditional clustering techniques tend to assume that the data are free from measurement error, which is often an unrealistic assumption. In this paper, we perform a Monte Carlo study to investigate the sensitivity of different clustering techniques to measurement error. We focus on three commonly used approaches: latent profile analysis (LPA), hierarchical clustering using Ward’s method, and k-means. We examine how the error affects the interpretability of the clusters and the classification of observations into clusters. Our results indicate that LPA fares better in the presence of error. In fact, clustering results from LPA can still be trusted when there is random error affecting one variable. K-means and Ward’s method, on the other hand, appear to already’break down’ when random error affects one variable and lead to inaccurate classifications. When the error is systematic and/or it affects more variables, all clustering methods produce severely biased results.

Learning general-purpose user representations based on user behavioral logs is an increasingly popular user modeling approach. It benefits from easily available, privacy-friendly yet expressive data, and does not require extensive re-tuning of the upstream user model for different downstream tasks. While this approach has shown promise in search engines and e-commerce applications, its fit for instant messaging platforms, a cornerstone of modern digital communication, remains largely uncharted. We explore this research gap using Snapchat data as a case study. Specifically, we implement a Transformer-based user model with customized training objectives and show that the model can produce high-quality user representations across a broad range of evaluation tasks, among which we introduce three new downstream tasks that concern pivotal topics in user research: user safety, engagement and churn. We also tackle the challenge of efficient extrapolation of long sequences at inference time, by applying a novel positional encoding method.

Aims: Non-invasive remote patient monitoring is an increasingly popular technique to aid clinicians in the early detection of worsening heart failure (HF) alongside regular follow-ups. However, previous studies have shown mixed results in the performance of such systems. Therefore, we developed and evaluated a personalized monitoring algorithm aimed at increasing positive-predictive-value (PPV) (i.e. alarm quality) and compared performance with simple rule-of-thumb and moving average convergence-divergence algorithms (MACD). Methods and results: In this proof-of-concept study, the developed algorithm was applied to retrospective data of daily bodyweight, heart rate, and systolic blood pressure of 74 HF-patients with a median observation period of 327 days (IQR: 183 days), during which 31 patients experienced 64 clinical worsening HF episodes. The algorithm combined information on both the monitored patients and a group of stable HF patients, and is increasingly personalized over time, using linear mixed-effect modelling and statistical process control charts. Optimized on alarm quality, heart rate showed the highest PPV (Personalized: 92%, MACD: 2%, Rule-of-thumb: 7%) with an F1 score of (Personalized: 28%, MACD: 6%, Rule-of-thumb: 8%). Bodyweight demonstrated the lowest PPV (Personalized: 16%, MACD: 0%, Rule-of-thumb: 6%) and F1 score (Personalized: 10%, MACD: 3%, Rule-of-thumb: 7%) overall compared methods. Conclusion: The personalized algorithm with flexible patient-tailored thresholds led to higher PPV, and performance was more sensitive compared to common simple monitoring methods (rule-of-thumb and MACD). However, many episodes of worsening HF remained undetected. Heart rate and systolic blood pressure monitoring outperformed bodyweight in predicting worsening HF. The algorithm source code is publicly available for future validation and improvement.

Text-based personality computing (TPC) has gained many research interests in NLP. In this paper, we describe 15 challenges that we consider deserving the attention of the research community. These challenges are organized by the following topics: personality taxonomies, measurement quality, datasets, performance evaluation, modelling choices, as well as ethics and fairness. When addressing each challenge, not only do we combine perspectives from both NLP and social sciences, but also offer concrete suggestions. We hope to inspire more valid and reliable TPC research.

Text embedding models from Natural Language Processing can map text data (e.g. words, sentences, documents) to supposedly meaningful numerical representations (a.k.a. text embeddings). While such models are increasingly applied in social science research, one important issue is often not addressed: the extent to which these embeddings are valid representations of constructs relevant for social science research. We therefore propose the use of the classic construct validity framework to evaluate the validity of text embeddings. We show how this framework can be adapted to the opaque and high-dimensional nature of text embeddings, with application to survey questions. We include several popular text embedding methods (e.g. fastText, GloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct validity analyses. We find evidence of convergent and discriminant validity in some cases. We also show that embeddings can be used to predict respondent's answers to completely new survey questions. Furthermore, BERT-based embedding techniques and the Universal Sentence Encoder provide more valid representations of survey questions than do others. Our results thus highlight the necessity to examine the construct validity of text embeddings before deploying them in social science research.