MHMark Himmelstein

Mark Himmelstein

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Mark Himmelstein has a total of 25 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability

    by Mark Himmelstein, Sophie Ma Zhu, Nikolay Petrov, Ezra Karger, Jessica Helmer, Sivan Livnat, Page Hedley, Amory Bennett, Phil Tetlock

    Accurate predictions about the future are crucial for optimizing policy choices, but finding skilled forecasters is challenging. Prior work relied on forecasting tournaments to identify generalist "superforecasters" who made accurate predictions across domains, but these tournaments take months to complete and require large concurrent samples to compare performance among individuals. We solve these problems by developing a one-hour Forecasting Proficiency Test (FPT) that explains 75% of the variance in forecasters' accuracy about a wide variety of events. The FPT provides standardized percentile scores normed to a large (N=1,147) test development sample based on a combination of reusable real-world forecasting questions and standalone cognitive tests. Which questions and tests are included is highly customizable. Forecasting questions are the best predictors of forecasting proficiency, but cognitive tests help stabilize score distributions and facilitate score exchangeability across different custom test versions even as forecasting questions and resolution dates change. The FPT is primarily a practical tool for decision-makers who rely on accurate predictions to make decisions. However, it is based on one of the largest dedicated psychometric studies ever conducted on human forecasting, which also provides insight into the cognitive mechanisms that drive forecasting proficiency and guidance for future researchers interested in studying the psychology of forecasting.

  • Open Access

    The case for quantile forecasts: Enhancing reliability in the identification of forecaster talent

    by Sophie Ma Zhu, David V. Budescu, Nikolay Petrov, Ezra Karger, Mark Himmelstein

    What is the best way to elicit forecasts if the goal is to measure the skill of forecasters? For continuous outcomes, there are two common approaches: elicit probabilities at fixed quantiles of the outcome distribution (e.g., what is the probability gas will cost less than $4.00 per gallon in 1 month?), or elicit quantiles at fixed probabilities (e.g., what is the price such that there is a 25% chance a gallon of gas will cost less than it in 1 month?). We compare these methods using a simulation study and a longitudinal survey in which 1,194 participants provided five waves of forecasts in both formats on a total of 36 real-world questions. Although probability elicitation is more ubiquitous in the literature, we show in our simulation and survey data that quantile elicitation allows us to estimate forecaster skill more efficiently with less measurement error. While quantile elicitation is time-consuming and associated with comprehension difficulty indicated by incoherent forecasts (violations of monotonicity), there is a clear reduction in incoherence across survey waves, suggesting comprehension of the quantile elicitation format improves with practice and is trainable.

  • Identifying good forecasters via adaptive cognitive tests

    by Edgar C. Merkle, Nikolay Petrov, Sophie Ma Zhu, Ezra Karger, Philip E. Tetlock, Mark Himmelstein


  • Open Access

    Reducing Measurement Error in Forecaster Evaluation Using the Wisdom of Crowds

    by Jessica Helmer, Nikolay Petrov, Sophie Ma Zhu, Ezra Karger, Edgar C. Merkle, Mark Himmelstein

    Forecasting skill is often measured by the average error between forecasters’ predictions and the ground truth, but the inherent uncertainty in event outcomes adds an additional layer of measurement error onto skill estimation. Intersubjective measures offer an alternative approach to skill measurement: comparing forecasters’ predictions to those of their peers, typically to aggregated forecasts from groups of peers. A key advantage of this approach is that forecasts can be scored in real time, without having to wait for the ground truth outcome to be realized. However, it has another more subtle advantage as well: aggregate predictions can be less noisy than ground truth outcomes, leading to a potential reduction in this additionalmeasurement error. In a simulation study, we demonstrate conditions under which crowd aggregates provide a more reliable indicator of the optimal forecast for a given event than a single realized ground truth outcome, leading to skill measurement with reduced measurement error. We also demonstrate the effectiveness of intersubjective methods in a real-world forecasting study in which 894 participants made both forecasts and metapredictions, i.e., predictions of what others might forecast. As in our simulation, intersubjective measures capture forecasting ability more efficiently than do ground-truth scores, demonstrating their usefulness for identifying high-performing forecasters.

  • Open Access

    Measuring Forecasting Proficiency: A Psychometric Perspective

    by Fabio Setti, Leah Feuerstahler, Sophie Ma Zhu, Nikolay Petrov, Ezra Karger, Mark Himmelstein

    Identifying good forecasters, individuals who consistently make accurate predictions about future events, is one central research topic in the forecasting literature. One of the latest developments in this stream of research is the Forecasting Proficiency Test (FPT), which aims to provide a reliable and accessible measure of individual forecasting ability. A large portion of the FPT consists of quantile elicitation format (QEF) forecasts rather than the more common probability elicitation format (PEF). We propose a novel psychometric model that draws upon item response theory, with the goal of measuring item features of QEF items and capturing individual forecasting ability as a latent construct. In doing so, we outline important theoretical differences between latent constructs and accuracy metrics commonly employed in the forecasting literature to measure forecasting proficiency. We demonstrate the practical usefulness of the proposed model by applying it to the QEF questions of the FPT, where we find that it captures meaningful item features and adequately measures forecasting ability. Namely, the estimated latent construct forecasting ability showed a Pearson correlation of r = .85 with measures based on strictly proper scoring rules of forecasting accuracy. We conclude by discussing the practical benefits of measuring item parameters of forecasting items, the specific limitations of psychometric models applied to forecasting items, and when and where we believe they are most appropriate in the forecasting domain.

  • Forecasting long-run causal effects

    by David Rhys Bernard, Mark Himmelstein, Ezra Karger, Philipp Schoenegger

  • Open Access

    Causal Inference in the Absence of Direct Measurement Using Planned Missing Data

    by Mark Himmelstein, David Budescu

    Many psychological studies seek to quantify how treatments modify beliefs, attitudes, or behaviors by measuring the same participants on the same constructs both at baseline, before the treatment is administered, and again after it is administered. This procedure raises a problematic counterfactual: how do we know that the treatment is causing observed changes, and not the act of collecting a baseline measurement itself? This threat to validity is often referred to as \textit{measurement reactivity}. Researchers can attempt to quantify and even mitigate measurement reactivity by experimentally truncating baseline measurements, but those baseline measurements improve statistical power and may be critical for key analyses the researchers wish to conduct. We show that it is possible to indirectly recover baseline measurements by using auxiliary variables: variables which don't directly represent baseline measurements but have strong enough correlations with baseline measurements to facilitate informative imputation. We show how the measurement reactivity problem can be reframed using the logic of missing data mechanisms, demonstrate the feasibility of our proposed method in a simulation study, and then implement it in an experimental design on advice taking in which measurement reactivity is a known problem. We show that advice taking behavior clearly depends on the presence or absence of baseline measurement, and that important and ecologically valid insights about how people update their beliefs would remain undetected if measurement reactivity had remained unaddressed.

  • Open Access

    The Psychometric Properties of Probability and Quantile Forecasts

    by Sophie Ma Zhu, David Budescu, Nikolay Petrov, Ezra Karger, Mark Himmelstein

    Forecasting tournaments are a well established method for assessing human forecasting skills. Most forecasting tournaments are based on a format where participants estimate the probabilities of discrete events. For predictions of continuous values, the possible range of outcome values is divided into mutually exclusive bins covering the entire outcome distribution so that probabilities for each bin can be elicited. An alternative approach involves directly eliciting forecasts about quantiles of the continuous quantity. Using both simulated data and data from 1,147 participants who completed five surveys focused on forecasting tasks in a longitudinal study, we compared the psychometric properties of quantile and probability elicitation methods. In the simulation, we demonstrated that items in the quantile format recovered parameters that defined forecasters’ latent forecasting skill in fewer items than the probability format, and identified several idiosyncrasies in accuracy scores for the probability format that drive these differences. In the empirical analyses, we elicited forecasts about a set of 36 forecasting questions in both formats: quantile forecasts at five fixed probability values (5%, 25%, 50%, 75%, 95%) and probability forecasts for five pre-determined item-specific bins. Consistent with the simulated results, our findings revealed that forecasts in the quantile format showed considerably stronger internal consistency, achieving a suitable reliability level with fewer items. When cross-validating how well individual forecasters’ accuracy on in-sample questions predicted their performance in out-of-sample questions, the variability in the accuracy of quantile forecasts was more statistically explainable. Despite its desirable properties, errors and signs of comprehension difficulties were more frequently observed in the quantile format. Further research is needed to refine elicitations that optimize the effectiveness of quantile-based forecasting judgments.