Nicolas Della Penna
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Nicolas Della Penna has a total of 351 co-authors. Here are the top co-authors by number of shared publications:
- Robert Stretch
Division of Pulmonary and Critical Care, Department of Medicine, UCLA Medical Center, Los Angeles, CA, United States of America
3 shared publications Leo Anthony Celi
Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA
2 shared publicationsAaron Gerding
University of Massachusetts, Amherst
1 shared publicationAaron Rumack
Carnegie Mellon University
1 shared publicationAbdul H Kanji
University of Massachusetts, Amherst
1 shared publicationAddison J Hu
Carnegie Mellon University
1 shared publication
Latest Publications
- Open Access
Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US
Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance Statement This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.
Open AccessEvaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US
Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance Statement This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.
- Open Access
- 21 April 2018
- Published by Association for Computing Machinery (ACM)
- p. 1-13
An Experimental Study of Cryptocurrency Market Dynamics
As cryptocurrencies gain popularity and credibility, marketplaces for cryptocurrencies are growing in importance. Understanding the dynamics of these markets can help to assess how viable the cryptocurrnency ecosystem is and how design choices affect market behavior. One existential threat to cryptocurrencies is dramatic fluctuations in traders' willingness to buy or sell. Using a novel experimental methodology, we conducted an online experiment to study how susceptible traders in these markets are to peer influence from trading behavior. We created bots that executed over one hundred thousand trades costing less than a penny each in 217 cryptocurrencies over the course of six months. We find that individual "buy" actions led to short-term increases in subsequent buy-side activity hundreds of times the size of our interventions. From a design perspective, we note that the design choices of the exchange we study may have promoted this and other peer influence effects, which highlights the potential social and economic impact of HCI in the design of digital institutions.Comment: To appear in CHI 201
- 21 April 2018
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Open AccessAn Experimental Study of Cryptocurrency Market Dynamics
As cryptocurrencies gain popularity and credibility, marketplaces for cryptocurrencies are growing in importance. Understanding the dynamics of these markets can help to assess how viable the cryptocurrnency ecosystem is and how design choices affect market behavior. One existential threat to cryptocurrencies is dramatic fluctuations in traders' willingness to buy or sell. Using a novel experimental methodology, we conducted an online experiment to study how susceptible traders in these markets are to peer influence from trading behavior. We created bots that executed over one hundred thousand trades costing less than a penny each in 217 cryptocurrencies over the course of six months. We find that individual "buy" actions led to short-term increases in subsequent buy-side activity hundreds of times the size of our interventions. From a design perspective, we note that the design choices of the exchange we study may have promoted this and other peer influence effects, which highlights the potential social and economic impact of HCI in the design of digital institutions.Comment: To appear in CHI 201
- 1 April 2018
- Published by Wolters Kluwer Health in Critical Care Medicine
- Vol. 46 (4) , 525-531
Effect of Boarding on Mortality in ICUs
Objectives: Hospitals use a variety of strategies to maximize the availability of limited ICU beds. Boarding, which involves assigning patients to an open bed in a different subspecialty ICU, is one such practice employed when ICU occupancy levels are high, and beds in a particular unit are unavailable. Boarding disrupts the normal geographic colocation of patients and care teams, exposing patients to nursing staff with different training and expertise to those caring for nonboarders. We analyzed whether medical ICU patients boarding in alternative specialty ICUs are at increased risk of mortality. Design: Retrospective cohort study using an instrumental variable analysis to control for unmeasured confounding. A semiparametric bivariate probit estimation strategy was employed for the instrumental model. Propensity score matching and standard logistic regression (generalized linear modeling) were used as robustness checks. Setting: The medical ICU of a tertiary care nonprofit hospital in the United States between 2002 and 2012. Patients: All medical ICU admissions during the specified time period. Interventions: None. Measurements and Main Results: The study population consisted of 8,429 patients of whom 1,871 were boarders. The instrumental variable model demonstrated a relative risk of 1.18 (95% CI, 1.01–1.38) for ICU stay mortality for boarders. The relative risk of in-hospital mortality among boarders was 1.22 (95% CI, 1.00–1.49). GLM and propensity score matching without use of the instrument yielded similar estimates. Instrumental variable estimates are for marginal patients, whereas generalized linear modeling and propensity score matching yield population average effects. Conclusions: Mortality increased with boarding of critically ill patients. Further research is needed to identify safer practices for managing patients during periods of high ICU occupancy.
- 1 April 2018
- Published by Wolters Kluwer Health in Critical Care Medicine
- Vol. 46 (4) , 525-531
Effect of Boarding on Mortality in ICUs
Objectives: Hospitals use a variety of strategies to maximize the availability of limited ICU beds. Boarding, which involves assigning patients to an open bed in a different subspecialty ICU, is one such practice employed when ICU occupancy levels are high, and beds in a particular unit are unavailable. Boarding disrupts the normal geographic colocation of patients and care teams, exposing patients to nursing staff with different training and expertise to those caring for nonboarders. We analyzed whether medical ICU patients boarding in alternative specialty ICUs are at increased risk of mortality. Design: Retrospective cohort study using an instrumental variable analysis to control for unmeasured confounding. A semiparametric bivariate probit estimation strategy was employed for the instrumental model. Propensity score matching and standard logistic regression (generalized linear modeling) were used as robustness checks. Setting: The medical ICU of a tertiary care nonprofit hospital in the United States between 2002 and 2012. Patients: All medical ICU admissions during the specified time period. Interventions: None. Measurements and Main Results: The study population consisted of 8,429 patients of whom 1,871 were boarders. The instrumental variable model demonstrated a relative risk of 1.18 (95% CI, 1.01–1.38) for ICU stay mortality for boarders. The relative risk of in-hospital mortality among boarders was 1.22 (95% CI, 1.00–1.49). GLM and propensity score matching without use of the instrument yielded similar estimates. Instrumental variable estimates are for marginal patients, whereas generalized linear modeling and propensity score matching yield population average effects. Conclusions: Mortality increased with boarding of critically ill patients. Further research is needed to identify safer practices for managing patients during periods of high ICU occupancy.
- 1 December 2016
- Published by Wolters Kluwer Health in Critical Care Medicine
- Vol. 44 (12) , 362
1144: MORTALITY HETEROGENEITY OF GEOGRAPHIC CO-LOCALIZATION OF INTENSIVE CARE UNIT PATIENT AND CARE TEAM
- 1 December 2016
- Published by Wolters Kluwer Health in Critical Care Medicine
- Vol. 44 (12) , 362
1144: MORTALITY HETEROGENEITY OF GEOGRAPHIC CO-LOCALIZATION OF INTENSIVE CARE UNIT PATIENT AND CARE TEAM
- Open Access
- 10 September 2016
- Published by Springer Nature
- p. 285-294
Instrumental Variable Analysis of Electronic Health Records
Sources of variation in treatments received that are exogenous to patients can be used to estimate causal effects from observational data. We present an example of this methodology that estimates the effect of critically ill patients being cared for in “non-target ICUs” due to capacity constraints—a process known as boarding.
- 10 September 2016
- Published by Springer Nature
- p. 285-294
Open AccessInstrumental Variable Analysis of Electronic Health Records
Sources of variation in treatments received that are exogenous to patients can be used to estimate causal effects from observational data. We present an example of this methodology that estimates the effect of critically ill patients being cared for in “non-target ICUs” due to capacity constraints—a process known as boarding.
- 4 March 2016
- Published by American Association for the Advancement of Science (AAAS) in Science
- Vol. 351 (6277) , 1037
Response to Comment on “Estimating the reproducibility of psychological science”
Gilbert et al . conclude that evidence from the Open Science Collaboration’s Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.
- 4 March 2016
- Published by American Association for the Advancement of Science (AAAS) in Science
- Vol. 351 (6277) , 1037
Response to Comment on “Estimating the reproducibility of psychological science”
Gilbert et al . conclude that evidence from the Open Science Collaboration’s Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.
- Open Access
Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard
A common use of crowd sourcing is to obtain labels for a dataset. Several algorithms have been proposed to identify uninformative members of the crowd so that their labels can be disregarded and the cost of paying them avoided. One common motivation of these algorithms is to try and do without any initial set of trusted labeled data. We analyse this class of algorithms as mechanisms in a game-theoretic setting to understand the incentives they create for workers. We find an impossibility result that without any ground truth, and when workers have access to commonly shared 'prejudices' upon which they agree but are not informative of true labels, there is always equilibria where all agents report the prejudice. A small amount amount of gold standard data is found to be sufficient to rule out these equilibria.
Open AccessCrowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard
A common use of crowd sourcing is to obtain labels for a dataset. Several algorithms have been proposed to identify uninformative members of the crowd so that their labels can be disregarded and the cost of paying them avoided. One common motivation of these algorithms is to try and do without any initial set of trusted labeled data. We analyse this class of algorithms as mechanisms in a game-theoretic setting to understand the incentives they create for workers. We find an impossibility result that without any ground truth, and when workers have access to commonly shared 'prejudices' upon which they agree but are not informative of true labels, there is always equilibria where all agents report the prejudice. A small amount amount of gold standard data is found to be sufficient to rule out these equilibria.
- Open Access
- 16 April 2012
- Published by Association for Computing Machinery (ACM)
- p. 859-868
New objective functions for social collaborative filtering
This paper examines the problem of social collaborative filtering (CF) to recommend items of interest to users in a social network setting. Unlike standard CF algorithms using relatively simple user and item features, recommendation in social networks poses the more complex problem of learning user preferences from a rich and complex set of user profile and interaction information. Many existing social CF methods have extended traditional CF matrix factorization, but have overlooked important aspects germane to the social setting. We propose a unified framework for social CF matrix factorization by introducing novel objective functions for training. Our new objective functions have three key features that address main drawbacks of existing approaches: (a) we fully exploit feature-based user similarity, (b) we permit direct learning of user-to-user information diffusion, and (c) we leverage co-preference (dis)agreement between two users to learn restricted areas of common interest. We evaluate these new social CF objectives, comparing them to each other and to a variety of (social) CF baselines, and analyze user behavior on live user trials in a custom-developed Facebook App involving data collected over five months from over 100 App users and their 37,000+ friends.
- 16 April 2012
- Published by Association for Computing Machinery (ACM)
- p. 859-868
Open AccessNew objective functions for social collaborative filtering
This paper examines the problem of social collaborative filtering (CF) to recommend items of interest to users in a social network setting. Unlike standard CF algorithms using relatively simple user and item features, recommendation in social networks poses the more complex problem of learning user preferences from a rich and complex set of user profile and interaction information. Many existing social CF methods have extended traditional CF matrix factorization, but have overlooked important aspects germane to the social setting. We propose a unified framework for social CF matrix factorization by introducing novel objective functions for training. Our new objective functions have three key features that address main drawbacks of existing approaches: (a) we fully exploit feature-based user similarity, (b) we permit direct learning of user-to-user information diffusion, and (c) we leverage co-preference (dis)agreement between two users to learn restricted areas of common interest. We evaluate these new social CF objectives, comparing them to each other and to a variety of (social) CF baselines, and analyze user behavior on live user trials in a custom-developed Facebook App involving data collected over five months from over 100 App users and their 37,000+ friends.