Giles Hooker
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Giles Hooker has a total of 40 co-authors. Here are the top co-authors by number of shared publications:
- Lucas Mentch
University of Pittsburgh, Pittsburgh, USA
6 shared publications - Stephen P. Ellner
Department of Ecology and Evolutionary Biology, Cornell University Ithaca NY USA
4 shared publications - Indrayudh Ghosal
Department of Statistical Science Cornell University Ithaca 14850 NY USA
3 shared publications - Yichen Zhou
Department of Statistics and Data Science, Cornell University, Ithaca, USA
3 shared publications - Zhengze Zhou
Department of Statistical Science Department of Statistical Science Cornell University Cornell University
3 shared publications - Christina M. Hernández
Department of Biology Oxford University Oxford UK
2 shared publications
Latest Publications
- 1 June 2026
- Published by Institute of Mathematical Statistics in The Annals of Applied Statistics
- Vol. 20 (2) , 1033-1057
Targeted maximum likelihood estimation for integral projection models in population ecology
In population ecology, integral projection models (IPMs) are widely used to study population growth and the dynamics of population structure (e.g., age and size distributions). These models typically use data on the growth, survival, and reproduction of marked individuals to parameterize models for each demographic rate. The resulting models can be used to predict changes in the population from one time point to the next and to predict long-term properties such as long-term population growth rate, the sensitivity of that growth rate to environmental factors and to changes in demographic rates, and the variation in lifetime outcomes among individuals. These quantities must be inferred from the model, and we often lack any way to ground-truth their plausibility directly from the available data. They nonetheless reflect key ecological processes with significant management and policy implications. Building IPMs requires us to develop sub-models for individual fates over the next time step—Did they survive? How much did they grow or shrink? Did they reproduce?—conditional on their initial state as well as on environmental covariates. This must be done in a manner that gives the most precise possible estimates, and most accurate uncertainty quantification, for the long-term model properties that we are interested in predicting. These models include three core demographic submodels—for growth, survival, and reproduction rates—to describe how individuals change from one time point to the next. Targeted maximum likelihood estimation (TMLE) methods are particularly well suited to a situation in which we are largely interested in estimation and inference on quantities derived from models. These methods build machine learning-based models that estimate the probability distribution from which the empirical data were drawn, and then the user specifies a target of inference as a function of this distribution. An initial estimate for the distribution is then modified by tilting in the direction of the influence function to both de-bias the parameter estimate and provide more accurate inference. In this paper we employ TMLE to develop robust and efficient estimators for quantities derived from a fitted IPM as targets of interest, with a particular focus on long-term stable population growth, its elasticity to fecundity, and the expected growth rate under year-specific covariates. Mathematically, we derive the influence functions for these targets of influence and formulate and use these to update our model so as to remove bias from our estimates. Empirically, we conduct extensive simulations and demonstrate our method’s efficacy using empirical data from a long-term study of plant communities on the Idaho steppe and from experimental rotifer populations.
- 1 June 2026
- Published by Institute of Mathematical Statistics in The Annals of Applied Statistics
- Vol. 20 (2) , 1033-1057
Targeted maximum likelihood estimation for integral projection models in population ecology
In population ecology, integral projection models (IPMs) are widely used to study population growth and the dynamics of population structure (e.g., age and size distributions). These models typically use data on the growth, survival, and reproduction of marked individuals to parameterize models for each demographic rate. The resulting models can be used to predict changes in the population from one time point to the next and to predict long-term properties such as long-term population growth rate, the sensitivity of that growth rate to environmental factors and to changes in demographic rates, and the variation in lifetime outcomes among individuals. These quantities must be inferred from the model, and we often lack any way to ground-truth their plausibility directly from the available data. They nonetheless reflect key ecological processes with significant management and policy implications. Building IPMs requires us to develop sub-models for individual fates over the next time step—Did they survive? How much did they grow or shrink? Did they reproduce?—conditional on their initial state as well as on environmental covariates. This must be done in a manner that gives the most precise possible estimates, and most accurate uncertainty quantification, for the long-term model properties that we are interested in predicting. These models include three core demographic submodels—for growth, survival, and reproduction rates—to describe how individuals change from one time point to the next. Targeted maximum likelihood estimation (TMLE) methods are particularly well suited to a situation in which we are largely interested in estimation and inference on quantities derived from models. These methods build machine learning-based models that estimate the probability distribution from which the empirical data were drawn, and then the user specifies a target of inference as a function of this distribution. An initial estimate for the distribution is then modified by tilting in the direction of the influence function to both de-bias the parameter estimate and provide more accurate inference. In this paper we employ TMLE to develop robust and efficient estimators for quantities derived from a fitted IPM as targets of interest, with a particular focus on long-term stable population growth, its elasticity to fecundity, and the expected growth rate under year-specific covariates. Mathematically, we derive the influence functions for these targets of influence and formulate and use these to update our model so as to remove bias from our estimates. Empirically, we conduct extensive simulations and demonstrate our method’s efficacy using empirical data from a long-term study of plant communities on the Idaho steppe and from experimental rotifer populations.
- Open Access
- 13 January 2026
- Published by SAGE Publications in Journal of Marketing
Does Amazon's Dual Role Weaken Marketplace Competition?
Amazon's dual role, as both marketplace owner and first-party (1P) seller, gives it power over third-party (3P) sellers who sell similar items. This dual role can weaken 3P sellers’ ability to compete, possibly harming 3P sellers and consumers. We examine three aspects of marketplace competition. First, we examine price change dependencies. We find that 1P prices drop after either Buy Box (i.e., the “Add to Cart” or default sales box on Amazon's product page) prices increase or large 3P price increases occur; 3P prices decrease subsequently. Second, we analyze Buy Box seller selection because this is a critical conduit for demand. We find that both high 1P and 3P prices are penalized in Buy Box selection. Low-reputation and intermittent 3P sellers cannot win the Buy Box even at significantly lower prices. For some prices, the Buy Box favors 1P sellers over equal-priced 3P sellers and vice versa for other prices. Third, to see whether entry barriers weaken competition, we estimate a 3P seller entry model. Higher 1P prices are associated with more 3P sellers, suggesting low entry barriers. Combined, our results suggest that Amazon's dual role does not weaken competition in the marketplace. We discuss implications for marketplace participants, antitrust policy, and research.
- 13 January 2026
- Published by SAGE Publications in Journal of Marketing
Open AccessDoes Amazon's Dual Role Weaken Marketplace Competition?
Amazon's dual role, as both marketplace owner and first-party (1P) seller, gives it power over third-party (3P) sellers who sell similar items. This dual role can weaken 3P sellers’ ability to compete, possibly harming 3P sellers and consumers. We examine three aspects of marketplace competition. First, we examine price change dependencies. We find that 1P prices drop after either Buy Box (i.e., the “Add to Cart” or default sales box on Amazon's product page) prices increase or large 3P price increases occur; 3P prices decrease subsequently. Second, we analyze Buy Box seller selection because this is a critical conduit for demand. We find that both high 1P and 3P prices are penalized in Buy Box selection. Low-reputation and intermittent 3P sellers cannot win the Buy Box even at significantly lower prices. For some prices, the Buy Box favors 1P sellers over equal-priced 3P sellers and vice versa for other prices. Third, to see whether entry barriers weaken competition, we estimate a 3P seller entry model. Higher 1P prices are associated with more 3P sellers, suggesting low entry barriers. Combined, our results suggest that Amazon's dual role does not weaken competition in the marketplace. We discuss implications for marketplace participants, antitrust policy, and research.
- Open Access
Targeted Maximum Likelihood Estimation for Integral Projection Models in Population Ecology
Integral projection models (IPMs) are widely used to study population growth and the dynamics of demographic structure (e.g. age and size distributions) within a population.These models use data on individuals' growth, survival, and reproduction to predict changes in the population from one time point to the next and use these in turn to ask about long-term growth rates, the sensitivity of that growth rate to environmental factors, and aspects of the long term population such as how much reproduction concentrates in a few individuals; these quantities are not directly measurable from data and must be inferred from the model. Building IPMs requires us to develop models for individual fates over the next time step -- Did they survive? How much did they grow or shrink? Did they Reproduce? -- conditional on their initial state as well as on environmental covariates in a manner that accounts for the unobservable quantities that are are ultimately interested in estimating.Targeted maximum likelihood estimation (TMLE) methods are particularly well-suited to a framework in which we are largely interested in the consequences of models. These build machine learning-based models that estimate the probability distribution of the data we observe and define a target of inference as a function of these. The initial estimate for the distribution is then modified by tilting in the direction of the efficient influence function to both de-bias the parameter estimate and provide more accurate inference. In this paper, we employ TMLE to develop robust and efficient estimators for properties derived from a fitted IPM. Mathematically, we derive the efficient influence function and formulate the paths for the least favorable sub-models. Empirically, we conduct extensive simulations using real data from both long term studies of Idaho steppe plant communities and experimental Rotifer populations.
Open AccessTargeted Maximum Likelihood Estimation for Integral Projection Models in Population Ecology
Integral projection models (IPMs) are widely used to study population growth and the dynamics of demographic structure (e.g. age and size distributions) within a population.These models use data on individuals' growth, survival, and reproduction to predict changes in the population from one time point to the next and use these in turn to ask about long-term growth rates, the sensitivity of that growth rate to environmental factors, and aspects of the long term population such as how much reproduction concentrates in a few individuals; these quantities are not directly measurable from data and must be inferred from the model. Building IPMs requires us to develop models for individual fates over the next time step -- Did they survive? How much did they grow or shrink? Did they Reproduce? -- conditional on their initial state as well as on environmental covariates in a manner that accounts for the unobservable quantities that are are ultimately interested in estimating.Targeted maximum likelihood estimation (TMLE) methods are particularly well-suited to a framework in which we are largely interested in the consequences of models. These build machine learning-based models that estimate the probability distribution of the data we observe and define a target of inference as a function of these. The initial estimate for the distribution is then modified by tilting in the direction of the efficient influence function to both de-bias the parameter estimate and provide more accurate inference. In this paper, we employ TMLE to develop robust and efficient estimators for properties derived from a fitted IPM. Mathematically, we derive the efficient influence function and formulate the paths for the least favorable sub-models. Empirically, we conduct extensive simulations using real data from both long term studies of Idaho steppe plant communities and experimental Rotifer populations.
- 28 March 2024
- Published by Wiley in Ecology Letters
- Vol. 27 (3) , e14390
The natural history of luck: A synthesis study of structured population models
Chance pervades life. In turn, life histories are described by probabilities (e.g. survival and breeding) and averages across individuals (e.g. mean growth rate and age at maturity). In this study, we explored patterns of luck in lifetime outcomes by analysing structured population models for a wide array of plant and animal species. We calculated four response variables: variance and skewness in both lifespan and lifetime reproductive output (LRO), and partitioned them into contributions from different forms of luck. We examined relationships among response variables and a variety of life history traits. We found that variance in lifespan and variance in LRO were positively correlated across taxa, but that variance and skewness were negatively correlated for both lifespan and LRO. The most important life history trait was longevity, which shaped variance and skew in LRO through its effects on variance in lifespan. We found that luck in survival, growth, and fecundity all contributed to variance in LRO, but skew in LRO was overwhelmingly due to survival luck. Rapidly growing populations have larger variances in LRO and lifespan than shrinking populations. Our results indicate that luck‐induced genetic drift may be most severe in recovering populations of species with long mature lifespan and high iteroparity.
- 28 March 2024
- Published by Wiley in Ecology Letters
- Vol. 27 (3) , e14390
The natural history of luck: A synthesis study of structured population models
Chance pervades life. In turn, life histories are described by probabilities (e.g. survival and breeding) and averages across individuals (e.g. mean growth rate and age at maturity). In this study, we explored patterns of luck in lifetime outcomes by analysing structured population models for a wide array of plant and animal species. We calculated four response variables: variance and skewness in both lifespan and lifetime reproductive output (LRO), and partitioned them into contributions from different forms of luck. We examined relationships among response variables and a variety of life history traits. We found that variance in lifespan and variance in LRO were positively correlated across taxa, but that variance and skewness were negatively correlated for both lifespan and LRO. The most important life history trait was longevity, which shaped variance and skew in LRO through its effects on variance in lifespan. We found that luck in survival, growth, and fecundity all contributed to variance in LRO, but skew in LRO was overwhelmingly due to survival luck. Rapidly growing populations have larger variances in LRO and lifespan than shrinking populations. Our results indicate that luck‐induced genetic drift may be most severe in recovering populations of species with long mature lifespan and high iteroparity.
- Open Access
A Generic Approach for Reproducible Model Distillation
Model distillation has been a popular method for producing interpretable machine learning. It uses an interpretable "student" model to mimic the predictions made by the black box "teacher" model. However, when the student model is sensitive to the variability of the data sets used for training even when keeping the teacher fixed, the corresponded interpretation is not reliable. Existing strategies stabilize model distillation by checking whether a large enough corpus of pseudo-data is generated to reliably reproduce student models, but methods to do so have so far been developed for a specific student model. In this paper, we develop a generic approach for stable model distillation based on central limit theorem for the average loss. We start with a collection of candidate student models and search for candidates that reasonably agree with the teacher. Then we construct a multiple testing framework to select a corpus size such that the consistent student model would be selected under different pseudo samples. We demonstrate the application of our proposed approach on three commonly used intelligible models: decision trees, falling rule lists and symbolic regression. Finally, we conduct simulation experiments on Mammographic Mass and Breast Cancer datasets and illustrate the testing procedure throughout a theoretical analysis with Markov process. The code is publicly available at https://github.com/yunzhe-zhou/GenericDistillation.
Open AccessA Generic Approach for Reproducible Model Distillation
Model distillation has been a popular method for producing interpretable machine learning. It uses an interpretable "student" model to mimic the predictions made by the black box "teacher" model. However, when the student model is sensitive to the variability of the data sets used for training even when keeping the teacher fixed, the corresponded interpretation is not reliable. Existing strategies stabilize model distillation by checking whether a large enough corpus of pseudo-data is generated to reliably reproduce student models, but methods to do so have so far been developed for a specific student model. In this paper, we develop a generic approach for stable model distillation based on central limit theorem for the average loss. We start with a collection of candidate student models and search for candidates that reasonably agree with the teacher. Then we construct a multiple testing framework to select a corpus size such that the consistent student model would be selected under different pseudo samples. We demonstrate the application of our proposed approach on three commonly used intelligible models: decision trees, falling rule lists and symbolic regression. Finally, we conduct simulation experiments on Mammographic Mass and Breast Cancer datasets and illustrate the testing procedure throughout a theoretical analysis with Markov process. The code is publicly available at https://github.com/yunzhe-zhou/GenericDistillation.
- Open Access
- 1 February 2023
- Published by Wiley in Methods in Ecology and Evolution
- Vol. 14 (3) , 939-951
An exact version of Life Table Response Experiment analysis, and the R package exactLTRE
Matrix population models are frequently built and used by ecologists to analyse demography and elucidate the processes driving population growth or decline. Life Table Response Experiments (LTREs) are comparative analyses that decompose the realized difference or variance in population growth rate () into contributions from the differences or variances in the vital rates (i.e. the matrix elements). Since their introduction, LTREs have been based on approximations and have not included biologically relevant interaction terms. We used the functional analysis of variance framework to derive an exact LTRE method, which calculates the exact response of to the difference or variance in a given vital rate, for all interactions among vital rates—including higher‐order interactions neglected by the classical methods. We used the publicly available COMADRE and COMPADRE databases to perform a meta‐analysis comparing the results of exact and classical LTRE methods. We analysed 186 and 1487 LTREs for animal and plant matrix population models, respectively. We found that the classical methods often had small errors, but that very high errors were possible. Overall error was related to the difference or variance in the matrices being analysed, consistent with the Taylor series basis of the classical method. Neglected interaction terms accounted for most of the errors in fixed design LTRE, highlighting the importance of two‐way interaction terms. For random design LTRE, errors in the contribution terms present in both classical and exact methods were comparable to errors due to neglected interaction terms. In most examples we analysed, evaluating exact contributions up to three‐way interaction terms was sufficient for interpreting 90% or more of the difference or variance in . Relative error, previously used to evaluate the accuracy of classical LTREs, is not a reliable metric of how closely the classical and exact methods agree. Error compensation between estimated contribution terms and neglected contribution terms can lead to low relative error despite faulty biological interpretation. Trade‐offs or negative covariances among matrix elements can lead to high relative error despite accurate biological interpretation. Exact LTRE provides reliable and accurate biological interpretation, and the R package exactLTRE makes the exact method accessible to ecologists.
- 1 February 2023
- Published by Wiley in Methods in Ecology and Evolution
- Vol. 14 (3) , 939-951
Open AccessAn exact version of Life Table Response Experiment analysis, and the R package exactLTRE
Matrix population models are frequently built and used by ecologists to analyse demography and elucidate the processes driving population growth or decline. Life Table Response Experiments (LTREs) are comparative analyses that decompose the realized difference or variance in population growth rate () into contributions from the differences or variances in the vital rates (i.e. the matrix elements). Since their introduction, LTREs have been based on approximations and have not included biologically relevant interaction terms. We used the functional analysis of variance framework to derive an exact LTRE method, which calculates the exact response of to the difference or variance in a given vital rate, for all interactions among vital rates—including higher‐order interactions neglected by the classical methods. We used the publicly available COMADRE and COMPADRE databases to perform a meta‐analysis comparing the results of exact and classical LTRE methods. We analysed 186 and 1487 LTREs for animal and plant matrix population models, respectively. We found that the classical methods often had small errors, but that very high errors were possible. Overall error was related to the difference or variance in the matrices being analysed, consistent with the Taylor series basis of the classical method. Neglected interaction terms accounted for most of the errors in fixed design LTRE, highlighting the importance of two‐way interaction terms. For random design LTRE, errors in the contribution terms present in both classical and exact methods were comparable to errors due to neglected interaction terms. In most examples we analysed, evaluating exact contributions up to three‐way interaction terms was sufficient for interpreting 90% or more of the difference or variance in . Relative error, previously used to evaluate the accuracy of classical LTREs, is not a reliable metric of how closely the classical and exact methods agree. Error compensation between estimated contribution terms and neglected contribution terms can lead to low relative error despite faulty biological interpretation. Trade‐offs or negative covariances among matrix elements can lead to high relative error despite accurate biological interpretation. Exact LTRE provides reliable and accurate biological interpretation, and the R package exactLTRE makes the exact method accessible to ecologists.
- 11 January 2023
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 38 (5) , 3308-3346
Approximation trees: statistical reproducibility in model distillation
- 11 January 2023
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 38 (5) , 3308-3346
Approximation trees: statistical reproducibility in model distillation
- Open Access
- 30 September 2022
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 36 (6) , 2237-2271
Decision tree boosted varying coefficient models
Varying coefficient models are a flexible extension of generic parametric models whose coefficients are functions of a set of effect-modifying covariates instead of fitted constants. They are capable of achieving higher model complexity while preserving the structure of the underlying parametric models, hence generating interpretable predictions. In this paper we study the use of gradient boosted decision trees as those coefficient-deciding functions in varying coefficient models with linearly structured outputs. In contrast to the traditional choices of splines or kernel smoothers, boosted trees are more flexible since they require no structural assumptions in the effect modifier space. We introduce our proposed method from the perspective of a localized version of gradient descent, prove its theoretical consistency under mild assumptions commonly adapted by decision tree research, and empirically demonstrate that the proposed tree boosted varying coefficient models achieve high performance qualified by their training speed, prediction accuracy and intelligibility as compared to several benchmark algorithms.
- 30 September 2022
- Published by Springer Nature in Data Mining and Knowledge Discovery
- Vol. 36 (6) , 2237-2271
Open AccessDecision tree boosted varying coefficient models
Varying coefficient models are a flexible extension of generic parametric models whose coefficients are functions of a set of effect-modifying covariates instead of fitted constants. They are capable of achieving higher model complexity while preserving the structure of the underlying parametric models, hence generating interpretable predictions. In this paper we study the use of gradient boosted decision trees as those coefficient-deciding functions in varying coefficient models with linearly structured outputs. In contrast to the traditional choices of splines or kernel smoothers, boosted trees are more flexible since they require no structural assumptions in the effect modifier space. We introduce our proposed method from the perspective of a localized version of gradient descent, prove its theoretical consistency under mild assumptions commonly adapted by decision tree research, and empirically demonstrate that the proposed tree boosted varying coefficient models achieve high performance qualified by their training speed, prediction accuracy and intelligibility as compared to several benchmark algorithms.
- 23 April 2022
- Published by Springer Nature in Statistics and Computing
- Vol. 32 (3) , 1-12
Selecting the derivative of a functional covariate in scalar-on-function regression
- 23 April 2022
- Published by Springer Nature in Statistics and Computing
- Vol. 32 (3) , 1-12
Selecting the derivative of a functional covariate in scalar-on-function regression
- Top Cited PapersOpen Access
- 29 October 2021
- Published by Springer Nature in Statistics and Computing
- Vol. 31 (6) , 1-16
Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance
This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling. In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects across various model setups and find support for previous claims in the literature that PaP metrics can vastly over-emphasize correlated features in both variable importance measures and partial dependence plots. As an alternative, we discuss and recommend more direct approaches that involve measuring the change in model performance after muting the effects of the features under investigation.
- 29 October 2021
- Published by Springer Nature in Statistics and Computing
- Vol. 31 (6) , 1-16
Top Cited PapersOpen AccessUnrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance
This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling. In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects across various model setups and find support for previous claims in the literature that PaP metrics can vastly over-emphasize correlated features in both variable importance measures and partial dependence plots. As an alternative, we discuss and recommend more direct approaches that involve measuring the change in model performance after muting the effects of the features under investigation.