Silke Janitza

The Strengths and Difficulties Questionnaire (SDQ) is the most widely used mental health screening instrument for children and adolescents. It is a short questionnaire including 25 items that can be answered by parents, teachers or children. There are two studies which report norms for the German SDQ parent version. They do not include children younger than 6 years. Moreover, whether the German SDQ parent version is measurement invariant across age has not yet been investigated. The absence of measurement invariance across age would support the use of age-specific norms that are not yet available for the German SDQ parent version. We used data of the German Health Interview and Examination Survey for Children and Adolescents (KiGGS), a nationally representative survey including 14,835 children aged 3–17 years, to assess measurement invariance of the German SDQ parent version across the full age range. Multi-group confirmatory factor analysis revealed that the hyperactivity and emotional symptoms subscales are not comparable between children of different ages. This supports the use of age-specific norms for these two subscales and for the total SDQ. We used methods of centile estimation to smoothly model the centiles of the SDQ total score and the subscale scores in dependence on age. These age-specific centiles reflect the developmental course of SDQ problems in children (including preschoolers) and adolescents living in Germany. They can be used to identify children and adolescents with abnormal behaviour, while accounting for the developmental course of emotional and behaviour problems.

Although an “epidemic” of depression is frequently claimed, empirical evidence is inconsistent, depending on country, study design and depression assessment. Little is known about changes in depression over time in Germany, although health insurance companies report frequency increases. Here we examined time trends in depression prevalence, severity and health-related correlates in the general population. Data were obtained from the mental health module of the “German Health Interview and Examination Survey for Adults” (2009–2012, n = 3265) and the mental health supplement of the “German National Health Interview and Examination Survey 1998” (1997–1999, n = 4176), excluding respondents older than 65. 12-month major depressive disorder (MDD), severity and symptoms were assessed based on the WHO Composite International Diagnostic Interview. Health-related quality of life (SF-36), self-reported sick days or days with limitations in normal daily life activities were examined, too. Calculations were carried out population-weighted. Additional age-standardized analyses were conducted to account for demographic changes. Overall, MDD 12-month prevalence remained stable at 7.4%. Women showed a shifted age distribution with increased prevalence at younger ages, and increasing MDD severity. Time trends in health-related correlates occurred both in participants with and without MDD. Mental health disability increased over time, particularly among men with MDD, reflected by the mental component score of the SF-36 and days with activity limitation due to mental health problems. Demographic changes had a marginal impact on the time trends. In contrast to the ongoing international debate regarding increased depression rates in western countries, we found no increase in overall MDD prevalence in Germany over a long period. In conclusion, increased depression frequencies in national health insurance data and growing health care costs associated with depression are not attributable to overall prevalence changes at a population level. However, shifted age distribution and increased severity among women may reflect a rising depression risk within this specific subgroup, and changes in health-related correlates indicate a growing mental health care need for depression, particularly among men.

The ensemble method random forests has become a popular classification tool in bioinformatics and related fields. The out-of-bag error is an error estimation technique often used to evaluate the accuracy of a random forest and to select appropriate values for tuning parameters, such as the number of candidate predictors that are randomly drawn for a split, referred to as mtry. However, for binary classification problems with metric predictors it has been shown that the out-of-bag error can overestimate the true prediction error depending on the choices of random forests parameters. Based on simulated and real data this paper aims to identify settings for which this overestimation is likely. It is, moreover, questionable whether the out-of-bag error can be used in classification tasks for selecting tuning parameters like mtry, because the overestimation is seen to depend on the parameter mtry. The simulation-based and real-data based studies with metric predictor variables performed in this paper show that the overestimation is largest in balanced settings and in settings with few observations, a large number of predictor variables, small correlations between predictors and weak effects. There was hardly any impact of the overestimation on tuning parameter selection. However, although the prediction performance of random forests was not substantially affected when using the out-of-bag error for tuning parameter selection in the present studies, one cannot be sure that this applies to all future data. For settings with metric predictor variables it is therefore strongly recommended to use stratified subsampling with sampling fractions that are proportional to the class sizes for both tuning parameter selection and error estimation in random forests. This yielded less biased estimates of the true prediction error. In unbalanced settings, in which there is a strong interest in predicting observations from the smaller classes well, sampling the same number of observations from each class is a promising alternative.

Ideally, prediction rules should be published in such a way that readers may apply them, for example, to make predictions for their own data. While this is straightforward for simple prediction rules, such as those based on the logistic regression model, this is much more difficult for complex prediction rules derived by machine learning tools. We conducted a survey of articles reporting prediction rules that were constructed using the random forest algorithm and published in PLOS ONE in 2014–2015 in the field “medical and health sciences”, with the aim of identifying issues related to their applicability. Making a prediction rule reproducible is a possible way to ensure that it is applicable; thus reproducibility is also examined in our survey. The presented prediction rules were applicable in only 2 of 30 identified papers, while for further eight prediction rules it was possible to obtain the necessary information by contacting the authors. Various problems, such as nonresponse of the authors, hampered the applicability of prediction rules in the other cases. Based on our experiences from this illustrative survey, we formulate a set of recommendations for authors who aim to make complex prediction rules applicable for readers. All data including the description of the considered studies and analysis codes are available as supplementary materials.

Automated variable selection procedures, such as backward elimination, are commonly employed to perform model selection in the context of multivariable regression. The stability of such procedures can be investigated using a bootstrap-based approach. The idea is to apply the variable selection procedure on a large number of bootstrap samples successively and to examine the obtained models, for instance, in terms of the inclusion of specific predictor variables. In this paper, we aim to investigate a particular important problem affecting this method in the case of categorical predictor variables with different numbers of categories and to give recommendations on how to avoid it. For this purpose, we systematically assess the behavior of automated variable selection based on the likelihood ratio test using either bootstrap samples drawn with replacement or subsamples drawn without replacement from the original dataset. Our study consists of extensive simulations and a real data example from the NHANES study. Our main result is that if automated variable selection is conducted on bootstrap samples, variables with more categories are substantially favored over variables with fewer categories and over metric variables even if none of them have any effect. Importantly, variables with no effect and many categories may be (wrongly) preferred to variables with an effect but few categories. We suggest the use of subsamples instead of bootstrap samples to bypass these drawbacks.

Identification of microorganisms in positive blood cultures still relies on standard techniques such as Gram staining followed by culturing with definite microorganism identification. Alternatively, matrix-assisted laser desorption/ionization time-of-flight mass spectrometry or the analysis of headspace volatile compound (VC) composition produced by cultures can help to differentiate between microorganisms under experimental conditions. This study assessed the efficacy of volatile compound based microorganism differentiation into Gram-negatives and -positives in unselected positive blood culture samples from patients. Headspace gas samples of positive blood culture samples were transferred to sterilized, sealed, and evacuated 20 ml glass vials and stored at −30 °C until batch analysis. Headspace gas VC content analysis was carried out via an auto sampler connected to an ion–molecule reaction mass spectrometer (IMR-MS). Measurements covered a mass range from 16 to 135 u including CO2, H2, N2, and O2. Prediction rules for microorganism identification based on VC composition were derived using a training data set and evaluated using a validation data set within a random split validation procedure. One-hundred-fifty-two aerobic samples growing 27 Gram-negatives, 106 Gram-positives, and 19 fungi and 130 anaerobic samples growing 37 Gram-negatives, 91 Gram-positives, and two fungi were analysed. In anaerobic samples, ten discriminators were identified by the random forest method allowing for bacteria differentiation into Gram-negative and -positive (error rate: 16.7 % in validation data set). For aerobic samples the error rate was not better than random. In anaerobic blood culture samples of patients IMR-MS based headspace VC composition analysis facilitates bacteria differentiation into Gram-negative and -positive.

The bootstrap method has become a widely used tool applied in diverse areas where results based on asymptotic theory are scarce. It can be applied, for example, for assessing the variance of a statistic, a quantile of interest or for significance testing by resampling from the null hypothesis. Recently, some approaches have been proposed in the biometrical field where hypothesis testing or model selection is performed on a bootstrap sample as if it were the original sample. P-values computed from bootstrap samples have been used, for example, in the statistics and bioinformatics literature for ranking genes with respect to their differential expression, for estimating the variability of p-values and for model stability investigations. Procedures which make use of bootstrapped information criteria are often applied in model stability investigations and model averaging approaches as well as when estimating the error of model selection procedures which involve tuning parameters. From the literature, however, there is evidence that p-values and model selection criteria evaluated on bootstrap data sets do not represent what would be obtained on the original data or new data drawn from the overall population. We explain the reasons for this and, through the use of a real data set and simulations, we assess the practical impact on procedures relevant to biometrical applications in cases where it has not yet been studied. Moreover, we investigate the behavior of subsampling (i.e., drawing from a data set without replacement) as a potential alternative solution to the bootstrap for these procedures.

Summary: In recent years, increasing attention has been devoted to the problem of the stability of multivariable regression models, understood as the resistance of the model to small changes in the data on which it has been fitted. Resampling techniques, mainly based on the bootstrap, have been developed to address this issue. In particular, the approaches based on the idea of “inclusion frequency” consider the repeated implementation of a variable selection procedure, for example backward elimination, on several bootstrap samples. The analysis of the variables selected in each iteration provides useful information on the model stability and on the variables’ importance. Recent findings, nevertheless, show possible pitfalls in the use of the bootstrap, and alternatives such as subsampling have begun to be taken into consideration in the literature. Using model selection frequencies and variable inclusion frequencies, we empirically compare these two different resampling techniques, investigating the effect of their use in selected classical model selection procedures for multivariable regression. We conduct our investigations by analyzing two real data examples and by performing a simulation study. Our results reveal some advantages in using a subsampling technique rather than the bootstrap in this context.