Markus Toews
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Markus Toews has a total of 12 co-authors. Here are the top co-authors by number of shared publications:
- Angelika Eisele-Metzger
Institute for Evidence in Medicine, Medical Center – University of Freiburg, Faculty of Medicine, University of Freiburg
3 shared publications - Joerg Johannes Meerpohl
University of Freiburg
3 shared publications - Waldemar Siemens
Institute for Evidence in Medicine, Medical Center - University of Freiburg/Medical Faculty - University of Freiburg, Freiburg, Germany
3 shared publications - Christian Haverkamp
Institute of Digitalization in Medicine, Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany
2 shared publications - Daniel Böhringer
Eye Center, University Hospital Freiburg, Freiburg, Germany
2 shared publications - Felix Heilmeyer
Affiliation Institute for Digitalization in Medicine, University Medical Center Freiburg, Freiburg, Germany
2 shared publications
Latest Publications
- Top Cited PapersOpen Access
- 26 February 2025
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 181, 111746
Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review
- 26 February 2025
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 181, 111746
Top Cited PapersOpen AccessLarge language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review
- Open Access
Large language models for conducting systematic reviews: on the rise, but not yet ready for use – a scoping review
Background Machine learning (ML) promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. Objective To provide an overview of ML and specifically LLM applications in SR conduct in health research. Study design We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: 26 February 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review with a related research question. Two reviewers independently screened studies for eligibility; after piloting, one reviewer extracted data, checked by another. Results Our database search yielded 8054 hits, and we identified 33 articles from our hand search. Of the 196 included reports, 159 described more traditional ML techniques, 37 focused on LLMs. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n=15, 41%), study selection (n=14, 38%), and data extraction (n=11, 30%). The mostly recurring LLM was GPT (n=33, 89%). Validation studies were predominant (n=21, 57%). In half of the studies, authors evaluated LLM use as promising (n=20, 54%), one quarter as neutral (n=9, 24%) and one fifth as non-promising (n=8, 22%). Conclusions Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance. HIGHLIGHTS Machine learning (ML) offers promising support for systematic review (SR) creation. GPT was the most commonly used large language model (LLM) to support SR production. LLM application included 10 of 13 defined SR steps, most often literature search. Validation studies predominated, but fully established LLM applications are rare. LLM research for SR conduct is surging, highlighting the increasing relevance.
Open AccessLarge language models for conducting systematic reviews: on the rise, but not yet ready for use – a scoping review
Background Machine learning (ML) promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. Objective To provide an overview of ML and specifically LLM applications in SR conduct in health research. Study design We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: 26 February 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review with a related research question. Two reviewers independently screened studies for eligibility; after piloting, one reviewer extracted data, checked by another. Results Our database search yielded 8054 hits, and we identified 33 articles from our hand search. Of the 196 included reports, 159 described more traditional ML techniques, 37 focused on LLMs. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n=15, 41%), study selection (n=14, 38%), and data extraction (n=11, 30%). The mostly recurring LLM was GPT (n=33, 89%). Validation studies were predominant (n=21, 57%). In half of the studies, authors evaluated LLM use as promising (n=20, 54%), one quarter as neutral (n=9, 24%) and one fifth as non-promising (n=8, 22%). Conclusions Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance. HIGHLIGHTS Machine learning (ML) offers promising support for systematic review (SR) creation. GPT was the most commonly used large language model (LLM) to support SR production. LLM application included 10 of 13 defined SR steps, most often literature search. Validation studies predominated, but fully established LLM applications are rare. LLM research for SR conduct is surging, highlighting the increasing relevance.
- Open Access
Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2
Systematic reviews are essential for evidence based healthcare, but conducting them is time and resource consuming. To date, efforts have been made to accelerate and (semi-) automate various steps of systematic reviews through the use of artificial intelligence and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for risk of bias assessment of 100 randomized controlled trials using the revised Cochrane risk of bias tool (“RoB 2”; involving judgements for five specific domains and an overall judgement). We assessed the agreement of risk of bias judgements by Claude with human judgements published in Cochrane Reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 (“outcome measurement”). Cohen’s κ was lowest for domain 5 (“selective reporting”; 0.10 (95% confidence interval (CI): −0.10-0.31)) and highest for domain 3 (“missing data”; 0.31 (95% CI: 0.10-0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen’s κ: 0.22 (95% CI: 0.06-0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude’s RoB 2 judgements cannot replace human risk of bias assessment. However, the potential of LLMs to support risk of bias assessment should be further explored.
Open AccessExploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2
Systematic reviews are essential for evidence based healthcare, but conducting them is time and resource consuming. To date, efforts have been made to accelerate and (semi-) automate various steps of systematic reviews through the use of artificial intelligence and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for risk of bias assessment of 100 randomized controlled trials using the revised Cochrane risk of bias tool (“RoB 2”; involving judgements for five specific domains and an overall judgement). We assessed the agreement of risk of bias judgements by Claude with human judgements published in Cochrane Reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 (“outcome measurement”). Cohen’s κ was lowest for domain 5 (“selective reporting”; 0.10 (95% confidence interval (CI): −0.10-0.31)) and highest for domain 3 (“missing data”; 0.31 (95% CI: 0.10-0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen’s κ: 0.22 (95% CI: 0.06-0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude’s RoB 2 judgements cannot replace human risk of bias assessment. However, the potential of LLMs to support risk of bias assessment should be further explored.