Ali Nemati
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Ali Nemati has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Anmar Abdul-Rahman
Department of Ophthalmology, Counties Manukau District Health Board, Auckland, New Zealand
1 shared publication - Hamid GholamHosseini
School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology, Auckland 1010, New Zealand
1 shared publication - Jake Luo
Health Informatics Program, Zilber School of Public Health, University of Wisconsin-Milwaukee, Milwaukee, USA
1 shared publication - Mohammad Assadi Shalmani
Health Informatics Program, Zilber School of Public Health, University of Wisconsin-Milwaukee, Milwaukee, USA
1 shared publication - Mohammad Norouzifard
Auckland Bioengineering Institute, The University of Auckland, Auckland 1142, New Zealand
1 shared publication - Qiang Lu
College of Artificial Intelligence, China University of Petroleum, Beijing, China
1 shared publication
Latest Publications
- Open Access
- 24 March 2025
- Published by MDPI AG in Future Internet
- Vol. 17 (4) , 138
Benchmarking Large Language Models from Open and Closed Source Models to Apply Data Annotation for Free-Text Criteria in Healthcare
Large language models (LLMs) hold the potential to significantly enhance data annotation for free-text healthcare records. However, ensuring their accuracy and reliability is critical, especially in clinical research applications requiring the extraction of patient characteristics. This study introduces a novel evaluation framework based on Multi-Criteria Decision Analysis (MCDA) and the Order of Preference by Similarity to Ideal Solution (TOPSIS) technique, designed to benchmark LLMs on their annotation quality. The framework defines ten evaluation metrics across key criteria such as age, gender, BMI, disease presence, and blood markers (e.g., white blood count and platelets). Using this methodology, we assessed leading open source and commercial LLMs, achieving accuracy scores of 0.59, 1, 0.84, 0.56, and 0.92, respectively, for the specified criteria. Our work not only provides a rigorous framework for evaluating LLM capabilities in healthcare data annotation but also highlights their current performance limitations and strengths. By offering a comprehensive benchmarking approach, we aim to support responsible adoption and decision-making in healthcare applications.
- 24 March 2025
- Published by MDPI AG in Future Internet
- Vol. 17 (4) , 138
Open AccessBenchmarking Large Language Models from Open and Closed Source Models to Apply Data Annotation for Free-Text Criteria in Healthcare
Large language models (LLMs) hold the potential to significantly enhance data annotation for free-text healthcare records. However, ensuring their accuracy and reliability is critical, especially in clinical research applications requiring the extraction of patient characteristics. This study introduces a novel evaluation framework based on Multi-Criteria Decision Analysis (MCDA) and the Order of Preference by Similarity to Ideal Solution (TOPSIS) technique, designed to benchmark LLMs on their annotation quality. The framework defines ten evaluation metrics across key criteria such as age, gender, BMI, disease presence, and blood markers (e.g., white blood count and platelets). Using this methodology, we assessed leading open source and commercial LLMs, achieving accuracy scores of 0.59, 1, 0.84, 0.56, and 0.92, respectively, for the specified criteria. Our work not only provides a rigorous framework for evaluating LLM capabilities in healthcare data annotation but also highlights their current performance limitations and strengths. By offering a comprehensive benchmarking approach, we aim to support responsible adoption and decision-making in healthcare applications.