Anisa Idris

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). N/A

Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). We have several comments on the recent publication of [1], in which repeated testing of four symptom assessment applications with clinical vignettes was carried out to look for “hints of ‘non-locked learning algorithms’”. As the developer of one of the symptom assessment applications studied by [1], we are supportive of studies evaluating app performance, however there are important limitations in the methodology of the study. Most importantly, the methodology used in this study is not capable of addressing its main objective. The approach used to look for evidence of non-locked algorithms was the quantification of differences in performance using three ophthalmology vignettes, first in 2018 then in 2020. This methodology, although highly limited due to the use of only three vignettes in one medical specialism, could be used to detect changes in app performance over time. It however cannot be used to distinguish between non-locked algorithms and the manual updating of the apps’ medical intelligence, through the normal process of manual release of updated app versions.