flowSim: Near duplicate detection for flow cytometry data
Open Access
- 29 August 2023
- journal article
- research article
- Published by Wiley in Cytometry Part A
- Vol. 103 (11) , 889-901
Abstract
The analysis of large amounts of data is important for the development of machine learning (ML) models. flowSim is the first algorithm designed to visualize, detect and remove highly redundant information in flow cytometry (FCM) training sets to decrease the computational time for training and increase the performance of ML algorithms by reducing overfitting. flowSim performs near duplicate image detection by combining community detection algorithms with the density analysis of the marker expression values. flowSim clustering compared to consensus manual cluster-ing on a dataset composed of 160 images of bivariate FCM data had a mean Adjusted Rand Index of 0.90, demonstrating its efficiency in identifying similar patterns. flow-Sim selectively discarded near duplicate files in datasets constructed with known redundancy, and removed 92.6% of FCM images in a dataset of over 500,000 drawn from public repositories.Keywords
Funding Information
- National Institutes of Health (U19AI118608)
- Natural Sciences and Engineering Research Council of Canada
This publication has 35 references indexed in Scilit:
- ImmPort: disseminating data to the public for the future of immunologyImmunologic Research, 2014
- Chimeric antigen receptor modified T cell therapy for B cell malignanciesInternational Journal of Hematology, 2013
- Immune Monitoring in Cancer Vaccine Clinical Trials: Critical Issues of Functional Flow Cytometry-Based AssaysBioMed Research International, 2013
- Large Scale Near‐Duplicate Celebrity Web Images Retrieval Using Visual and Textual FeaturesThe Scientific World Journal, 2013
- EuroFlow standardization of flow cytometer instrument settings and immunophenotyping protocolsLeukemia, 2012
- FlowRepository: A resource of annotated flow cytometry datasets associated with peer‐reviewed publicationsCytometry Part A, 2012
- BASIL: Effective Near-Duplicate Image Detection Using Gene Sequence Alignment2010
- A Survey of Flow Cytometry Data Analysis MethodsAdvances in Bioinformatics, 2009
- Clustering by CompressionIEEE Transactions on Information Theory, 2005
- Vaccination in Humans Generates Broad T Cell Cytokine ResponsesThe Journal of Immunology, 2004