Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging
Open Access
- 8 February 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-13
Abstract
Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32 $$\times $$ 32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.
Keywords
All Related Versions
This publication has 51 references indexed in Scilit:
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Fluorescence lifetime imaging – techniques and applicationsJournal of Microscopy, 2012
- Video-rate fluorescence lifetime imaging camera with CMOS single-photon avalanche diode arrays and high-speed imaging algorithmJournal of Biomedical Optics, 2011
- Real-time fluorescence lifetime imaging system with a 32 × 32 013μm CMOS low dark-count single-photon avalanche diode arrayOptics Express, 2010
- Imaging protein molecules using FRET and FLIM microscopyCurrent Opinion in Biotechnology, 2005
- Long Short-Term MemoryNeural Computation, 1997
- How many photons are necessary for fluorescence-lifetime measurements?Chemical Physics Letters, 1992
- Maximum likelihood method for the analysis of time-resolved fluorescence decay curvesEuropean Biophysics Journal, 1991
- On the analysis of fluorescence decay kinetics by the method of least-squaresAnalytical Biochemistry, 1974
- The Analysis of Fluorescence Decay by a Method of MomentsBiophysical Journal, 1969