Katharina Schmidt

Virtual staining with generative adversarial networks is an efficient, non-invasive and scalable alternative to conventional cell staining, minimizing the need for destructive and time-consuming protocols. In this study, we investigate the explainability of a network trained to virtually stain osteoclast cultures, using intensity-based label-free input images. The model enables analysis of cell cultures without immunostaining. Explainability assessments, including receptive field and feature map analyses, show that the background in input images significantly influences staining predictions within cellular regions and the trained network performs an internal segmentation during the image transformation process. This suggests that contextual cues beyond cell boundaries are implicitly learned and integrated during training. By eliminating repetitive staining procedures, virtual staining enables longitudinal studies, allows multiplexing of individual samples, and reduces reagents and laboratory waste. Our findings enhance understanding of the virtual staining process and highlight its potential for biomedical research applications.

Fluorescence microscopy is the gold standard for investigation of biological samples. While powerful, this technology has its limitations, including phototoxicity, technical difficulties in introducing fluorescent markers, and limited simultaneous labeling of different structures due to the need for spectral separation. Here, artificial intelligence-driven virtual staining can provide label-free imaging. Deep neural networks are trained to learn the correlation between a label-free image and a ground-truth fluorescence image. However, the main challenge for the transition from traditional to virtual staining is the acquisition of huge datasets including labeled and unlabeled data. We have performed proof-of-concept experiments to investigate the possibilities of transfer learning in virtual staining. Therefore, U-Net architectures were pretrained on a larger dataset and later on trained again on a smaller and more specific dataset. We show that transfer learning decreases the needed dataset size and may even improve prediction quality. Furthermore, the interpretability of the trained networks was studied.This was investigated using guided backpropagation and modified input images.

The gold standard in histology is to use chemical stains or genetic modified tissue, where some internal structures emit a fluorescent signal. These methods require trained staff and several hours or days of preparation. Virtual staining employs trained neural networks to take over the staining process. Based on an unlabeled microscopic images the network can predict the corresponding fluorescent image for DAPI and Phalloidin488 staining, enabling studies on cell nuclei and the cytoskeleton. Neural networks usually need a huge amount of training data, so the possibilities of transfer learning for a reduction of the dataset size were investigated. In addition, we also present first studies the interpretability of the trained network to find ideal image acquisition techniques and optimize the training.

Chromatic aberrations can significantly diminish image quality when employing multiple wavelengths for imaging with a single optical system due to dispersion in both optical system and samples. To tackle this problem, we propose a novel approach using an adaptive achromatic lens, which is controlled by a trained Reinforcement Learning agent as part of a machine learning method. Notably, our method corrects chromatic aberrations prior to the imaging process, distinguishing it from conventional software-based post-processing approaches.

Various techniques in microscopy are based on point-wise acquisition, which provides advantages in acquiring sectioned images, for example in confocal or two-photon microscopy. The advantages come along with the need to perform three-dimensional scanning, which is often realized by mechanical movement achieved by stage-scanning or piezo-based scanning in the axial direction. Lateral scanning often employs galvo-mirrors, leading to a reflective setup and hence to a folded beam path. In this paper, we introduce a fully refractive microscope capable of three-dimensional scanning, which employs the combination of an adaptive lens, an adaptive prism, and a tailored telecentric f-theta objective. Our results show that this microscope is capable to perform flexible three-dimensional scanning, with low scan-induced aberrations, at a uniform resolution over a large tuning range of X=Y=6300 μm and Z=480 μm with only transmissive components. We demonstrate the capabilities at the example of volumetric measurements on the transgenic fluorescence of the thyroid of a zebrafish embryo and mixed pollen grains. This is the first step towards flexible aberration-free volumetric smart microscopy of three-dimensional samples like embryos and organoids, which could be exploited for the demands in both lateral and axial dimensions in biomedical samples without compromising image quality.

In fluorescence microscopy a multitude of labels are used that bind to different structures of biological samples. These often require excitation at different wavelengths and lead to different emission wavelengths. The presence of different wavelengths can induce chromatic aberrations, both in the optical system and induced by the sample. These lead to a detuning of the optical system, as the focal positions shift in a wavelength dependent manner and finally to a decrease in the spatial resolution. We present the correction of chromatic aberrations by using an electrical tunable achromatic lens driven by reinforcement learning. The tunable achromatic lens consists of two lens chambers filled with different optical oils and sealed with deformable glass membranes. By deforming the membranes of both chambers in a targeted manner, the chromatic aberrations present in the system can be manipulated to tackle both systematic and sample induced aberrations. We demonstrate chromatic aberration correction of up to 2200 mm and shift of the focal spot positions of 4000 mm. For control of this non-linear system with four input voltages, several reinforcement learning agents are trained and compared. The experimental results show that the trained agent can correct system and sample induced aberration and thereby improve the imaging quality, this is demonstrated using biomedical samples. In this case human thyroid was used for demonstration.

Adaptive lenses offer axial scanning without mechanical translation and thus are promising to replace mechanical-movement-based axial scanning in microscopy. The scan is accomplished by sweeping the applied voltage. However, the relation between the applied voltage and the resulting axial focus position is not unambiguous. Adaptive lenses suffer from hysteresis effects, and their behaviour depends on environmental conditions. This is especially a hurdle when complex adaptive lenses are used that offer additional functionalities and are controlled with more degrees of freedom. In such case, a common approach is to iterate the voltage and monitor the adaptive lens. Here, we introduce an alternative approach which provides a single shot estimation of the current axial focus position by a convolutional neural network. We use the experimental data of our custom confocal microscope for training and validation. This leads to fast scanning without photo bleaching of the sample and opens the door to automatized and aberration-free smart microscopy. Applications in different types of laser-scanning microscopes are possible. However, maybe the training procedure of the neural network must be adapted for some use cases.