Qiang Fu

Quantitative, volumetric imaging of cerebrovascular networks and microcirculation is essential for understanding brain function. However, rapid mesoscopic 3D imaging remains challenging because of fundamental trade-offs between spatiotemporal resolution, field of view, and sensitivity to functional parameters. Here we present a mesoscopic fluorescence imaging platform featuring a double-helix phase mask for real-time, depth-resolved measurements through the intact mouse skull. The compact phase-mask design is compatible with both laser-scanning and widefield microscopy. Using multifocal laser scanning, we demonstrate real-time volumetric in vivo imaging while discriminating calvarial from cerebral vasculature across 6.6×6.6×0.8 mm ³ volume. Beyond high-resolution structural imaging, perfusion time-to-peak values are extracted from the laser-scanning configuration while accurate flow velocity/direction information is provided via widefield tracking of fluorescently labeled cells. We demonstrate the platform’s capabilities by analyzing brain-layer-specific perfusion dynamics and vascular topology in glioma-bearing mouse brains, offering unprecedented views for probing cerebrovascular alterations in both physiological and pathological contexts.

Classical lens design minimizes optical aberrations to produce sharp images, but is typically decoupled from downstream computer vision tasks. Existing end-to-end optical design learns optical encoding through joint optimization, but often suffers from an unstable training process. We propose task-driven lens design, what we believe to be a new optimization philosophy for joint optics-network systems. We freeze the pretrained vision model and optimize only the lens so that the image formation better fits the model’s feature preferences. This network-frozen setting yields a low-dimensional and stable optimization process, enabling lens design from scratch without human intervention, thereby exploring a broader design space. Multiple computer vision experiments show that TaskLenses outperform classical ImagingLenses with the same or even fewer elements. Our analysis reveals that the learned optics exhibit long-tailed point spread functions, better preserving preferred structural cues when aberrations cannot be fully corrected. These results highlight task-driven design as a practical route for optical lenses that are compatible with modern vision models, and also inspire believed to be new optical design objectives beyond traditional aberration minimization.

Differentiable optics, as an emerging paradigm that jointly optimizes optics and (optional) image processing algorithms, has made many innovative optical designs possible across a broad range of imaging and display applications. Many of these systems utilize diffractive optical components for holography, PSF engineering, or wavefront shaping. Existing approaches have, however, mostly remained limited to laboratory prototypes, owing to a large quality gap between simulation and manufactured devices. We aim at lifting the fundamental technical barriers to the practical use of learned diffractive optical systems. To this end, we propose a fabrication-aware design pipeline for diffractive optics fabricated by direct-write grayscale lithography followed by replication with nano-imprinting, which is directly suited for inexpensive mass-production of large area designs. We propose a super-resolved neural lithography model that can accurately predict the 3D geometry generated by the fabrication process. This model can be seamlessly integrated into existing differentiable optics frameworks, enabling fabrication-aware, end-to-end optimization of computational optical systems. To tackle the computational challenges, we also devise tensor-parallel compute framework centered on distributing large-scale FFT computation across many GPUs. As such, we demonstrate large scale diffractive optics designs up to 32.16 mm × 21.44 mm, simulated on grids of up to 128,640 by 85,760 feature points. We find adequate agreement between simulation and fabricated prototypes for applications such as holography and PSF engineering. We also achieve high image quality from an imaging system comprised only of a single diffractive optical element, with images processed only by a one-step inverse filter utilizing the simulation PSF. We believe our findings lift the fabrication limitations for real-world applications of diffractive optics and differentiable optical design.

Hyperspectral imaging empowers machine vision systems with the distinct capability of identifying materials through recording their spectral signatures. Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images captured by cost-effective RGB cameras, instead of dedicated hardware. Published work reports exceedingly high numerical scores for this reconstruction task, yet real-world performance lags substantially behind. In this paper we systematically analyze the performance of such methods with three groups of dedicated experiments. First, we evaluate the practical overfitting limitations with respect to current datasets by training the networks with less data, validating the trained models with unseen yet slightly modified data, and cross-dataset validation. Second, we reveal fundamental limitations in the ability of RGB to spectral methods to deal with metameric or near-metameric conditions, which have so far gone largely unnoticed due to the insufficiencies of existing datasets. We achieve this by validating the trained models with metamer data generated by metameric black theory and re-training the networks with various forms of metamers. This methodology can also be used for data augmentation as a partial mitigation of the dataset issues, although the RGB to spectral inverse problem remains fundamentally ill-posed. Finally, we analyze the potential for modifying the problem setting to achieve better performance by exploiting some form of optical encoding provided by either incidental optical aberrations or some form of deliberate optical design. Our experiments show that such approaches do indeed provide improved results under certain circumstances, however their overall performance is limited by the same dataset issues as in the plain RGB to spectral scenario. We therefore conclude that future progress on snapshot spectral imaging will heavily depend on the generation of improved datasets which can then be used to design effective optical encoding strategies.

Differentiable optical design builds optical simulations using automatic differentiation frameworks, enabling the automated and accurate computation of gradients for all optical parameters via a single backpropagation step. This approach fully leverages AI-driven optimization algorithms and GPU parallel acceleration. Compared to classical optical design methods, differentiable optical design offers numerous advantages, including automated optimization and stable convergence, demonstrating its potential to become the foundation for the next generation of optical design software.

End-to-end (E2E) designed imaging systems integrate coded optical designs with decoding algorithms to enhance imaging fidelity for diverse visual tasks. However, existing E2E designs encounter significant challenges in maintaining high image fidelity at wide fields of view, due to high computational complexity, as well as difficulties in modeling off-axis wave propagation while accounting for off-axis aberrations. In particular, the common approach of placing the encoding element into the aperture or pupil plane results in only a global control of the wavefront. To overcome these limitations, this work explores an additional design choice by positioning a DOE off-aperture, enabling a spatial unmixing of the degrees of freedom and providing local control over the wavefront over the image plane. Our approach further leverages hybrid refractive-diffractive optical systems by linking differentiable ray and wave optics modeling, thereby optimizing depth imaging quality and demonstrating system versatility. Experimental results reveal that the off-aperture DOE enhances the imaging quality by over 5 dB in PSNR at a FoV of approximately 45° when paired with a simple thin lens, outperforming traditional on-aperture systems. Furthermore, we successfully recover color and depth information at nearly 28° FoV using off-aperture DOE configurations with compound optics. Physical prototypes for both applications validate the effectiveness and versatility of the proposed method.

Neural rendering techniques, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D‐GS), have led to significant advancements in scene reconstruction and novel view synthesis (NVS). These methods assume the use of an ideal pinhole model, which is free from lens distortion and optical aberrations. However, fisheye lenses introduce unavoidable aberrations due to their wide‐angle design and complex manufacturing, leading to multi‐view inconsistencies that compromise scene reconstruction quality. In this paper, we propose an end‐to‐end framework that integrates a standard 3D reconstruction pipeline with our lens aberration model to simultaneously calibrate lens aberrations and reconstruct 3D scenes. By modelling the real imaging process and jointly optimising both tasks, our framework eliminates the impact of aberration‐induced inconsistencies on reconstruction. Additionally, we propose a curriculum learning approach that ensures stable optimisation and high‐quality reconstruction results, even in the presence of multiple aberrations. To address the limitations of existing benchmarks, we introduce AbeRec, a dataset composed of scenes captured with lenses exhibiting severe aberrations. Extensive experiments on both existing public datasets and our proposed dataset demonstrate that our method not only significantly outperforms previous state‐of‐the‐art methods on fisheye lenses with severe aberrations but also generalises well to scenes captured by non‐fisheye lenses. Code and datasets are available at https://github.com/CPREgroup/Calibrating‐Fisheye‐Lens‐Aberration‐for‐NVS.

This paper proposes and demonstrates a linearized model for phase diversity wavefront sensing, facilitating real-time processing and much less data required for training. Specifically, we find that the low-frequency Fourier coefficients of point spread function images are linearly proportional to pupil aberration coefficients under certain conditions. Simulation and experimental results show that the model can greatly reduce the processing time to several milliseconds by merely requiring hundreds of training samples while maintaining a comparatively high accuracy with state-of-the-art methods.

Compressive spectral imaging (CSI) is a technique used to capture high-dimensional hyperspectral images (HSIs) with a few multiplexed measurements, thereby reducing data acquisition costs and complexity. However, existing CSI methods often rely on end-to-end learning from training sets, which may struggle to generalize well to unseen scenes and phenomena. In this paper, we present a progressive self-supervised method specifically tailored for coded aperture snapshot spectral imaging (CASSI). Our proposed method enables HSI reconstruction solely from the measurements, without requiring any ground truth spectral data. To achieve this, we integrate positional encoding and spectral cluster-centroid features within a novel progressive training framework. Additionally, we employ an attention mechanism and a multi-scale architecture to enhance the robustness and accuracy of HSI reconstruction. Through extensive experiments on both synthetic and real datasets, we validate the effectiveness of our method. Our results demonstrate significantly superior performance compared to state-of-the-art self-supervised CASSI methods, while utilizing fewer parameters and consuming less memory. Furthermore, our proposed approach showcases competitive performance in terms of reconstruction quality when compared to state-of-the-art supervised methods. The related code and data are available at https://github.com/ccccddd1/ceinr .