Liyun Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Liyun Zhang has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Dian Ding
Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Feitong Tan
School of Computing Science, Simon Fraser University, Burnaby, BC, Canada
1 shared publication - Guangtao Xue
Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Hao Pan
Shanghai Jiao Tong University, Shanghai, China
1 shared publication - Ju Ren
Tsinghua University
1 shared publication - Ran Wang
Shanghai Jiao Tong University, Shanghai, China
1 shared publication
Latest Publications
- Open Access
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 744-756
M3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile Cameras
The demand for ultra-high-resolution imaging in mobile phone photography is continuously increasing. However, the image resolution of mobile devices is typically constrained by the size of the CMOS sensor. Although deep learning-based super-resolution (SR) techniques have the potential to overcome this limitation, existing SR neural network models require large computational resources, making them unsuitable for real-time SR imaging on current mobile devices. Additionally, cloud-based SR systems pose privacy leakage risks. In this paper, we propose M3Cam, an innovative and lightweight SR imaging system for mobile phones. M3Cam can ensure high-quality 16× SR image (4× in both height and width) visualization with almost negligible latency. In detail, we utilize an optical image stabilization (OIS) module for lens control and introduce a new modality of data, namely gyroscope readings, to achieve high-precision and compact optical flow estimation modules. Building upon this concept, we design a multi-frame-based SR model utilizing the Swin Transformer. Our proposed system can generate a 16× SR image from four captured low-resolution images in real-time, with low computational load, low inference latency, and minimal reliance on runtime RAM. Through extensive experiments, we demonstrate that our proposed multi-modal optical flow model significantly enhances pixel alignment accuracy between multiple frames and delivers outstanding 16× SR imaging results under various shooting scenarios. Code and dataset are available at: https://github.com/liangjindeamo-yuer/M3CAM
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 744-756
Open AccessM3Cam: Extreme Super-resolution via Multi-Modal Optical Flow for Mobile Cameras
The demand for ultra-high-resolution imaging in mobile phone photography is continuously increasing. However, the image resolution of mobile devices is typically constrained by the size of the CMOS sensor. Although deep learning-based super-resolution (SR) techniques have the potential to overcome this limitation, existing SR neural network models require large computational resources, making them unsuitable for real-time SR imaging on current mobile devices. Additionally, cloud-based SR systems pose privacy leakage risks. In this paper, we propose M3Cam, an innovative and lightweight SR imaging system for mobile phones. M3Cam can ensure high-quality 16× SR image (4× in both height and width) visualization with almost negligible latency. In detail, we utilize an optical image stabilization (OIS) module for lens control and introduce a new modality of data, namely gyroscope readings, to achieve high-precision and compact optical flow estimation modules. Building upon this concept, we design a multi-frame-based SR model utilizing the Swin Transformer. Our proposed system can generate a 16× SR image from four captured low-resolution images in real-time, with low computational load, low inference latency, and minimal reliance on runtime RAM. Through extensive experiments, we demonstrate that our proposed multi-modal optical flow model significantly enhances pixel alignment accuracy between multiple frames and delivers outstanding 16× SR imaging results under various shooting scenarios. Code and dataset are available at: https://github.com/liangjindeamo-yuer/M3CAM