Neural supersampling for real-time rendering
Authors:
Open Access
- 12 August 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 39 (4)
Abstract
Due to higher resolutions and refresh rates, as well as more photorealistic effects, real-time rendering has become increasingly challenging for video games and emerging virtual reality headsets. To meet this demand, modern graphics hardware and game engines often reduce the computational cost by rendering at a lower resolution and then upsampling to the native resolution. Following the recent advances in image and video superresolution in computer vision, we propose a machine learning approach that is specifically tailored for high-quality upsampling of rendered content in real-time applications. The main insight of our work is that in rendered content, the image pixels are point-sampled, but precise temporal dynamics are available. Our method combines this specific information that is typically available in modern renderers (i.e., depth and dense motion vectors) with a novel temporal network design that takes into account such specifics and is aimed at maximizing video quality while delivering real-time performance. By training on a large synthetic dataset rendered from multiple 3D scenes with recorded camera motion, we demonstrate high fidelity and temporally stable results in real-time, even in the highly challenging 4 × 4 upsampling scenario, significantly outperforming existing superresolution and temporal antialiasing work.Keywords
This publication has 28 references indexed in Scilit:
- Burst photography for high dynamic range and low-light imaging on mobile camerasACM Transactions on Graphics, 2016
- Towards foveated rendering for gaze-tracked virtual realityACM Transactions on Graphics, 2016
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Accurate Image Super-Resolution Using Very Deep Convolutional Networks2016
- Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network2016
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Amortized supersamplingACM Transactions on Graphics, 2009
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Reality Engine graphics1993
- Pyramidal parametrics1983