Semantic–Electromagnetic Inversion With Pretrained Multimodal Generative Model
Open Access
- 9 September 2024
- journal article
- research article
- Published by Wiley in Advanced Science
- Vol. 11 (42) , e2406793
Abstract
Across diverse domains of science and technology, electromagnetic (EM) inversion problems benefit from the ability to account for multimodal prior information to regularize their inherent ill‐posedness. Indeed, besides priors that are formulated mathematically or learned from quantitative data, valuable prior information may be available in the form of text or images. Besides handling semantic multimodality, it is furthermore important to minimize the cost of adapting to a new physical measurement operator and to limit the requirements for costly labeled data. Here, these challenges are tackled with a frugal and multimodal semantic–EM inversion technique. The key ingredient is a multimodal generator of reconstruction results that can be pretrained, being agnostic to the physical measurement operator. The generator is fed by a multimodal foundation model encoding the multimodal semantic prior and a physical adapter encoding the measured data. For a new physical setting, only the lightweight physical adapter is retrained. The authors’ architecture also enables a flexible iterative step‐by‐step solution to the inverse problem where each step can be semantically controlled. The feasibility and benefits of this methodology are demonstrated for three EM inverse problems: a canonical two‐dimensional inverse‐scattering problem in numerics, as well as three‐dimensional and four‐dimensional compressive microwave meta‐imaging experiments.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFA1401002, 2023YFB3811502)
- National Natural Science Foundation of China (62288101)
This publication has 35 references indexed in Scilit:
- Semantic regularization of electromagnetic inverse problemsNature Communications, 2024
- Deep generative molecular design reshapes drug discoveryCell Reports Medicine, 2022
- High‐Throughput Discovery of Novel Cubic Crystal Materials Using Deep Generative Neural NetworksAdvanced Science, 2021
- Phase imaging with an untrained neural networkLight: Science & Applications, 2020
- Intelligent Electromagnetic Sensing with Learnable Data Acquisition and ProcessingPatterns, 2020
- Learned Integrated Sensing Pipeline: Reconfigurable Metasurface Transceivers as Trainable Physical Layer in an Artificial Neural NetworkAdvanced Science, 2019
- Solving inverse problems using data-driven modelsActa Numerica, 2019
- Machine-learning reprogrammable metasurface imagerNature Communications, 2019
- Bayes' theoremNature Methods, 2015
- Time‐lapse imaging of saline‐tracer transport in fractured rock using difference‐attenuation radar tomographyWater Resources Research, 2003