Theoretical vibrational spectroscopy methods are often necessary to interpret condensed-phase spectra. For some methods, such as those that use vibrational spectroscopic maps, the requisite atomistic molecular dynamics simulations can become prohibitively expensive for systems of interest. Here, we present a two-stage method for predicting amide-I region infrared spectra from coarse-grained protein simulations, which reduces the computational cost of spectral prediction. The first stage is a minimal, spectroscopically motivated backmapping diffusion model, and the second stage is a transformer-based, site-frequency perturbation model. The predictions replace the standard vibrational spectroscopic maps while preserving the core of the spectral calculation. We show that this framework predicts accurate infrared spectra for a wide variety of proteins. Finally, we show that the framework is modular, allowing for the integration of existing coarse-grained backmapping models.