Single-pixel imaging (SPI) records the integrated intensity of an encoded scene with a single-pixel detector and reconstructs a two-dimensional image from undersampled one-dimensional measurements. Because SPI measurements mix full-image information, global structures are difficult to disentangle at low sampling ratios, while low-energy high-frequency components, such as edges and directional textures, are prone to degradation. Transformers improve global modeling, but the quadratic complexity of self-attention limits lightweight deployment. Existing local-enhancement and multi-scale methods improve detail recovery, yet still struggle to balance global structures, directional high-frequency information, and deployment efficiency at low sampling ratios. To address these issues, we propose DuFRe-Net, a physics-guided dual-frequency-domain reconstruction network. It unfolds the Kronecker SPI model and embeds measurement-consistency constraints into each reconstruction stage. A Dual-Domain Wavelet–Fourier Modulation module is developed. The Fourier branch establishes long-range structural relationships at low cost, while the wavelet branch uses stable low-frequency structures to guide directional high-frequency recovery. Both branches adopt lightweight multi-head modulation to avoid explicit correlation computation. The Partial-Channel Feed-forward Network further reduces redundancy in feature refinement. Experiments across multiple sampling ratios on Set11 and BSD68, together with validation on a self-built real SPI system, show that DuFRe-Net achieves the best or competitive reconstruction performance. At SR=0.10, it contains only 0.645 M parameters and occupies 2.46 MB. These results demonstrate a favorable balance among global structure recovery, directional detail preservation, and deployment efficiency at low complexity.