Fixed-environment matched-field processing (MFP) is vulnerable when sound-channel-axis mismatch and array noise coexist. To address this problem, an energy–residual weighted joint source–environment matched-field localization method is proposed. Source range, source depth, and sound-channel-axis depth are incorporated into a unified search space, and candidates are evaluated using element-received energy, residuals after complex-gain alignment, conventional coherent matching, and weighted matching. To improve stability in the multimodal search space, the method combines multi-start genetic search, MFP-guided initialization, dynamic-neighborhood exploitation, local fine-grid refinement, low-fitness re-search, and posterior screening based on ambiguity score and peak contrast. Observation and replica fields are generated using a Munk sound-speed profile and a four-path image-source approximation. With a fixed 100-m sound-channel-axis mismatch and no added noise, the RMSEs of conventional MFP, standard GA-MFP, joint MFP, and ER-MPF are 417.61, 550.88, 2.52, and 33.08 m, respectively, showing that joint source–environment search is the primary mechanism for deterministic mismatch compensation. Under 5-dB circular complex Gaussian noise, energy–residual weighting reduces RMSE from 90.52 to 39.39 m and increases the 100-m hit rate from 93\% to 97\%. These results demonstrate improved robustness to environmental mismatch and noise under controlled simulation conditions.