To address the disadvantage of high cost, long cycle and low throughput of traditional hot tearing evaluation method of magnesium alloys, the present study develops a machine learning-based prediction method for the hot-tearing susceptibility criterion (HSC) of Mg–Zn–Y ternary alloys. Based on the Clyne–Davies criterion, a HSC database covering 225 alloy compositions is established. After model evaluation and optimization, the random forest regression (RFR) model achieves a correlation coefficient (R2) of 0.91 and a root mean squared error (RMSE) of 0.09. T-shaped hot tearing experiments confirm that the predicted results are highly reliable. Alloys constitute analysis demonstrates that a low Zn/Y ratio (≤ 1) promotes the formation of LPSO phase to restrain hot tearing, whereas a high Zn/Y ratio (≥ 2.25) aggravates hot tearing due to the precipitation of brittle I phase and the widened solidification temperature range of 207.5 °C. Feature optimization by introducing Zn/Y ratio further improves the model performance, with the R2 increased to 0.96 and the RMSE reduced to 0.05. This work reveals a relationship among alloy composition, phase constitute, solidification behavior and defect susceptibility, providing a data-driven approach for the design of magnesium alloys with low hot tearing susceptibility.