End-to-end latency alone cannot guide migration to a new accelerator: a slowdown reveals neither which operators account for the gap nor which optimizations will transfer across hardware. We present HET, a white-box benchmark and attribution framework for inference migration. HET combines (i) counterfactual operator performance attribution (COPA), which estimates each operator's contribution to a migration gap with a surrogate-assisted antithetic Shapley estimator, and (ii) a model-operator-hardware knowledge graph (MOH-KG), which represents structural and performance relations and drives a KG-guided PPO optimizer (KG-A2O). The benchmark contains 34 models and 6,244 profiled operator instances spanning five workload categories and five computing platforms. The graph predictor achieves 14.3% mean relative error (MRE), compared with 25.6% for Roofline and 32.0% for DeepBench. COPA reaches 0.98% attribution MRE with 500 samples, while its surrogate reduces query latency by 288-8,252×. In the evaluated A100-to-Ascend 910B migration case, the complete optimization plan yields a 15.6% speedup and a 66% cost reduction while meeting the taskquality threshold.