With the advent of foundation models, fine-tuning has become a prominent approach for long-tailed learning. Recent methods primarily explore different long-tailed strategies under the standard fine-tuning paradigm. However, this default paradigm typically discards the pre-trained classifier and instead re-trains a new one from scratch. In this paper, we observe that such re-trained classifiers suffer from weakened discriminative ability and semantic awareness under tail-data scarcity, exhibiting severe imbalance in class-discriminative channels and mislearning general features for tail classes. In contrast, the overlooked pre-trained classifier, shaped by large-scale pre-training, behaves much closer to the oracle. Motivated by this, we propose a simple yet effective fine-tuning paradigm, PTClf (Pre-Trained Classifier helps), which leverages priors from the pre-trained classifier to guide the re-trained one in learning tail classes. Experiments show that PTClf delivers remarkable benefits for long-tailed data, especially on tail classes, while also exhibiting strong versatility in few-shot learning and domain generalization. The code is available at https://github.com/ckshang/PTClf.