Annotation agreement is an important indicator of annotation quality in supervised text classification; however, its effect on financial sentiment classification has received limited attention. This study investigates the influence of annotation agreement on classification performance using the Financial PhraseBank dataset. Three mutually exclusive datasets representing high, medium, and low agreement annotations were constructed to eliminate duplicated instances across agreement levels. Nine within-level and cross-level experimental settings were evaluated using two preprocessing configurations, TF-IDF unigram features, and eight traditional machine learning classifiers. Experimental results showed that classification performance consistently decreased as annotation agreement declined. Models evaluated on higher-agreement test datasets consistently achieved better performance than those evaluated on lower-agreement datasets, regardless of the annotation agreement level of the training data. Linear SVM and SGD (Hinge) achieved the highest Macro-F1 scores across most experimental settings, whereas lemmatization provided only limited and inconsistent improvements. These findings demonstrate that annotation agreement is an important characteristic of annotated datasets and should be explicitly considered when constructing benchmark datasets, evaluating financial sentiment classification models, and interpreting experimental results.