This work explores how the Fractions Skill Score (FSS), originally developed for deterministic forecasts of binary events, can be used for probabilistic forecast verification. By comparing a selection of four ensemble-based methods to compute the FSS, we highlight their distinct behaviour with ensemble size, neighbourhood size, and frequency of occurrence of the forecast event. Our study emphasizes that only a specific variant of the FSS, which we refer to as "probabilistic FSS", demonstrates reasonable behaviour with ensemble size. We reveal that the probabilistic FSS depends on ensemble size in a similar way as the Brier Skill Score, despite performing a neighbourhood-based instead of a grid-point-based forecast evaluation. We derive a formula that describes the expected behaviour of the probabilistic FSS with changes in ensemble size. Finally, utilizing a unique dataset of high-resolution 1000-member ensemble precipitation forecasts for Germany, we explore the impact of ensemble and neighbourhood size on the predictive skill by studying various subsamples of the full ensemble.