Recent large-scale investigations suggest that artificial intelligence (AI) can forecast study results in the social sciences with surprising accuracy. Such forecasts have strong potential to advance the social sciences; for example, they could be used for sample size planning, materials pretesting, or even to replace actual data collection. However, previous studies on AI forecasts largely relied on using highly specialized AI models or complex programming. Thus, a critical practical question remains: can publicly available off-the-shelf AI models, such as ChatGPT and Gemini, likewise achieve a high performance? To answer this question, we conducted a series of 26 studies, using thesis projects in our lab. All studies used the Dictator Game or the Taking Game (N = 7,393) to capture real incentivized behaviors. Prior to data collection, five different AI models (ChatGPT, Claude, Gemini, LeChat, and Perplexity) forecasted the expected effect sizes, which were then compared against the actual results. Across our studies, we observed that while the models predicted the direction of the observed effects better than random guessing, they significantly overestimated the actual effect sizes. Moreover, the correlation between predicted and observed effects was substantially smaller than reported in previous literature, similar to human forecasts, and often statistically non-significant. Overall, our findings suggest that standard, publicly available AI models do not currently constitute an effective tool for precise effect forecasting. Consequently, relying on such forecasts, for example, in power analyses or pretesting, is currently not a recommended method for the social sciences.