Attention is largely guided by objects; when attending to a location on an object, observers react faster to targets appearing in other locations within the object than outside the object. While crucial to our understanding of attention, the mechanisms underlying object-based attention remain a topic of debate. Moreover, the behavioral effects of object-based attention are often small and inconsistent, posing challenges for obtaining unequivocal evidence about its mechanisms. To address these challenges, we employ a continuous, response-free measure of object-based attention that utilizes attentional modulation of the pupillary light response (PLR). This method enables continuous tracking of attention across space while avoiding potential inconsistencies introduced by behavioral responses. In each trial, a cue appeared inside or outside a black or white object, followed by a target in the same location as the cue or in a different location. Across four experiments, attentional modulations of the PLR were consistently stronger when attention shifted from a location outside the object to inside it (cue outside, target inside) than in the reverse scenario (cue inside, target outside). This finding reveals a critical difference in the dynamics of attentional shifts involving objects, suggesting that spatial reallocation of attention is more effective when disengagement from an object is not required. These results align with the shifting account of object-based attention, which underscores the higher cost of shifting attention between objects than within an object.