When two peripheral objects are presented in close proximity, saccades towards one of these objects land at a weighted average location between the two objects. This phenomenon, known as the 'global effect' or 'saccade averaging', disappears when the distance between the objects increases. When objects are further apart, outside the averaging zone, saccades land on one of the objects with little or no saccade averaging. Although it is known that the strength of the global effect is dependent on the specific features of the two objects, it is unclear if the size of the zone in which averaging can occur (i.e., the averaging zone) is adaptive. The aim of the current study was to investigate whether the size of the averaging zone adapts to variations in object luminance contrast of the objects. In order to systematically assess changes in the averaging zone, in two experiments, observers made saccadic eye movements while the luminance of the target and the distractor varied. We report three major findings: 1) When a distractor was more luminant relative to the target, the averaging zone increased (Exp. 1). Notably, saccade averaging never entirely ceased to exist, even for remote distractors. 2) When target and distractor were equiluminant, the averaging zone did not change with absolute luminance (Exp. 2). 3) Higher (relative and absolute) luminance increased the averaging zone especially for shorter saccadic response times (SRT). We conclude that the averaging zone is adaptive and becomes larger with increasing relative luminance and especially when SRTs are short.