Reflection gratings enable light coupling to optical cavities without transmission through substrates. Gratings that have three ports and are mounted in a second-order Littrow configuration even allow the coupling to high-finesse cavities using low diffraction efficiencies. In contrast to conventional transmissive cavity couplers, however, the phase of the diffracted light depends on the lateral position of the grating, which introduces an additional noise coupling. Here, we experimentally demonstrate that this kind of noise cancels out once both diffracted output ports of the grating are combined. We achieve the same signal-to-shot-noise ratio as for a conventional coupler. From this perspective, three-port grating couplers in a second-order Littrow configuration remain a valuable approach to reducing optical absorption of cavity coupler substrates in future gravitational-wave detectors.