Quantum electrodynamics predicts that the quantum vacuum is birefringent, but due to the very small cross-section this is yet to be confirmed by experiment. Vacuum birefringence arises as the elastic part of photon-photon scattering; the inelastic part is Breit-Wheeler pair production. We outline how measurements of the photon-polarized nonlinear Breit-Wheeler process can be used to infer a measurement of all-order, nonlinear vacuum birefringence. As an example scenario, we calculate the accuracy of such a measurement for parameters anticipated at upcoming laser-particle experiments.