Recently a powerful duality between color and kinematics has been proposed for integrands of scattering amplitudes in quite general gauge theories. In this paper the duality proposal is extended to the more general class of gauge theory observables formed by form factors. After a discussion of the general setup the existence of the duality is verified in two- and three-loop examples in four-dimensional maximally supersymmetric Yang-Mills theory which involve the stress energy tensor multiplet. In these cases the duality reproduces known results in a particularly transparent and uniform way. As a nontrivial application we obtain a very simple form of the integrand of the four-loop two-point (Sudakov) form factor which passes a large set of unitarity cut checks.