A clear separation of the timescales governing the dynamics of "slow"and "fast"degrees of freedom often serves as a prerequisite for the emergence of an independent low-energy theory. Here, we consider (slow) classical spins exchange coupled to a tight-binding system of (fast) conduction electrons. The effective equations of motion are derived under the constraint that the quantum state of the electron system at any instant of time t lies in the n-dimensional low-energy subspace for the corresponding spin configuration at t. The effective low-energy theory unfolds itself straightforwardly and takes the form of a non-Abelian gauge theory with the gauge freedom given by the arbitrariness of the basis spanning the instantaneous low-energy sector. The holonomic constraint generates a gauge-covariant spin-Berry curvature tensor in the equations of motion for the classical spins. In the non-Abelian theory for n>1, opposed to the n=1 adiabatic spin dynamics theory, the spin-Berry curvature is generically nonzero, even for time-reversal-symmetric systems. Its expectation value with the representation of the electron state is gauge invariant and gives rise to an additional geometrical spin torque. Aside from anomalous precession, the n≥2 theory also captures the spin nutational motion, which is usually considered as a retardation effect. This is demonstrated by proof-of-principle numerical calculations for a minimal model with a single classical spin. Already for n=2 and in parameter regimes where the n=1 adiabatic theory breaks down, we find good agreement with results obtained from the full (unconstrained) theory.