The feedback of the geometrical Berry phase, accumulated in an electron system, on the slow dynamics of classical degrees of freedom is governed by the Berry curvature. Here, we study local magnetic moments, modeled as classical spins, which are locally exchange coupled to the (spinful) Haldane model for a Chern insulator. In the emergent equations of motion for the slow classical-spin dynamics there is a an additional anomalous geometrical spin torque, which originates from the corresponding spin Berry curvature. Due to the explicitly broken time-reversal symmetry, this is nonzero but usually small in a condensed-matter system. We develop the general theory and compute the spin Berry curvature, mainly in the limit of weak exchange coupling, in various parameter regimes of the Haldane model, particularly close to a topological phase transition and for spins coupled to sites at the zigzag edge of the model in a ribbon geometry. The spatial structure of the spin Berry curvature tensor, its symmetry properties, the distance dependence of its nonlocal elements, and further properties are discussed in detail. For the case of two classical spins, the effect of the geometrical spin torque leads to an anomalous non-Hamiltonian spin dynamics. It is demonstrated that the magnitude of the spin Berry curvature is decisively controlled by the size of the insulating gap, the system size, and the strength of local exchange coupling.