Quantum confinement leads to the emergence of several magnon modes in ultrathin layered magnetic structures. We probe the lifetime of these quantum confined modes in a model system composed of three atomic layers of Co grown on different surfaces. We demonstrate that the quantum confined magnons exhibit nonlinear decay rates, which strongly depend on the mode number, in sharp contrast to what is assumed in the classical dynamics. Combining the experimental results with those of linear-response density functional calculations we provide a quantitative explanation for this nonlinear damping effect. The results provide new insights into the decay mechanism of spin excitations in ultrathin films and multilayers and pave the way for tuning the dynamical properties of such structures.