A modular fusion category C allows one to define projective representations of the mapping class groups of closed surfaces of any genus. We show that if all these representations are irreducible, then C has a unique Morita-class of simple non- degenerate algebras, namely that of the tensor unit. This improves on a result by Andersen and Fjelstad, albeit under stronger assumptions. One motivation to look at this problem comes from questions in three-dimensional quantum gravity.