Mapping class group averages appear in the study of 3D gravity partition functions. In this paper, we work with 3D topological field theories to establish a bulk-boundary correspondence between such averages and correlators of 2D rational CFTs whose chiral mapping class group representations are irreducible and satisfy a finiteness property. We show that Ising-type modular fusion categories satisfy these properties on surfaces with or without field insertions, extending results in [Jian et al., JHEP 10 (2020) 129], and we comment on the absence of invertible global symmetries in the examples we consider.