It is shown how cosmological perturbation theory arises from a fully quantized perturbative theory of quantum gravity. Central for the derivation is a non-perturbative concept of gauge-invariant local observables by means of which perturbative invariant expressions of arbitrary order are generated. In particular, in the linearised theory, first order gauge-invariant observables familiar from cosmological perturbation theory are recovered. Explicit expressions of second order quantities are presented as well.