The three-dimensional confinement characterizing a nanocrystal (NC) leads to the formation of discrete electronic states. The energy gap between these states in colloidal NCs can be up to an order of magnitude larger than the vibrational energy of the host material. This large energetic mismatch (not given in self-assembled quantum dots) leads to the expectation that an electron occupying an excited state would be unable to release its energy to vibrations and a “phonon bottleneck” should finally be observed. Using large-scale ab initio calculations and a time-dependent formalism, we show that on the contrary, a phonon bottleneck can be observed only in a narrow window of diameters for CdSe and InAs NCs and should not occur at all in Si NCs. Two relaxation pathways enable fast carrier relaxation. For smaller structures (below 20-Å radius), the coupling strength and energy detuning are such that quantum mechanics allows us to effectively bridge electronic gaps much larger than the vibronic energy. For larger structures, the coupling to passivant modes, although very weak, leads to an efficient picosecond carrier relaxation. This work provides insight into the nature of carrier relaxation in colloidal nanostructures and highlights that defects, of any kind, are not necessary to explain the observed fast carrier relaxation.