Ultrasmall CdSe quantum dots (QDs) with diameters up to 2 nm show broad photoluminescence (PL) spectra presumably due to emission from band-edge excitons and defect states. However, the origin of the defect emission and the effect of defects on the band-edge excitons is not fully understood. Based on spin–orbit density functional theory and screened configuration interaction singles, we show that Cd-dimer and Se defects form in-gap defect states. In comparison with experiment, we discuss the role of deep and shallow defect states for the PL and cover the dependence of their contributions to the PL with respect to the QD size. Further, we observe that these defects lead to a localization of the molecular orbitals (MOs) involved in the band-edge excitons creating large electric dipoles in the MOs. In the excitonic states, these dipoles cause multiexponential PL decay from the band-edge states with a highly anisotropic polarization of the emission. The polarization is found to be very sensitive with respect to the exact composition of the surface.