While the phase diagrams of the one- and multiorbital Hubbard model have been well studied, the physics of real Mott insulators is often much richer, material dependent, and poorly understood. In the prototype Mott insulator V2O3, chemical pressure was initially believed to explain why the paramagnetic-metal to antiferromagnetic-insulator transition temperature is lowered by Ti doping while Cr doping strengthens correlations, eventually rendering the high-temperature phase paramagnetic insulating. However, this scenario has been recently shown both experimentally and theoretically to be untenable. Based on full structural optimization, we demonstrate via the charge self-consistent combination of density functional theory and dynamical mean-field theory that changes in the V2O3 phase diagram are driven by defect-induced local symmetry breakings resulting from dramatically different couplings of Cr and Ti dopants to the host system. This finding emphasizes the high sensitivity of the Mott metal-insulator transition to the local environment and the importance of accurately accounting for the one-electron Hamiltonian, since correlations crucially respond to it.