We demonstrate that quantum well states (QWS) of thin Pb films are highly perturbed within the proximity of intrinsic film defects. Scanning tunneling microscopy/spectroscopy measurements indicate that the energy of these states has a strong distance dependence within 4 nm of the defect with the strongest energetic fluctuations equaling up to 100 meV. These localized perturbations show large spatially dependent asymmetries in the local density of states around the defect site for each corresponding quantum well state. These energetic fluctuations can be described by a simple model, which accounts for fluctuations in the confinement potential induced by topographic changes.