We study the temperature dependence as well as anisotropy of optical conductivity (sigma(1)) in the pseudocubic single crystal Pr0.5Ca1.5MnO4 using spectrocopic ellipsometry. Three transition temperatures are observed and can be linked to charge-orbital (T-CO/OO similar to 320 K), two-dimensional-antiferromagnetic (2D-AFM) (similar to 200 K), and three-dimensional AFM (T-N similar to 125 K) orderings. Below T-CO/OO, sigma(1) shows a charge-ordering peak (similar to 0.8 eV) with a significant blue shift as the temperature decreases. Calculations based on a model that incorporates a static Jahn-Teller distortion and assumes the existence of a local charge imbalance between two different sublattices support this assignment and explain the blue shift. This view is further supported by the partial spectral weight analysis showing the onset of optical anisotropy at T-CO/OO in the charge-ordering region (0.5-2.5 eV). Interestingly, in the charge-transfer region (2.5-4 eV), the spectral weight shows anomalies around the T2D-AFM that we attribute to the role of oxygen-p orbitals in stabilizing the CE-type magnetic ordering. Our result shows the importance of spin, charge, orbital, and lattice degrees of freedom in this layered manganite.