We demonstrate that the fusion algebra of conformal defects of a two-dimensional conformal field theory contains information about the internal symmetries of the theory and allows one to read off generalizations of Kramers-Wannier duality. We illustrate the general mechanism in the examples of the Ising model and the three-state Potts model.