We provide a physical interpretation of the first and second order terms occurring in Ruelle's response formalism. We show that entropy fluxes play a major role in determining the response of the system to perturbations. Along this line, we show that our framework allows one to recover a wealth of previous results of response theory in both deterministic and stochastic contexts. In particular, we are able to shed light on the crosslinks between ate dynamical systems approach a la Ruelle and large deviations methods.