Based on Monte Carlo calculations, the magnetic ordering in itinerant antiferromagnetic monolayers with geometric frustration has been analyzed. For description of the itinerant magnetism exchange interactions beyond the Heisenberg model such as the biquadratic exchange and the four-spin exchange interaction have been taken into account. We demonstrate that the higher-order exchange interactions remove the structural degeneracy and lead to configurational phase transitions at finite temperatures. A new, entropy driven structural phase transition in V/Ag(111) is predicted theoretically.