Numerous complex non-collinear spin structures such as spin spirals and skyrmions have been observed recently in ultrathin transition-metal films at surfaces. In these systems the interfacial Dzyaloshinskii-Moriya interaction (DMI) plays a decisive role for the non-collinearity. Here, we demonstrate by density functional theory (DFT) and spin-polarized scanning tunneling microscopy (SP-STM) the occurrence of a spin spiral state which, in contrast, is induced by the frustration of exchange interactions. The DMI further stabilizes right rotating cycloidal spirals, however, the energy gain is small compared to that from the exchange. STM images reveal pseudomorphic growth of single atomic Rh layers on an atomic Fe layer on Ir(111) and show a stripe pattern with periods of 1.5 nm and 1.1 nm for the fcc and hcp Rh stacking, respectively, in very good agreement with the spin spiral periods obtained from DFT. For the fcc Rh stacking the spin spirals run along the $\left<11\bar{2}\right>$ directions, whereas for the hcp stacking the propagation direction of the spin spiral is tilted by an angle of about $\pm 10^{\circ}$ from this high-symmetry direction. Spin dynamics simulations show that magnetic fields of about 50 T are required to stabilize a hexagonal lattice of nanoscale skyrmions.