Using spin-polarized scanning tunneling microscopy and density functional theory we demonstrate the occurrence of a novel type of noncollinear spin structure in Rh/Fe atomic bilayers on Ir(111). We find that higher-order exchange interactions depend sensitively on the stacking sequence. For fcc-Rh/Fe/Ir(111), frustrated exchange interactions are dominant and lead to the formation of a spin spiral ground state with a period of about 1.5 nm. For hcp-Rh/Fe/Ir(111), higher-order exchange interactions favor an up-up-down-down (↑↑↓↓) state. However, the Dzyaloshinskii-Moriya interaction at the Fe/Ir interface leads to a small angle of about 4° between adjacent magnetic moments resulting in a canted ↑↑↓↓ ground state.