Collective modes in three-dimensional crystals of stacked permalloy disks with magnetic vortices are investigated by ferromagnetic resonance spectroscopy and scanning transmission X-ray microscopy. The size of the arrangements is increased step by step to identify the different contributions to the interaction between the vortices. These contributions are the key requirement to understand complex dynamics of three dimensional vortex crystals. Both vertical and horizontal coupling determine the collective modes. In -plane dipoles strongly influence the interaction between the disks in the stacks and lead to polarity -dependent resonance frequencies. Weaker contributions discern arrangements with different polarities and circularities that result from the lateral coupling of the stacks and the interaction of the core regions inside a stack. All three contributions are identified in the experiments and are explained in a rigid particle model.